# Snowplow Documentation

> Authoritative Snowplow documentation for implementing event tracking, validation, enrichment, governance, and delivery of clean event-level behavioral data. Focus areas include composable analytics, composable CDP, in-product personalization, AI agentic applications, and feeding AI-ready real-time data into warehouses, lakes, streams, and real-time tools.

Documentation for previous versions of components is available on the site but is not included in this file.

---

# Manage your Snowplow account using the Credentials API
> Manage your Snowplow account configuration, users, and API keys through Console, including instructions for obtaining JWT tokens via the Credentials API.
> Source: https://docs.snowplow.io/docs/account-management/

Manage your account configuration and users using the Snowplow Console. You can also use the underlying API directly. This page describes how to acquire an API key.

## Credentials API

The API that drives Console's functionality is [publicly documented](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml) and available for customers to invoke via code. All calls to it need to be properly authenticated using JSON Web Tokens (JWT) that can be acquired via the Credentials API.

The following view is available in [Console](https://console.snowplowanalytics.com/), under **Settings** in the navigation bar, then **Manage organization**, then **API keys for managing Snowplow**. Users can view this page only if they have the "view" permission on API keys.

![](/assets/images/accessing-generated-api-keys-8dd552f45cdaec4af7a5a070c498786e.png)

API keys generation view

You can create multiple API keys, and it's also possible to delete any key. When a new API key is generated, the following view will appear:

![](/assets/images/generated-api-key-v3-b28f53c0b129d6da072b23a0a98d4cf4.png)

Newly created API key view

Both the API key ID and API key are required. The API key functions like a combination of a username and password, and should be treated with the same level of security. Once you have an API key and key ID, exchanging it for a JWT is straightforward. For example, using curl, the process would look like this:

```bash
curl \
  --header 'X-API-Key-ID: <API_KEY_ID>' \
  --header 'X-API-Key: <API_KEY>' \
  https://console.snowplowanalytics.com/api/msc/v1/organizations/<ORGANIZATION_ID>/credentials/v3/token
```

You can find your Organization ID [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

The curl command above will return a JWT as follows:

```json
{ "accessToken": "<JWT>" }
```

You can then use this access token to supply authorization headers for subsequent API requests:

```bash
curl --header 'Authorization: Bearer <JWT>'
```

### Previous versions

A previous version of the token exchange endpoint is still available, only requiring the API key:

```bash
curl \
  --header 'X-API-Key: <API_KEY>' \
  https://console.snowplowanalytics.com/api/msc/v1/organizations/<ORGANIZATION_ID>/credentials/v2/token
```

While this method will continue to work, the endpoint is now deprecated and will be removed in the future. Use the v3 endpoint detailed above instead.

---

# Available Console user permissions and roles
> Configure user permissions in Snowplow Console with Global Admin, User, and Custom roles to control access to environments, data structures, tracking plans, data models, and API keys.
> Source: https://docs.snowplow.io/docs/account-management/managing-permissions/

To set a user's permissions, navigate to **Settings** > **Users** and then to the user whose account you'd like to manage.

## What permissions can be set?

Snowplow Console sets permissions for each area of Console as summarized below:

| **Console feature** | **Description**                                                                                                            | **Possible permissions**                                     |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| User management     | The management and addition of user access. This permission cannot be configured on a Custom role.                         | No access Edit Create                                        |
| Environments        | The management of pipeline and development environments. This includes managing which Enrichments run on each environment. | No access View Edit                                          |
| Tracking plans      | The management and creation of tracking plans.                                                                             | No access View Edit Create                                   |
| Data structures     | The management and creation of the schemas that define the events and entities you are capturing.                          | No access View Edit on development Edit on production Create |
| Data models         | The management and creation of your data models.                                                                           | No access View Edit Create                                   |
| API keys            | The management and creation of API keys.                                                                                   | No access View Manage                                        |

## How are permissions set?

To set permissions for a user, navigate to **Settings** > **Users** and select the user, within the management screen for their user you will be able to set their permissions.

There are three ways of setting user permissions:

- Global Admin (pre-defined role)
- User (pre-defined role)
- Custom (custom permissions role)

The following tables describe the default permissions for each role.

#### User permission set

| **Console feature** | **Permissions** |
| ------------------- | --------------- |
| Environments        | View access     |
| User management     | View access     |
| Tracking plans      | View access     |
| Data structures     | View access     |
| Data models         | View access     |
| API keys            | View access     |

#### Global Admin permission set

| **Console feature** | **Permissions** |
| ------------------- | --------------- |
| User management     | Full access     |
| Environments        | Full access     |
| Tracking plans      | Full access     |
| Data structures     | Full access     |
| Data models         | Full access     |
| API keys            | Full access     |

#### Custom permission set

| **Console feature** | **Permissions**             |
| ------------------- | --------------------------- |
| User management     | Customized by you, per user |
| Environments        | Customized by you, per user |
| Tracking plans      | Customized by you, per user |
| Data structures     | Customized by you, per user |
| Data models & jobs  | Customized by you, per user |
| API keys            | Customized by you, per user |

A note on API keys and permissions

Please note:

1. Any API keys you create have full admin permissions

2. Any existing Iglu API keys allow permissions to be side-stepped by connecting directly to Iglu servers

The recommended approach is to remove all existing API keys and Iglu keys, and set the API keys permission respectively so that only trusted users can create new keys.

## What does each permission mean?

### Environments

An environment is the collective name for your Production pipelines, QA pipelines and development environments.

An environment has three permissions:

- **No access** - the user will not see the environment management screens.
- **View** - the user can see the environment management screen, but cannot edit anything. This is the default setting for the User role.
- **Edit** - the user can make edits to the environment. This includes configuration such as enrichment enablement, enrichment configuration and collector configuration.

### Tracking plans

Tracking plans have four permissions:

- **No access** - the user will not see the tracking plan management screens.
- **View** - the user can see the tracking plan management screens, but cannot edit anything. This permission and all tracking plan permissions below require the user to have at least the **View** permission on data structures.
- **Edit** - the user can see the tracking plan management screens, and can make edits to existing tracking plans.
- **Create** - the user can create new tracking plans.

### Data structures

Data structures have five permissions:

- **No access** - the user will not see the data structure management screen.
- **View** - the user can see the data structure management screen, but cannot edit anything.
- **Edit on development** - the user can see the data structure management screen, and can make edits to data structures but only publish them to the development registry.
- **Edit on production** - the user can see the data structure management screen, and can make edits to data structures, and can publish changes to the production registry.
- **Create** - the user can create new data structures.

### Data models

Data models and jobs have four permissions:

- **No access** - the user will not see the data model management screens.
- **View** - the user can see the data model management screens, but cannot edit anything. This is the default setting for the User role.
- **Edit** - the user can see the data model management and can make edits to data models in production. This is the default setting for the Global Admin role.
- **Create** - the user can create new data models.

### API keys

API keys have four permissions:

- **No access** - the user will not see the API key management screens.
- **View** - the user can see the API key descriptions but cannot see the keys themselves or manage them.
- **Manage** - the user can see and manage the API keys.
- **Create** - the user can generate new API keys.

## Troubleshooting

You shouldn’t be required to logout for new permissions to take effect, but if you do find permissions aren’t applying as requested logging out and back in should force the new permissions to apply.

---

# Manage users in Console directly or with SSO
> Add and remove users in Snowplow Console, configure Single Sign-On with identity providers including Google Workspace, Entra ID, Okta, and OpenID Connect.
> Source: https://docs.snowplow.io/docs/account-management/managing-users/

There are two ways to add and remove users in Console: directly managed in Console, or managed through your Single Sign-On (SSO) provider.

SSO is an authentication process that allows users to access multiple applications after signing in to a central Identity Provider. Snowplow supports SSO integration for the majority of identity providers.

For organizations **not using SSO**, users can be added and removed directly in Console by navigating to **Settings** > **Users** in the navigation and creating a new user, or removing an existing user from there. Newly added users will receive an email to set their password and will be added with limited permissions, which you can then widen.

For organizations **using SSO**, you will need to configure your account with your Identity Provider before you can add or remove users.

## SSO permissions

Only system administrators can set up SSO for their company. For information on setting permissions for individual users, see [Managing user permissions](/docs/account-management/managing-permissions/).

## How to enable SSO for your account

Setting up SSO for your account requires some information to be exchanged between you as the Identity Provider and Snowplow as the Service Provider. Depending on your Identity Provider, the information that is required is slightly different.

To enable single sign-on (SSO) for Snowplow, follow these steps inside Console:

1. Go to the [manage organization](https://console.snowplowanalytics.com/settings) page.
2. Select [Single sign-on (SSO)](https://console.snowplowanalytics.com/users) from the Users panel. The SSO configuration is only visible to users with Admin role.
3. Click **Continue** and follow the steps for your Identity Provider.

## Which Identity Providers (IdPs) are supported?

Snowplow’s SSO capability enables connections with many IdPs, including: 

- ADFS
- Auth0
- Entra ID (formerly known as Azure AD)
- Google Workspace
- Keycloak
- Okta
- PingFederate

Because Snowplow supports OpenID Connect and SAML, virtually any external Identity Provider that uses those standards should work.

## What information will you need from us?

This will differ depending on your Identity Provider, but typically will include information such as:

- **Entity ID** - the URL that identifies the identity provider issuing a SAML request, this will be specific to your identity provider.
- **Metadata URL** - the URL that allows access to obtain SSO configuration data, this will be specific to your identity provider.
- **Redirect Login URL** - the URL where users in the company sign in to the identity provider.
- **User information mapping** - locations of information required by Snowplow Console such as first name, last name and, optionally, job title.

## What happens when SSO is enabled?

### Adding new users

Snowplow supports just-in-time provisioning with SSO connections. When a user logs in for the first time, a corresponding user account with the same email is created in Snowplow.

A new user created via SSO will have a custom permissions set that allows them to view-only, as outlined below. This can then be edited by anyone with the Global Admin role on your account. For more details on setting user access, see [Managing user permissions](/docs/account-management/managing-permissions/).

### Existing users

If a user already has a Snowplow account prior to SSO being enabled, the two accounts will be merged, and the user's current permissions will be applied.

### Logging in 

When SSO is enabled, anybody who signs into Snowplow Console with an email address that uses your specified domain will be authenticated via SSO and your Identity Provider.

Once SSO is enabled, users on your domain can no longer sign in with their old email address and password, or manage their personal details or password as these will all be managed within your Identity Provider.

## Disabling SSO

If your company enables SSO, and later decides to disable it:

- Users who did not set up a password before SSO was enabled must click Reset password on the login page to obtain a password.
- Users who set up a password before SSO was enabled can log in with their old username and password.

---

# Go Analytics SDK
> Go library for parsing Snowplow enriched events with efficient field access and transformation to JSON or maps for serverless functions.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-go/

## 1. Overview

The [Snowplow Analytics SDK for Go](https://github.com/snowplow/snowplow-golang-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Go event processing, data modeling and machine-learning jobs. You can use this SDK with [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions/), [Google App Engine](https://cloud.google.com/appengine) and other Golang-compatible data processing frameworks.

## 2. Compatibility

Snowplow Analytics SDK fo Go was tested with Go versions 1.13-1.15.

There are only two external dependencies currently:

- `github.com/json-iterator/go` - used to parse JSON

- `github.com/pkg/errors` - used to provide an improvement on the standard error reporting.

## 3. Setup

snowplow/snowplow-golang-analytics-sdk can be imported to a project as a go module:

`go get github.com/snowplow/snowplow-golang-analytics-sdk`

## 4. Usage

### 4.1 Overview

The [Snowplow Analytics SDK for Go](https://github.com/snowplow/snowplow-golang-analytics-sdk) provides you an API to parse an enriched event from it's TSV-string form to a `ParsedEvent` slice of strings, then a set of methods to transform the entire event or a subset of fields into either `JSON` or `map` form. It also offers methods to efficiently get a field from the `ParsedEvent`.

### 4.2 Summary of example usage

```bash
go get github.com/snowplow/snowplow-golang-analytics-sdk
```

```go
import "github.com/snowplow/snowplow-golang-analytics-sdk/analytics"


parsed, err := ParseEvent(event) // Where event is a valid TSV string Snowplow event.
if err != nil {
  fmt.Println(err)
}

parsed.ToJson() // whole event to JSON
parsed.ToMap() // whole event to map
parsed.GetValue("page_url") // get a value for a single canonical field
parsed.GetSubsetMap("page_url", "domain_userid", "contexts", "derived_contexts") // Get a map of values for a set of canonical fields
parsed.GetSubsetJson("page_url", "unstruct_event") // Get a JSON of values for a set of canonical fields
```

### 4.3 API

```go
func ParseEvent(event string) (ParsedEvent, error)
```

ParseEvent takes a Snowplow Enriched event tsv string as input, and returns a 'ParsedEvent' typed slice of strings. Methods may then be called on the resulting ParsedEvent type to transform the event, or a subset of the event to Map or Json.

```go
func (event ParsedEvent) ToJson() ([]byte, error)
```

ToJson transforms a valid Snowplow ParsedEvent to a JSON object.

```go
func (event ParsedEvent) ToMap() (map[string]interface{}, error)
```

ToMap transforms a valid Snowplow ParsedEvent to a Go map.

```go
func (event ParsedEvent) GetSubsetJson(fields ...string) ([]byte, error)
```

GetSubsetJson returns a JSON object containing a subset of the event, containing only the atomic fields provided, without processing the rest of the event.

For custom events and contexts, only "unstruct\_event", "contexts", or "derived\_contexts" may be provided, which will produce the entire data object for that field.

For contexts, the resultant map will contain all occurrences of all contexts within the provided field.

```go
func (event ParsedEvent) GetSubsetMap(fields ...string) (map[string]interface{}, error)
```

GetSubsetMap returns a map of a subset of the event, containing only the atomic fields provided, without processing the rest of the event.

For custom events and entites, only "unstruct\_event", "contexts", or "derived\_contexts" may be provided, which will produce the entire data object for that field.

For contexts, the resultant map will contain all occurrences of all contexts within the provided field.

```go
func (event ParsedEvent) GetValue(field string) (interface{}, error)
```

GetValue returns the value for a provided atomic field, without processing the rest of the event. For unstruct\_event, it returns a map of only the data for the unstruct event. For contexts and derived\_contexts, it returns the data for all contexts or derived\_contexts in the event.

```go
func (event ParsedEvent) ToJsonWithGeo() ([]byte, error)
```

ToJsonWithGeo adds the geo\_location field, and transforms a valid Snowplow ParsedEvent to a JSON object.

```go
func (event ParsedEvent) ToMapWithGeo() (map[string]interface{}, error)
```

ToMapWithGeo adds the geo\_location field, and transforms a valid Snowplow ParsedEvent to a Go map.

---

# JavaScript and TypeScript Analytics SDK
> Lightweight JavaScript and TypeScript library to transform Snowplow enriched TSV events into JSON for serverless functions and Node.js.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-javascript/

## Overview

The [Snowplow JavaScript and TypeScript Analytics SDK](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your JavaScript event processing, data modeling and machine-learning jobs. You can use this SDK with [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions/), [Google App Engine](https://cloud.google.com/appengine) and other JavaScript-compatible frameworks.

## Setup

Install using your preferred package manager, such as npm:

```bash
npm install --save snowplow-analytics-sdk
```

## Usage

### Overview

The [Snowplow JavaScript and TypeScript Analytics SDK](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk) provides you an API to parse an enriched event from it's TSV-string form to a `JSON` string.

### Example

To consume in an AWS lambda you would do something like this in your `app.js`:

```javascript
const { transform } = require('snowplow-analytics-sdk');

module.exports.handler = (input) => {
  let event = transform(
    new Buffer(input.Records[0].kinesis.data, 'base64').toString('utf8'),
  );

  // ...
};
```

Or in `app.ts`:

```javascript
import { transform } from 'snowplow-analytics-sdk';

export function handler(input: any) {
  let event = transform(
    new Buffer(input.Records[0].kinesis.data, 'base64').toString('utf8'),
  );

  // ...
}
```

## API

### `transform(event: string): Event`

- `event: string` - TSV string containing event data.

Returns decoded [Snowplow enriched event](/docs/fundamentals/canonical-event/).

---

# .NET Analytics SDK
> .NET SDK with JSON event transformer for processing Snowplow enriched events in Azure Data Lake Analytics, Azure Functions, and C# applications.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-net/

## 1. Overview

The [Snowplow Analytics SDK for .NET](https://github.com/snowplow/snowplow-dotnet-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your .NET event processing, data modeling and machine-learning jobs. You can use this SDK with [Azure Data Lake Analytics](https://azure.microsoft.com/en-gb/services/data-lake-analytics/), [Azure Functions](https://azure.microsoft.com/en-gb/services/functions/), [AWS Lambda](https://aws.amazon.com/lambda/), [Microsoft Orleans](https://dotnet.github.io/orleans/) and other .NET-compatible data processing frameworks.

The .NET Analytics SDK makes it significantly easier to build applications that consume Snowplow enriched data directly from Event Hubs or Azure Blob Storage.

## 2. Compatibility

Snowplow .NET Analytics SDK targets [.NET Standard 1.3](https://github.com/dotnet/standard/blob/master/docs/versions.md).

## 3. Setup

To add the .NET Analytics as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Analytics
```

## 4. Event Transformer

### 4.1 Overview

The Snowplow enriched event is a relatively complex TSV string containing self-describing JSONs. Rather than work with this structure directly, Snowplow analytics SDKs ship with _event transformers_, which translate the Snowplow enriched event format into something more convenient for engineers and analysts.

As the Snowplow enriched event format evolves towards a cleaner [Apache Avro](https://avro.apache.org/)-based structure, we will be updating this Analytics SDK to maintain compatibility across different enriched event versions.

Working with the Snowplow .NET Analytics SDK therefore has two major advantages over working with Snowplow enriched events directly:

1. The SDK reduces your development time by providing analyst- and developer-friendly transformations of the Snowplow enriched event format
2. The SDK futureproofs your code against new releases of Snowplow which update our enriched event format

Currently the Analytics SDK for .NET ships with one event transformer: the JSON Event Transformer.

### 4.2 The JSON Event Transformer

The JSON Event Transformer takes a Snowplow enriched event and converts it into a JSON ready for further processing. This transformer was adapted from the code used to load Snowplow events into Elasticsearch in the Kinesis real-time pipeline.

The JSON Event Transformer converts a Snowplow enriched event into a single JSON like so:

```json
{
  "app_id":"demo",
  "platform":"web",
  "etl_tstamp":"2015-12-01T08:32:35.048Z",
  "collector_tstamp":"2015-12-01T04:00:54.000Z",
  "dvce_tstamp":"2015-12-01T03:57:08.986Z",
  "event":"page_view",
  "event_id":"f4b8dd3c-85ef-4c42-9207-11ef61b2a46e",
  "txn_id":null,
  "name_tracker":"co",
  "v_tracker":"js-2.5.0",
  "v_collector":"clj-1.0.0-tom-0.2.0",...
```

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's `unstruct_event`, `contexts` and `derived_contexts` fields. All self-describing JSONs found in the event are flattened into top-level plain (i.e. not self-describing) objects within the enriched event JSON.

For example, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the final JSON would contain:

```json
{
  "app_id":"demo",
  "platform":"web",
  "etl_tstamp":"2015-12-01T08:32:35.048Z",
  "unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {
    "targetUrl":"http://www.example.com",
    "elementClasses":["foreground"],
    "elementId":"exampleLink"
  },...
```

### 4.3 Examples

You can convert an enriched event TSV string to a JSON like this:

```csharp
using Snowplow.Analytics.Json;
using Snowplow.Analytics.Exceptions;

try
{
    EventTransformer.Transform(enrichedEventTsv);
}
catch (SnowplowEventTransformationException sete)
{
    sete.ErrorMessages.ForEach((message) => Console.WriteLine(message));
}
```

If there are any problems in the input TSV (such as unparseable JSON fields or numeric fields), the `transform` method will throw a `SnowplowEventTransformationException`. This exception contains a list of error messages - one for every problematic field in the input.

---

# Parse enriched events in Azure Data Lake
> Custom U-SQL extractor for parsing Snowplow enriched events in Azure Data Lake Analytics with direct access to nested context fields.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-net/snowplow-event-extractor/

[Azure Data Lake](https://azure.microsoft.com/en-in/solutions/data-lake/) is a secure and scalable data storage and analytics service. [Azure Data Lake Analytics](https://azure.microsoft.com/en-in/services/data-lake-analytics/) includes [U-SQL](https://blogs.msdn.microsoft.com/visualstudio/2015/09/28/introducing-u-sql-a-language-that-makes-big-data-processing-easy/), a big-data query language for writing queries that analyze data.

## Event Extractor

Snowplow Event Extractor is an ADLA custom extractor that allows you to parse **[Snowplow enriched events](/docs/fundamentals/canonical-event/)**. Snowplow’s enrichment process outputs enriched events in a TSV format consisting of 131 fields.

EventExtractor implements IExtractor interface:

```csharp
[SqlUserDefinedExtractor]
public class EventExtractor : IExtractor
{
    private static readonly string ROW_DELIMITER = '\t';

    public override IEnumerable<IRow> Extract(IUnstructuredReader input, IUpdatableRow output)
    {
       //split the input based on ROW_DELIMITER
       //set the output data on the output object
       //EventExtractor only outputs columns and values that are defined with the output.
    }
}
```

## Usage

Following is base U-SQL script that uses a Event Extractor:

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";

@rs0 =
    EXTRACT
        app_id string,
        platform string
    FROM @input_file
    USING new Snowplow.EventExtractor();
```

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's unstruct\_event, contexts and derived\_contexts fields. Consider contexts found in the tsv:

```json
{
    'schema': 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0',
    'data': [{
        'schema': 'iglu:org.schema/WebPage/jsonschema/1-0-0',
        'data': {
            'genre': 'blog',
            'inLanguage': 'en-US',
            'datePublished': '2014-11-06T00:00:00Z',
            'author': 'Devesh Shetty',
            'breadcrumb': ['blog', 'releases']
        }
    }, {
        'schema': 'iglu:org.w3/PerformanceTiming/jsonschema/1-0-0',
        'data': {
            'navigationStart': 1415358089861,
            'unloadEventStart': 1415358090270,
            'unloadEventEnd': 1415358090287,
            'redirectStart': 0,
            'redirectEnd': 0
        }
    }]
}
```

One of the ways to fetch data from context would be to use user-defined function(UDF):

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";

//extract context from tsv
@rs0 =
    EXTRACT
        context string
    FROM @input_file
    USING new Snowplow.EventExtractor();

/*
context has nested data array
*/
@parseData =
    SELECT Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(context, "data[*]").Values AS data_arr,
    FROM @rs0;

/*
The nested data array inside context consists of an array from which we parse the inner data field
*/
@parseGenre =
    SELECT Microsoft.Analytics.Samples.Formats.Json.JsonFunctions.JsonTuple(data_arr, "$.data.genre").Values AS genre,
    FROM @parseData;
```

The above process can get quite complex. So to abstract away the complexity, Snowplow Event Extractor follows a simple mapping:

```sql
DECLARE @input_file string = @"\snowplow\event.tsv";

//extract genre from context directly
@rsGenre =
    EXTRACT
        context.data.genre
    FROM @input_file
    USING new Snowplow.EventExtractor();
```

---

# Python Analytics SDK
> Python SDK for processing Snowplow enriched events with run manifest support for idempotent data processing in PySpark and AWS Lambda.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-python/

## 1. Overview

The [Snowplow Analytics SDK for Python](https://github.com/snowplow/snowplow-python-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Python event processing, data modeling and machine-learning jobs. You can use this SDK with [Apache Spark](http://spark.apache.org/), [AWS Lambda](https://aws.amazon.com/lambda/), and other Python-compatible data processing frameworks.

## 2. Compatibility

Snowplow Python Analytics SDK was tested with Python of versions: 2.7, 3.3, 3.4, 3.5.

As analytics SDKs supposed to be used heavily in conjunction with data-processing engines such as [Apache Spark](http://spark.apache.org/), our goal is to maintain compatibility with all versions that PySpark supports. Whenever possible we try to maintain compatibility with broader range of Python versions and computing environments. This is achieved mostly by minimazing and isolating third-party dependencies and libraries.

There are only one external dependency currently:

- [Boto3](https://aws.amazon.com/sdk-for-python/) - AWS Python SDK that used to provide access to Event Load Manifests.

These dependencies can be installed from the package manager of the host system or through PyPi.

## 3. Setup

### 3.1 PyPI

The Snowplow Python Analytics SDK is published to [PyPI](https://pypi.python.org/), the official third-party software repository for the Python programming language.

This makes it easy to either install the SDK locally, or to add it as a dependency into your own Python app or Spark job.

### 3.2 pip

To install the Snowplow Python Analytics SDK locally, assuming you already have Pip installed:

```bash
$ pip install snowplow_analytics_sdk --upgrade
```

To add the Snowplow Analytics SDK as a dependency to your own Python app, edit your `requirements.txt` and add:

```text
snowplow_analytics_sdk==0.2.3
```

### 3.3 easy\_install

If you are still using easy\_install:

```bash
$ easy_install -U snowplow_analytics_sdk
```

## 4. Run Manifests

### 4.1 Overview

The [Snowplow Analytics SDK for Python](https://github.com/snowplow/snowplow-python-analytics-sdk) provides you an API to work with run manifests. Run manifests is simple way to mark chunk (particular run) of enriched data as being processed, by for example Apache Spark data-modeling job.

### 4.2 Usage

Run manifests functionality resides in new `snowplow_analytics_sdk.run_manifests` module.

Main class is `RunManifests`, that provides access to DynamoDB table via `contains` and `add`, as well as `create` method to initialize table with appropriate settings. Other commonly-used function is `list_runids` that is gives S3 client and path to folder such as `enriched.archive` or `shredded.archive` from `config.yml` lists all folders that match Snowplow run id format (`run-YYYY-mm-DD-hh-MM-SS`). Using `list_runids` and `RunManifests` you can list job runs and safely process them one by one without risk of reprocessing.

### 4.3 Example

Here's a short usage example:

```python
from boto3 import client
from snowplow_analytics_sdk.run_manifests import *

s3 = client('s3')
dynamodb = client('dynamodb')

dynamodb_run_manifests_table = 'snowplow-run-manifests'
enriched_events_archive = 's3://acme-snowplow-data/storage/enriched-archive/'
run_manifests = RunManifests(dynamodb, dynamodb_run_manifests_table)

run_manifests.create()   # This should be called only once

for run_id in list_runids(s3, enriched_events_archive):
    if not run_manifests.contains(run_id):
        process(run_id)
        run_manifests.add(run_id)
    else:
        pass
```

In above example, we create two AWS service clients for S3 (to list job runs) and for DynamoDB (to access manifests). These clients are provided via [boto3](https://aws.amazon.com/sdk-for-python/) Python AWS SDK and can be initialized with static credentials or with system-provided credentials.

Then we list all run ids in particular S3 path and process (by user-provided `process` function) only those that were not processed already. Note that `run_id` is simple string with S3 key of particular job run.

`RunManifests` class is a simple API wrapper to DynamoDB, using which you can:

- `create` DynamoDB table for manifests,
- `add` run to table
- check if table `contains` run id

---

# Scala Analytics SDK
> Parse Snowplow enriched events into case classes with JSON transformation and event inventory metadata for Apache Spark, Flink, and AWS Lambda.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/analytics-sdk-scala/

## 1. Overview

The [Snowplow Analytics SDK for Scala](https://github.com/snowplow/snowplow-scala-analytics-sdk) lets you work with [Snowplow enriched events](/docs/fundamentals/canonical-event/) in your Scala event processing, data modeling and machine-learning jobs. You can use this SDK with [Apache Spark](http://spark.apache.org/), [AWS Lambda](https://aws.amazon.com/lambda/), [Apache Flink](https://flink.apache.org/), [Scalding](https://github.com/twitter/scalding), [Apache Samza](http://samza.apache.org/) and other JVM-compatible data processing frameworks.

The Scala Analytics SDK makes it significantly easier to build applications that consume Snowplow enriched data directly from Kinesis or S3.

## 2. Compatibility

Snowplow Scala Analytics SDK was compiled against Scala versions 2.12 and 2.13. Minimum required Java Runtime is JRE8.

## 3. Setup

The latest version of Snowplow Scala Analytics SDK is 3.0.0 and it is available on Maven Central.

### 3.1 SBT

If you’re using SBT, add the following lines to your build file:

```scala
// Dependency
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-analytics-sdk" % "3.0.0"
```

Note the double percent (`%%`) between the group and artifactId. This will ensure that you get the right package for your Scala version.

### 3.2 Gradle

If you are using Gradle in your own job, then add following lines in your `build.gradle` file:

```gradle
dependencies {
  ...
  // Snowplow Scala Analytics SDK
  compile 'com.snowplowanalytics:snowplow-scala-analytics-sdk_2.12:3.0.0'
}
```

Note that you need to change `_2.12` to `_2.13` in artifactId if you're using Scala 2.13.

### 3.3 Maven

If you are using Maven in your own job, then add following lines in your `pom.xml` file:

```xml
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-scala-analytics-sdk_2.12</artifactId>
  <version>3.0.0</version>
</dependency>
```

Note that you need to change `_2.12` to `_2.13` in artifactId if you're using Scala 2.13.

## 4. Scala Analytics SDK Event Transformer

### 4.1 Overview

The Snowplow enriched event is a relatively complex TSV string containing scalars and self-describing JSONs. Rather than work with this structure directly, Snowplow analytics SDKs ship with _event transformers_, which translate the Snowplow enriched event format into other formats that are more convenient for engineers and analysts.

As the Snowplow enriched event format evolves towards a cleaner [Apache Avro](https://avro.apache.org/)-based structure, we will be updating this SDK to maintain compatibility across different enriched event versions.

Working with the Snowplow Scala Analytics SDK therefore has two major advantages over working with Snowplow enriched events directly:

1. The SDK reduces your development time by providing analyst- and developer-friendly transformations of the Snowplow enriched event format;
2. The SDK futureproofs your code against new releases of Snowplow which update our enriched event format.

Currently the Analytics SDK for Scala ships with one event transformer: the JSON Event Transformer.

### 4.2 The JSON Event Transformer

The JSON Event Transformer takes a Snowplow enriched event and converts it into a JSON ready for further processing. This transformer was adapted from the code used to load Snowplow events into Elasticsearch in the Kinesis real-time pipeline.

The JSON Event Transformer converts a Snowplow enriched event into an instance of the `Event` case class, a representation of a canonical Snowplow event, like so:

```scala
Event(
  app_id = Some("angry-birds"),
  platform = Some("web"),
  etl_tstamp = Some(Instant.parse("2017-01-26T00:01:25.292Z")),
  collector_tstamp = Instant.parse("2013-11-26T00:02:05Z"),
  dvce_created_tstamp = Some(Instant.parse("2013-11-26T00:03:57.885Z")),
  event = Some("page_view"),
  event_id = UUID.fromString("c6ef3124-b53a-4b13-a233-0088f79dcbcb"),
  txn_id = Some(41828),
  name_tracker = Some("cloudfront-1"),
  v_tracker = Some("js-2.1.0"),
  v_collector = "clj-tomcat-0.1.0",
  v_etl = "serde-0.5.2"
  /* ... */
)
```

This case class can be rendered into a JSON object, and subsequently a JSON string, or worked with to interact with the event's fields in a typesafe manner.

The most complex piece of processing is the handling of the self-describing JSONs found in the enriched event's `unstruct_event`, `contexts` and `derived_contexts` fields. Currently there are two alternative behaviors for handling them in the event transformer:

1. Under the original "lossy" behavior, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the unstructured event field would be rendered in the final JSON like this:

```json
"unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {
  "targetUrl": "http://www.example.com",
  "elementClasses": ["foreground"],
  "elementId": "exampleLink"
}
```

2. Under the new "lossless" behavior, available since 0.3.1, if an enriched event contained a `com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`, then the final JSON (if turned into a string) would contain a self-describing JSON object instead:

```json
"unstruct_event": {
  "schema": "iglu:com.snowplowanalytics.snowplow/link_click/1-0-1",
  "data": {
    "targetUrl": "http://www.example.com",
    "elementClasses": ["foreground"],
    "elementId": "exampleLink"
  }
}
```

Along with the `Event` case class, the JSON Event Transformer comes with the following functions:

- `Event.parse(line)` - similar to the old `transform` function, this method accepts an enriched Snowplow event as a string and returns an `Event` instance as a result.
- `event.toJson(lossy)` - similar to the old `getValidatedJsonEvent` function, it transforms an `Event` into a validated JSON whose keys are the field names corresponding to the EnrichedEvent POJO of the Scala Common Enrich project. If the lossy argument is true, any self-describing events in the fields (unstruct\_event, contexts and derived\_contexts) are returned in a "shredded" format, e.g. `"unstruct_event_com_acme_1_myField": "value"`. If it is set to false, they are not flattened into underscore-separated top-level fields, using a standard self-describing format instead.
- `event.inventory` - extracts metadata from the event containing information about the types and Iglu URIs of its shred properties (unstruct\_event, contexts and derived\_contexts). Unlike version 0.3.0, it no longer requires a `transformWithInventory` call and can be obtained from any `Event` instance.
- `atomic` - returns the event as a map of keys to Circe JSON values, while dropping inventory fields. This method can be used to modify an event's JSON AST before converting it into a final result.
- `ordered` - returns the event as a list of key/Circe JSON value pairs. Unlike `atomic`, which has randomized key ordering, this method returns the keys in the order of the canonical event model, and is particularly useful for working with relational databases.

#### Event inventory

An event's inventory is simply a list of metadata about its shredded types:

1. Where it was extracted from: `unstruct_event` column (`UnstructEvent`), `contexts` column (`Contexts(CustomContexts)`) or `derived_contexts` column (`Contexts(DerivedContexts)`)
2. Its Iglu URI (e.g. `iglu:com.acme/context/jsonschema/1-0-0`), stored as an Iglu `SchemaKey` instance.

### 4.3 Examples

#### 4.3.1 Using from Apache Spark

The Scala Analytics SDK is a great fit for performing Snowplow [event data modeling](http://snowplowanalytics.com/blog/2016/03/16/introduction-to-event-data-modeling/) in Apache Spark and Spark Streaming.

Here’s the code we use internally for our own data modeling jobs:

```scala
import cats.data.Validated
import com.snowplowanalytics.snowplow.analytics.scalasdk.Event

val events = input.flatMap(line =>
  Event.parse(line) match {
    case Validated.Valid(event) => Some(event.toJson(true).noSpaces)
    case Validated.Invalid(_) => None
  }
)

val dataframe = spark.read.json(events: _*)
```

#### 4.3.2 Using from AWS Lambda

The Scala Analytics SDK is a great fit for performing analytics-on-write, monitoring or alerting on Snowplow event streams using AWS Lambda.

Here’s some sample code for transforming enriched events into JSON inside a Scala Lambda:

```scala
import com.snowplowanalytics.snowplow.analytics.scalasdk.Event

def recordHandler(event: KinesisEvent) {

  val events = for {
    rec <- event.getRecords
    line = new String(rec.getKinesis.getData.array())
    event = Event.parse(line)
  } yield event

  /* ... */
}
```

---

# Analytics SDKs for event data transformation
> Transform Snowplow enriched TSV events into JSON for data modeling and machine learning in Scala, JavaScript, Go, Python, and .NET.
> Source: https://docs.snowplow.io/docs/api-reference/analytics-sdk/

The Snowplow Analytics SDKs are designed for data engineers and data scientists working with Snowplow in a number of languages.

Some good use cases for the SDKs include:

1. Transforming the Enriched TSV to Enriched JSON for further processing
2. Developing AI/ML models on your event data
3. Performing analytics-on-write in AWS Lambda as part of our Kinesis real-time pipeline
4. Within Snowplow pipeline components to process event data

## Snowplow Analytics SDKs

- [Scala Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-scala/) - lets you work with Snowplow enriched events in your Scala event processing, data modeling and machine-learning jobs. You can use this SDK with Apache Spark, AWS Lambda, GCP Cloud Functions, Apache Flink and other Scala-compatible data processing frameworks.
- [JavaScript and TypeScript Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-javascript/) - lets you work with Snowplow enriched events in your Node.js or other JavaScript environments. This SDK can be used with AWS Lambda and Google Cloud Functions.
- [Go Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/) - lets you work with Snowplow enriched events in your Go environments. This SDK can be used with AWS Lambda and Google Cloud Functions.
- [Python Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-python/) - lets you work with Snowplow enriched events in your Python event processing, data modeling and machine-learning jobs. You can use this SDK with Apache Spark, AWS Lambda, GCP Cloud Functions and other Python-compatible data processing frameworks.
- [.NET Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-net/) - lets you work with Snowplow enriched events in your .NET event processing, data modeling and machine-learning jobs. You can use this SDK with Azure Data Lake Analytics, Azure Function, AWS Lambda, GCP Cloud Functions other .NET-compatible data processing frameworks.

---

# Dataflow Runner for AWS EMR clusters
> CLI tool for launching and managing AWS EMR clusters with templated playbooks for Hadoop and Spark jobs with distributed locking support.
> Source: https://docs.snowplow.io/docs/api-reference/dataflow-runner/

Dataflow Runner is a system for creating and running [AWS EMR](https://aws.amazon.com/emr/) jobflow clusters and steps. It uses templated playbooks to define your cluster, and the Hadoop/Spark/et al jobs that you want to run.

### Installation

- Platform native binaries are available from the GitHub releases [page](https://github.com/snowplow/dataflow-runner/releases).
- Docker images are available at [DockerHub](https://hub.docker.com/r/snowplow/dataflow-runner) as of version `0.7.3`.

### Cluster Configuration

A cluster configuration contains all of the information needed to create a new cluster which is ready to accept a playbook. Currently AWS EMR is the only supported data-flow fabric.

For the cluster template see: [config/cluster.json.sample](https://github.com/snowplow/dataflow-runner/blob/master/config/cluster.json.sample)

### Playbook Configuration

A playbook consists of one of more _steps_. Steps are added to the cluster and run in series.

For the playbook template see: [config/playbook.json.sample](https://github.com/snowplow/dataflow-runner/blob/master/config/playbook.json.sample)

### Templates

Configuration files are run through Golang’s [text template processor](http://golang.org/pkg/text/template/). The template processor can access all _variables_ defined on the command line using the `--vars` argument.

For example to use the `--vars` argument with a playbook step:

```json
{
  "type": "CUSTOM_JAR",
  "name": "Combine Months",
  "actionOnFailure": "CANCEL_AND_WAIT",
  "jar": "s3://snowplow-hosted-assets/3-enrich/hadoop-event-recovery/snowplow-hadoop-event-recovery-0.2.0.jar",
  "arguments": [
    "com.snowplowanalytics.hadoop.scalding.SnowplowEventRecoveryJob",
    "--hdfs",
    "--input",
    "hdfs:///local/monthly/{{.inputVariable}}",
    "--output",
    "hdfs:///local/recovery/{{.outputVariable}}"
  ]
}
```

You would then pass the following command:

```bash
host> ./dataflow-runner run --emr-playbook ${emr-playbook-path} --emr-cluster j-2DPBXD87LSGP9 --vars inputVariable,input,outputVariable,output
```

This would resolve to:

```json
{
  "type": "CUSTOM_JAR",
  "name": "Combine Months",
  "actionOnFailure": "CANCEL_AND_WAIT",
  "jar": "s3://snowplow-hosted-assets/3-enrich/hadoop-event-recovery/snowplow-hadoop-event-recovery-0.2.0.jar",
  "arguments": [
    "com.snowplowanalytics.hadoop.scalding.SnowplowEventRecoveryJob",
    "--hdfs",
    "--input",
    "hdfs:///local/monthly/input",
    "--output",
    "hdfs:///local/recovery/output"
  ]
}
```

The following custom functions are also supported:

- `nowWithFormat [timeFormat]`: where `timeFormat` is a valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `timeWithFormat [epoch] [timeFormat]`: where `epoch` is the number of seconds elapsed between January 1st 1970 and a certain point in time as a string and `timeFormat` is valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `systemEnv "ENV_VAR"`: where `ENV_VAR` is a key for a valid environment variable
- `base64 [string]`: will base64-encode the string passed as argument
- `base64File "path/to/file.txt"`: will base64-encode the content of the file located at the path passed as argument

### CLI Commands

There are several commands that can be used to manage your data-flow fabric.

#### `up`: Launches a new EMR cluster

```text
NAME:
   dataflow-runner up - Launches a new EMR cluster

USAGE:
   dataflow-runner up [command options] [arguments...]

OPTIONS:
   --emr-config value  EMR config path
   --vars value        Variables that will be used by the templater
```

This command will launch a new cluster ready for step execution, the output should look something like the following:

```text
NAME:
   dataflow-runner run - Adds jobflow steps to a running EMR cluster

USAGE:
   dataflow-runner run [command options] [arguments...]

OPTIONS:
   --emr-playbook value  Playbook path
   --emr-cluster value   Jobflow ID
   --async               Asynchronous execution of the jobflow steps
   --lock value          Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag.
   --softLock value      Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag. Released no matter if the operation failed or succeeded.
   --consul value        Address of the Consul server used for distributed locking
   --vars value          Variables that will be used by the templater
```

#### `run`: Adds jobflow steps to a running EMR cluster

```bash
host> ./dataflow-runner up --emr-config ${emr-config-path}
INFO[0001] Launching EMR cluster with name 'dataflow-runner - sample name'...
INFO[0001] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...
INFO[0021] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...
# this goes for a few lines, omitted for brevity
INFO[0227] EMR cluster is in state STARTING - need state WAITING, checking again in 20 seconds...
INFO[0248] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...
INFO[0269] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...
INFO[0289] EMR cluster is in state BOOTSTRAPPING - need state WAITING, checking again in 20 seconds...
INFO[0310] EMR cluster launched successfully; Jobflow ID: j-2DPBXD87LSGP9
```

This command adds new steps to the already running cluster. By default this command is blocking - however if you wish to submit and forget simply supply the `--async` argument, the output should look something like the following:

```bash
host> ./dataflow-runner run --emr-playbook ${emr-playbook-path} --emr-cluster j-2DPBXD87LSGP9
INFO[0310] Successfully added 2 steps to the EMR cluster with jobflow id 'j-2DPBXD87LSGP9'...
ERRO[0357] Step 'Combine Months' with id 's-9WZ0VFKC770J' was FAILED
ERRO[0358] Step 'Combine Months 2' with id 's-37F9PKSXBHDAU' was CANCELLED
ERRO[0358] 2/2 steps failed to complete successfully
```

In this case the first step failed which meant that the second step was cancelled. This behavior is dependent on your `actionOnFailure` - you can choose either to:

1. “CANCEL\_AND\_WAIT”: This will cancel all other currently queued jobs and return the cluster to a waiting state ready for new job submissions.
2. “CONTINUE”: This will go to the next step regardless if it failed or not.

**Note**: We have removed the ability to terminate the job flow on failure, to terminate you will need to use the `down` command.

Additionally, Dataflow Runner can acquire a lock before starting the job which can prevent other jobs from running at the same time. Its release will happen when:

- the job has terminated (whether successfully or with failure) with the `--softLock` flag
- the job has succeeded with the `--lock` flag (“hard lock”)

As the above implies, if a job were to fail and the `--lock` flag was used, manual cleaning of the lock will be required.

Additionally, supplying a [Consul](https://www.consul.io/) address, through the `--consul` flag will make this lock distributed.

When the `--consul` flag is used, the lock will be materialized by a key-value pair in Consul for which the key is the value supplied with the `--lock` or `--softLock` argument. Otherwise, it will be materialized by a file on the machine located at the specified path (either relative to your working directory or absolute).

#### `down`: Terminates a running EMR cluster

```text
NAME:
   dataflow-runner down - Terminates a running EMR cluster

USAGE:
   dataflow-runner down [command options] [arguments...]

OPTIONS:
   --emr-config value   EMR config path
   --emr-cluster value  Jobflow ID
   --vars value         Variables that will be used by the templater
```

When you are done with the EMR cluster you can terminate it by using the `down` command. This takes the original emr configuration and the job flow id to then go and terminate the cluster, the output should look something like the following:

```bash
host> ./dataflow-runner down --emr-config ${emr-config-path} --emr-cluster j-2DPBXD87LSGP9
INFO[0358] Terminating EMR cluster with jobflow id 'j-2DPBXD87LSGP9'...
INFO[0358] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...
INFO[0378] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...
INFO[0399] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...
INFO[0420] EMR cluster is in state TERMINATING - need state TERMINATED, checking again in 20 seconds...
INFO[0440] Transient EMR run completed successfully
```

#### `run-transient`: Launches, runs and then terminates an EMR cluster

```text
NAME:
   dataflow-runner run-transient - Launches, runs and then terminates an EMR cluster

USAGE:
   dataflow-runner run-transient [command options] [arguments...]

OPTIONS:
   --emr-config value    EMR config path
   --emr-playbook value  Playbook path
   --lock value          Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag.
   --softLock value      Path to the lock held for the duration of the jobflow steps. This is materialized by a file or a KV entry in Consul depending on the --consul flag. Released no matter if the operation failed or succeeded.
   --consul value        Address of the Consul server used for distributed locking
   --vars value          Variables that will be used by the templater
```

This command is a combination of `up`, `run` and `down` which is designed to mimic the current `EmrEtlRunner` behavior.

---

# Enrich configuration reference
> Complete HOCON configuration reference for Snowplow Enrich applications including monitoring, validation, and stream-specific settings.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/configuration-reference/

This page lists the configuration options for Enrich applications.

## License

Enrich is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Enrich, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```json
"license": {
  "accept": true
}
```

## Common parameters

| parameter                                                 | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpuParallelismFraction` (since _6.0.0_)                  | Optional. Default: `1`. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and cpuParallelismFactor = 0.75, then we process 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                         |
| `sinkParallelismFraction` (since _6.0.0_)                 | Optional. Default: `2`. Controls number of sink job that can be run in parallel. E.g. If there are 4 available processors, and sinkParallelismFraction = 2, then we run 8 sink job concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                                                                                                                                                                                                  |
| `assetsUpdatePeriod`                                      | Optional. E.g. `7 days`. Period after which enrich assets (e.g. the maxmind database for the IpLookups enrichment) should be checked for updates. Assets will never be updated if this key is missing.                                                                                                                                                                                                                                                                             |
| `monitoring.sentry.dsn`                                   | Optional. E.g. `http://sentry.acme.com`. To track uncaught runtime exceptions in Sentry.                                                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.environment` (since _6.7.0_)           | Optional. Environment name to use when reporting exceptions in Sentry.                                                                                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.sentry.tags.*`                                | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.hostname`                      | Optional. E.g. `localhost`. Hostname of the StatsD server to send enrichment metrics (latency and event counts) to.                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.port`                          | Optional. E.g. `8125`. Port of the StatsD server.                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.period`                        | Optional. E.g. `10 seconds`. How frequently to send metrics to StatsD server.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.tags`                          | Optional. E.g. `{ "env": "prod" }`. Key-value pairs attached to each metric sent to StatsD to provide contextual information.                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.prefix`                        | Optional. Default: `snowplow.enrich`. Pefix of StatsD metric names.                                                                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.healthProbe.port` (since _6.0.0_)             | Optional. Default: `8000`. Open a HTTP server that returns OK only if the app is healthy.                                                                                                                                                                                                                                                                                                                                                                                          |
| `monitoring.healthProbe.unhealthyLatency` (since _6.0.0_) | Optional. Default: `2 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time.                                                                                                                                                                                                                                                                                                                                         |
| `telemetry.disable`                                       | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                                                                          |
| `telemetry.userProvidedId`                                | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                                                                |
| `validation.acceptInvalid` (since _6.0.0_)                | Optional. Default: `false`. Enrich _6.0.0_ introduces the validation of the enriched events against atomic schema before emitting. If set to `false`, a failed event will be emitted instead of the enriched event if validation fails. If set to `true`, invalid enriched events will be emitted, as before.                                                                                                                                                                      |
| `validation.atomicFieldsLimits` (since _4.0.0_)           | Optional. For the defaults, see [here](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/resources/reference.conf). Configuration for custom maximum atomic fields (strings) length. It's a map-like structure with keys being atomic field names and values being their max allowed length.                                                                                                                                                                  |
| `validation.maxJsonDepth` (since _6.0.0_)                 | Optional. Default: `40`. Maximum allowed depth for the JSON entities in the events. Event will be sent to bad row stream if it contains JSON entity with a depth that exceeds this value.                                                                                                                                                                                                                                                                                          |
| `validation.exitOnJsCompileError` (since _6.0.0_)         | Optional. Default: `true`. If it is set to true, Enrich will exit with error if JS enrichment script is invalid. If it is set to false, Enrich will continue to run if JS enrichment script is invalid but every event will end up as bad row.                                                                                                                                                                                                                                     |
| `decompression.maxBytesInBatch` (since _6.1.0_)           | Optional. Default: `10000000` (10MB). Although a compressed message from the Collector is limited to 1MB, it could become several times bigger after decompression. To avoid loading an enormous amount of data into memory, Enrich will decompress the message in portions (batches). This parameter specifies the maximum size of such a batch. As soon as the decompressed batch reaches `maxBytesInBatch`, it is a emitted for further processing, and a new batch is started. |
| `decompression.maxBytesSinglePayload` (since _6.1.0_)     | Optional. Default: `10000000` (10 MB). Each compressed Collector message contains a number of payloads, which contain one or more events. While the Collector already enforces some payload size limits, this setting exists as a safety check to prevent Enrich from loading large amounts of data into memory. Specifically, if an individual payload exceeds `maxBytesSinglePayload`, it will result in a [size violation](/docs/api-reference/failed-events/#size-violation).  |
| `http.client.requestTimeout` (since _6.5.0_)              | Optional. Default: `5 seconds`. Timeout for internal HTTP requests used by Iglu resolver, alerts, telemetry, and metadata endpoints.                                                                                                                                                                                                                                                                                                                                               |
| `iglu.maxRetry` (since _6.5.0_)                           | Optional. Default: `2`. Maximum number of retries for failed Iglu requests. Lower values allow Enrich to fail faster when Iglu Server is unavailable, falling back to cached schemas instead of blocking on retries.                                                                                                                                                                                                                                                               |
| `iglu.maxWait` (since _6.5.0_)                            | Optional. Default: `1 second`. Maximum wait time for exponential backoff between Iglu request retries.                                                                                                                                                                                                                                                                                                                                                                             |
| `jsAllowedJavaClasses` (since _6.6.0_)                    | Optional. Default: `["*"]`. List of Java classes that the [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/) is allowed to access. This affects both `new <class>` and `<class>.<static method>` usage. Examples: `["java.lang.String", "java.net.URL"]`, `["java.net.*"]`. By default, all classes are allowed.                                                                                                              |

## enrich-pubsub

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.pubsub.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.pubsub.reference.hocon).

| parameter                                                        | description                                                                                                                                                                                                                                                                                                                                                      |
| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                             | Required. E.g. `projects/example-project/subscriptions/collectorPayloads`. PubSub subscription identifier for the collector payloads.                                                                                                                                                                                                                            |
| `input.durationPerAckExtension`                                  | Optional. Default: `15 seconds`. Pubsub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                                |
| `input.minRemainingAckDeadline`                                  | Optional. Default: `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the Source will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.retries.transientErrors.delay` (since _6.2.0_)            | Optional. Default: `100 millis`. Backoff delay for follow-up attempts of transient GRPC failures retries.                                                                                                                                                                                                                                                        |
| `input.retries.transientErrors.attempts` (since _6.2.0_)         | Optional. Default: `10`. Max number of attempts for transient GRPC failures retries.                                                                                                                                                                                                                                                                             |
| `output.good.topic`                                              | Required. E.g. `projects/example-project/topics/enriched`. Name of the PubSub topic that will receive the enriched events.                                                                                                                                                                                                                                       |
| `output.good.attributes`                                         | Optional. Enriched event fields to add as PubSub message attributes. For example, if this is `[ "app_id" ]` then the enriched event's `app_id` field will be an attribute of the PubSub message, as well as being a field within the enriched event.                                                                                                             |
| `output.good.batchSize`                                          | Optional. Default: `100`. Enriched events are sent to pubsub in batches not exceeding this size.                                                                                                                                                                                                                                                                 |
| `output.good.requestByteThreshold`                               | Optional. Default: `1000000`. Enriched events are sent to pubsub in batches not exceeding this size number of bytes.                                                                                                                                                                                                                                             |
| `output.good.retries.transientErrors.delay` (since _6.2.0_)      | Same as `input.retries.transientErrors.delay` for good events.                                                                                                                                                                                                                                                                                                   |
| `output.good.retries.transientErrors.attempts` (since _6.2.0_)   | Same as `input.retries.transientErrors.attempts` for good events.                                                                                                                                                                                                                                                                                                |
| `output.failed.topic`                                            | Required. E.g. `projects/example-project/topics/failed`. Name of the PubSub topic that will receive the failed events (same format as the enriched events).                                                                                                                                                                                                      |
| `output.failed.batchSize`                                        | Same as `output.good.batchSize` for failed events.                                                                                                                                                                                                                                                                                                               |
| `output.failed.requestByteThreshold`                             | Same as `output.good.requestByteThreshold` for failed events.                                                                                                                                                                                                                                                                                                    |
| `output.failed.retries.transientErrors.delay` (since _6.2.0_)    | Same as `input.retries.transientErrors.delay` for failed events.                                                                                                                                                                                                                                                                                                 |
| `output.failed.retries.transientErrors.attempts` (since _6.2.0_) | Same as `input.retries.transientErrors.attempts` for failed events.                                                                                                                                                                                                                                                                                              |
| `output.bad.topic`                                               | Required. E.g. `projects/example-project/topics/bad`. Name of the PubSub topic that will receive the failed events in the "bad row" format (JSON).                                                                                                                                                                                                               |
| `output.bad.batchSize`                                           | Same as `output.good.batchSize` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                                |
| `output.bad.requestByteThreshold`                                | Same as `output.good.requestByteThreshold` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                     |
| `output.bad.retries.transientErrors.delay` (since _6.2.0_)       | Same as `input.retries.transientErrors.delay` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                                  |
| `output.bad.retries.transientErrors.attempts` (since _6.2.0_)    | Same as `input.retries.transientErrors.attempts` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                               |

## enrich-kinesis

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.kinesis.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.kinesis.reference.hocon).

| parameter                                                           | description                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.appName`                                                     | Optional. Default: `snowplow-enrich-kinesis`. Name of the application which the KCL daemon should assume. A DynamoDB table with this name will be created.                                                                                                                                                                                    |
| `input.streamName`                                                  | Required. E.g. `raw`. Name of the Kinesis stream with the collector payloads to read from.                                                                                                                                                                                                                                                    |
| `input.initialPosition.type`                                        | Optional. Default: `TRIM_HORIZON`. Set the initial position to consume the Kinesis stream. Possible values: `LATEST` (most recent data), `TRIM_HORIZON` (oldest available data), `AT_TIMESTAMP` (start from the record at or after the specified timestamp).                                                                                  |
| `input.initialPosition.timestamp`                                   | Required for `AT_TIMESTAMP`. E.g. `2020-07-17T10:00:00Z`.                                                                                                                                                                                                                                                                                     |
| `input.retrievalMode.type`                                          | Optional. Default: `Polling`. Set the mode for retrieving records. Possible values: `Polling` or `FanOut`.                                                                                                                                                                                                                                    |
| `input.retrievalMode.maxRecords`                                    | Required for `Polling`. Default: `1000`. Maximum size of a batch returned by a call to `getRecords`. Records are checkpointed after a batch has been fully processed, thus the smaller `maxRecords`, the more often records can be checkpointed into DynamoDb, but possibly reducing the throughput.                                          |
| `input.retrievalMode.idleTimeBetweenReads`                          | Optional for `Polling`. Default: `200 millis`. Idle time between `getRecords` requests.                                                                                                                                                                                                                                                       |
| `input.workerIdentifier` (since _6.0.0_)                            | Required. Name of this KCL worker used in the DynamoDB lease table.                                                                                                                                                                                                                                                                           |
| `input.leaseDuration` (since _6.0.0_)                               | Optional. Default: `10 seconds`. Duration of shard leases. KCL workers must periodically refresh leases in the DynamoDB table before this duration expires.                                                                                                                                                                                   |
| `input.maxLeasesToStealAtOneTimeFactor` (since _6.0.0_)             | Optional. Default: `2.0`. Controls how to pick the max number of leases to steal at one time. E.g. If there are 4 available processors, and maxLeasesToStealAtOneTimeFactor = 2.0, then allow the KCL to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency.                  |
| `input.checkpointThrottledBackoffPolicy.minBackoff` (since _6.0.0_) | Optional. Default: `100 millis`. Minimum backoff before retrying when DynamoDB provisioned throughput exceeded.                                                                                                                                                                                                                               |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` (since _6.0.0_) | Optional. Default: `1 second`. Maximum backoff before retrying when DynamoDB provisioned throughput limit exceeded.                                                                                                                                                                                                                           |
| `input.debounceCheckpoints` (since _6.0.0_)                         | Optional. Default: `10 seconds`. How frequently to checkpoint our progress to the DynamoDB table. By increasing this value, we can decrease the write-throughput requirements of the DynamoDB table.                                                                                                                                          |
| `input.apiCallAttemptTimeout` (since _6.6.0_)                       | Optional. Default: `15 seconds`. Timeout for API call attempts to Kinesis, DynamoDB, and CloudWatch.                                                                                                                                                                                                                                          |
| `output.good.streamName`                                            | Required. E.g. `enriched`. Name of the Kinesis stream to write to the enriched events.                                                                                                                                                                                                                                                        |
| `output.good.partitionKey`                                          | Optional. How the output stream will be partitioned in Kinesis. Events with the same partition key value will go to the same shard. Possible values: `event_id`, `event_fingerprint`, `domain_userid`, `network_userid`, `user_ipaddress`, `domain_sessionid`, `user_fingerprint`. If not specified, the partition key will be a random UUID. |
| `output.good.throttledBackoffPolicy.minBackoff` (since _6.0.0_)     | Optional. Default: `100 milliseconds`. Minimum backoff before retrying when writing fails with exceeded kinesis write throughput.                                                                                                                                                                                                             |
| `output.good.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)     | Optional. Default: `1 second`. Maximum backoff before retrying when writing fails with exceeded kinesis write throughput.                                                                                                                                                                                                                     |
| `output.good.recordLimit`                                           | Optional. Default: `500`. Maximum allowed to records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                               |
| `output.good.byteLimit`                                             | Optional. Default: `5242880`. Maximum allowed to bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                             |
| `output.good.maxRetries` (since _6.3.0_)                            | Optional. Default: `10`. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                         |
| `output.failed.streamName`                                          | Required. E.g. `failed`. Name of the Kinesis stream that will receive the failed events (same format as the enriched events).                                                                                                                                                                                                                 |
| `output.failed.throttledBackoffPolicy.minBackoff` (since _6.0.0_)   | Same as `output.good.throttledBackoffPolicy.minBackoff` for failed events.                                                                                                                                                                                                                                                                    |
| `output.failed.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)   | Same as `output.good.throttledBackoffPolicy.maxBackoff` for failed events.                                                                                                                                                                                                                                                                    |
| `output.failed.recordLimit`                                         | Same as `output.good.recordLimit` for failed events.                                                                                                                                                                                                                                                                                          |
| `output.failed.byteLimit`                                           | Same as `output.good.byteLimit` for failed events.                                                                                                                                                                                                                                                                                            |
| `output.failed.maxRetries` (since _6.3.0_)                          | Same as `output.good.maxRetries` for failed events.                                                                                                                                                                                                                                                                                           |
| `output.bad.streamName`                                             | Required. E.g. `bad`. Name of the Kinesis stream that will receive the failed events in the "bad row" format (JSON).                                                                                                                                                                                                                          |
| `output.bad.throttledBackoffPolicy.minBackoff` (since _6.0.0_)      | Same as `output.good.throttledBackoffPolicy.minBackoff` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                     |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since _6.0.0_)      | Same as `output.good.throttledBackoffPolicy.maxBackoff` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                     |
| `output.bad.recordLimit`                                            | Same as `output.good.recordLimit` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                           |
| `output.bad.byteLimit`                                              | Same as `output.good.byteLimit` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                             |
| `output.bad.maxRetries` (since _6.3.0_)                             | Same as `output.good.maxRetries` for failed events in the "bad row" format (JSON).                                                                                                                                                                                                                                                            |

## enrich-kafka

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.kafka.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.kafka.reference.hocon).

| parameter                                       | description                                                                                                                                                                               |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                               | Required. Name of the Kafka topic to read collector payloads from.                                                                                                                        |
| `input.bootstrapServers`                        | Required. A list of `host:port` pairs to use for establishing the initial connection to the Kafka cluster                                                                                 |
| `input.debounceCommitOffsets` (since _6.0.0_)   | Optional. Default: `10 seconds`. How frequently to commit our progress back to kafka. By increasing this value, we decrease the number of requests made to the kafka broker.              |
| `input.commitTimeout` (since _6.3.0_)           | Optional. Default: `15 seconds`. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `input.consumerConf`                            | Optional. Kafka consumer configuration. See [the docs](https://kafka.apache.org/documentation/#consumerconfigs) for all properties.                                                       |
| `output.good.topicName`                         | Required. Name of the Kafka topic to write to                                                                                                                                             |
| `output.good.bootstrapServers`                  | Required. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                                   |
| `output.good.producerConf`                      | Optional. Kafka producer configuration. See [the docs](https://kafka.apache.org/documentation/#producerconfigs) for all properties                                                        |
| `output.good.partitionKey`                      | Optional. Enriched event field to use as Kafka partition key                                                                                                                              |
| `output.good.attributes`                        | Optional. Enriched event fields to add as Kafka record headers                                                                                                                            |
| `output.failed.topicName`                       | Optional. Name of the Kafka topic that will receive the failed events (same format as the enriched events)                                                                                |
| `output.failed.bootstrapServers`                | Same as `output.good.bootstrapServers` for failed events.                                                                                                                                 |
| `output.failed.producerConf`                    | Same as `output.good.producerConf` for failed events.                                                                                                                                     |
| `output.bad.topicName`                          | Optional. Name of the Kafka topic that will receive the failed events in the “bad row” format (JSON)                                                                                      |
| `output.bad.bootstrapServers`                   | Same as `output.good.bootstrapServers` for failed events in the "bad row" format (JSON).                                                                                                  |
| `output.bad.producerConf`                       | Same as `output.good.producerConf` for failed events in the "bad row" format (JSON).                                                                                                      |
| `blobClients.accounts` (since _6.0.0_)          | Optional. Array of Azure Blob Storage accounts to download enrichment assets.                                                                                                             |
| Example values for the Azure storage accounts : |                                                                                                                                                                                           |

- `{ "name": "storageAccount1"}`: public account with no auth
- `{ "name": "storageAccount2", "auth": { "type": "default"} }`: private account using default auth chain
- `{ "name": "storageAccount3", "auth": { "type": "sas", "value": "tokenValue"}}`: private account using SAS token auth

## enrich-nsq

A minimal configuration file can be found on the [Github repo](https://github.com/snowplow/enrich/blob/master/config/config.nsq.minimal.hocon), as well as a [comprehensive one](https://github.com/snowplow/enrich/blob/master/config/config.nsq.reference.hocon).

| parameter                              | description                                                                                                   |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `input.topic`                          | Required. Name of the NSQ topic with the collector payloads.                                                  |
| `input.lookupHost`                     | Required. The host name of NSQ lookup application.                                                            |
| `input.lookupPort`                     | Required. The port number of NSQ lookup application.                                                          |
| `input.channel`                        | Optional. Default: `collector-payloads-channel`. Name of the NSQ channel used to retrieve collector payloads. |
| `output.good.topic`                    | Required. Name of the NSQ topic that will receive the enriched events.                                        |
| `output.good.nsqdHost`                 | Required. The host name of nsqd application.                                                                  |
| `output.good.nsqdPort`                 | Required. The port number of nsqd application.                                                                |
| `output.failed.topic`                  | Required. Name of the NSQ topic that will receive the failed events (same format as the enriched events).     |
| `output.failed.nsqdHost`               | Required. The host name of nsqd application.                                                                  |
| `output.failed.nsqdPort`               | Required. The port number of nsqd application.                                                                |
| `output.bad.topic`                     | Required. Name of the NSQ topic that will receive the failed events in the "bad row" format (JSON).           |
| `output.bad.nsqdHost`                  | Required. The host name of nsqd application.                                                                  |
| `output.bad.nsqdPort`                  | Required. The port number of nsqd application.                                                                |
| `blobClients.accounts` (since _6.0.0_) | Optional. Array of Azure Blob Storage accounts to download enrichment assets.                                 |

## Enriched events validation against atomic schema

Enriched events are expected to match [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) schema. However, until `3.0.0`, it was never checked that the enriched events emitted by enrich were valid. If an event is not valid against `atomic` schema, a [failed event](/docs/fundamentals/failed-events/) should be emitted instead of the enriched event. However, this is a breaking change, and we want to give some time to users to adapt, in case today they are working downstream with enriched events that are not valid against `atomic`. For this reason, this new validation was added as a feature that can be deactivated like that:

```json
"validation": {
  "acceptInvalid": true
}
```

In this case, enriched events that are not valid against `atomic` schema will still be emitted as before, so that Enrich `3.0.0` can be fully backward compatible. It will be possible to know if the new validation would have had an impact by 2 ways:

1. A new metric `invalid_enriched` has been introduced. It reports the number of enriched events that were not valid against `atomic` schema. As the other metrics, it can be seen on stdout and/or StatsD.
2. Each time there is an enriched event invalid against `atomic` schema, a line will be logged with the failed event (add `-Dorg.slf4j.simpleLogger.log.InvalidEnriched=debug` to the `JAVA_OPTS` to see it).

If `acceptInvalid` is set to `false`, a failed event will be emitted instead of the enriched event in case it's not valid against `atomic` schema.

When we'll know that all our customers don't have any invalid enriched events any more, we'll remove the feature flags and it will be impossible to emit invalid enriched events.

Since `4.0.0`, it is possible to configure the lengths of the atomic fields, below is an example:

```hcl
{
  ...
  # Optional. Configuration section for various validation-oriented settings.
  "validation": {
    # Optional. Configuration for custom maximum atomic fields (strings) length.
    # Map-like structure with keys being field names and values being their max allowed length
    "atomicFieldsLimits": {
        "app_id": 5
        "mkt_clickid": 100000
        # ...and any other 'atomic' field with custom limit
    }
  }
}
```

## Enrichments

The list of the enrichments that can be configured can be found on [this page](/docs/pipeline/enrichments/available-enrichments/).

---

# Enrich Kafka for Azure deployments
> Standalone JVM application for enriching Snowplow events from Kafka topics with configurable enrichments and validation for Azure deployments.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-kafka/

`enrich-kafka` is a standalone JVM application that reads from and writes to Kafka. It can be run from anywhere, as long as it can communicate with your Kafka cluster.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \
-it --rm \
-v $PWD:/snowplow \
snowplow/snowplow-enrich-kafka:6.9.0 \
--enrichments /snowplow/enrichments \
--iglu-config /snowplow/resolver.json \
--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
2. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
3. [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich Kafka (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich Kinesis for AWS streams
> Standalone JVM application for enriching Snowplow events from AWS Kinesis streams with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-kinesis/

`enrich-kinesis` is a standalone JVM application that reads from and writes to Kinesis streams. It can be run from anywhere, as long as it has permissions to access the streams.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \
-it --rm \
-v $PWD:/snowplow \
-e AWS_ACCESS_KEY_ID=xxx \
-e AWS_SECRET_ACCESS_KEY=xxx \
snowplow/snowplow-enrich-kinesis:6.9.0 \
--enrichments /snowplow/enrichments \
--iglu-config /snowplow/resolver.json \
--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

- `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
- Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
- [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Depending on where the app runs, `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` might not be required.

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich Kinesis (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich NSQ for cloud-agnostic applications
> Cloud-agnostic standalone JVM application for enriching Snowplow events from NSQ with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-nsq/

`enrich-nsq` is a standalone JVM application that reads from and writes to NSQ. It can be run from anywhere, as long as it can communicate with your NSQ cluster.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \
-it --rm \
-v $PWD:/snowplow \
snowplow/snowplow-enrich-nsq:6.9.0 \
--enrichments /snowplow/enrichments \
--iglu-config /snowplow/resolver.json \
--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
2. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
3. [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above.

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich NSQ (since version 3.8.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich PubSub for Google Cloud
> Standalone JVM application for enriching Snowplow events from Google Cloud PubSub with configurable enrichments and validation.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/enrich-pubsub/

`enrich-pubsub` is a standalone JVM application that reads from and writes to PubSub topics. It can be run from anywhere, as long as it has permissions to access the topics.

It is published on Docker Hub and can be run with the following command:

```bash
docker run \
-it --rm \
-v $PWD:/snowplow \
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/snowplow-gcp-account-11aa55ff6b1b.json \
snowplow/snowplow-enrich-pubsub:6.9.0 \
--enrichments /snowplow/enrichments \
--iglu-config /snowplow/resolver.json \
--config /snowplow/config.hocon
```

Above assumes that you have following directory structure:

1. GCP credentials [JSON file](https://cloud.google.com/docs/authentication/getting-started)
2. `enrichments` directory, (possibly empty) with all [enrichment configuration JSONs](/docs/pipeline/enrichments/available-enrichments/)
3. Iglu Resolver [configuration JSON](/docs/api-reference/iglu/iglu-resolver/)
4. enrich-pubSub [configuration HOCON](/docs/api-reference/enrichment-components/configuration-reference/)

It is possible to use environment variables in all of the above (for Iglu and enrichments starting from `3.7.0` only).

Configuration guide can be found on [this page](/docs/api-reference/enrichment-components/configuration-reference/) and information about the monitoring on [this one](/docs/api-reference/enrichment-components/monitoring/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Enrich PubSub (since version 3.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Enrich applications for different platforms
> Technical documentation for Snowplow enrichment applications that validate and enrich collector payloads on Kinesis, PubSub, Kafka, and NSQ.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/

This is the technical documentation for the enrichment step. If you are not familiar yet with this step of the pipeline, please refer to [this page](/docs/pipeline/enrichments/).

Here is the list of the enrichment assets:

## [enrich-kinesis](/docs/api-reference/enrichment-components/enrich-kinesis/) (AWS)

Standalone JVM application that reads collector payloads events from a Kinesis stream and outputs back to Kinesis.

## [enrich-pubsub](/docs/api-reference/enrichment-components/enrich-pubsub/) (GCP)

Standalone JVM application that reads collector payloads from a PubSub subscription and outputs back to PubSub.

## [enrich-kafka](/docs/api-reference/enrichment-components/enrich-kafka/) (Azure)

Standalone JVM application that reads collector payloads from a Kafka topic and outputs back to Kafka.

## [enrich-nsq](/docs/api-reference/enrichment-components/enrich-nsq/) (cloud agnostic)

Standalone JVM application that reads collector payloads from NSQ and outputs back to NSQ.

---

# Monitoring in Enrich applications
> Monitor Snowplow Enrich applications with StatsD metrics for event counts, latency tracking, and health probes.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/monitoring/

Enrich app has monitoring built in, to help the pipeline operator.

## Statsd

[Statsd](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

Enrich can periodically emit event-based metrics to a statsd daemon. Here is a string representation of the metrics it sends:

```text
snowplow.enrich.raw:42|c|#tag1:value1
snowplow.enrich.good:30|c|#tag1:value1
snowplow.enrich.failed:10|c|#tag1:value1
snowplow.enrich.bad:12|c|#tag1:value1
snowplow.enrich.e2e_latency_millis:123.4|g|#tag1:value1
snowplow.enrich.latency_millis:123.4|g|#tag1:value1
snowplow.enrich.invalid_enriched:0|c|#tag1:value1
```

- `raw`: total number of raw collector payloads received.
- `good`: total number of good events successfully enriched.
- `failed`(`incomplete` before version _6.0.0_): total number of failed events due to schema violations or enrichment failures (if feature is enabled).
- `bad`: total number of failed events, e.g. due to schema violations, invalid collector payload, or an enrichment failure.
- `e2e_latency_millis`(`latency` before version _6.0.0_): time difference between the collector timestamp and time the event is emitted to the output stream
- `latency_millis` (since _6.0.0_): delay between the input record getting written to the stream and Enrich starting to process it
- `invalid_enriched`: number of enriched events that were not valid against [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) schema

Note, the count metrics (`raw`, `good`, `bad` and `invalid_enriched`) refer to the updated count since the previous metric was emitted. A collector payload can carry multiple events, so it is possible for `good` to be larger than `raw`.

The latency metrics (`e2e_latency_millis` and `latency_millis`) refer to the maximum latency of all events since the previous metric was emitted.

Statsd monitoring is configured by setting the `monitoring.metrics.statsd` section in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/):

```json
"monitoring": {
  "metrics": {
    "hostname": "localhost"
    "port": 8125
    "tags": {
      "tag1": "value1"
      "tag2": "value2"
    }
    "prefix": "snowplow.enrich."
    "period": "10 seconds"
  }
}
```

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. Enrich can send an error report to sentry whenever something unexpected happens when trying to enrich an event. The reasons for the error can then be explored in the sentry server’s UI.

Sentry monitoring is configured by setting the `monitoring.sentry.dsn` key in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/) with the url of your sentry server:

```json
"monitoring": {
  "dsn": "http://sentry.acme.com"
}
```

---

# Enrich 4.0.x upgrade guide
> Upgrade guide for Snowplow Enrich 4.0.x covering license acceptance, atomic field limits, and deprecated assets.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/

## Breaking changes

### New license

Since version 4.0.0, Enrich has been migrated to use the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

### `stream-enrich` assets and `enrich-rabbitmq` deprecated

As announced a while ago, these assets are now removed from the codebase.

## Upgrading

### License acceptance

You have to explicitly accept the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)). To do so, either set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable, or update the following section in the configuration:

```hcl
{
  license {
    accept = true
  }
  ...
}
```

### Atomic fields limits

Several [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) fields, such as `mkt_clickid` have length limits defined (in this case, 128 characters). Recent versions of Enrich enforce these limits, so that oversized data does not break loading into the warehouse columns. However, over time we’ve observed that valid data does not always fit these limits. For example, TikTok click ids can be up to 500 (or 1000, according to some sources) characters long.

In this release, we are adding a way to configure the limits, and we are increasing the default limits for several fields:

- `mkt_clickid` limit increased from `128` to `1000`
- `page_url` limit increased from `4096` to `10000`
- `page_referrer` limit increased from `4096` to `10000`

Depending on your [configuration](/docs/api-reference/enrichment-components/configuration-reference/), this might be a breaking change:

- If you have `featureFlags.acceptInvalid` set to `true` in Enrich, then you probably don’t need to worry, because you had no validation in the first place (although we do recommend to enable it).

- If you have `featureFlags.acceptInvalid` set to `false` (default), then previously invalid events might become valid (which is a good thing), and you need to prepare your warehouse for this eventuality:

  - For Redshift, you should resize the respective columns, e.g. to `VARCHAR(1000)` for `mkt_clickid`. If you don’t, Redshift will truncate the values.
  - For Snowflake and Databricks, we recommend removing the VARCHAR limit altogether. Otherwise, loading might break with longer values. Alternatively, you can alter the Enrich configuration to revert the changes in the defaults.
  - For BigQuery, no steps are necessary.

Below is an example of how to configure these limits:

```hcl
{
  ...
  # Optional. Configuration section for various validation-oriented settings.
  "validation": {
    # Optional. Configuration for custom maximum atomic fields (strings) length.
    # Map-like structure with keys being field names and values being their max allowed length
    "atomicFieldsLimits": {
        "app_id": 5
        "mkt_clickid": 100000
        # ...and any other 'atomic' field with custom limit
    }
  }
}
```

---

# Enrich 6.0.x upgrade guide
> Upgrade guide for Snowplow Enrich 6.0.x covering common-streams refactoring, configuration changes, and deprecated features.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/

In version 6.0.0, Enrich is refactored to use [common-streams](https://github.com/snowplow-incubator/common-streams) libraries under the hood. [common-streams](https://github.com/snowplow-incubator/common-streams) is the collection of libraries that contains streaming-related constructs commonly used across many Snowplow streaming applications.

[common-streams](https://github.com/snowplow-incubator/common-streams) allows for the adjustment of many different settings. It also provides default values for most of these settings, which are battle-tested. Therefore, we recommend using default values whenever it is possible. You can find more information about defaults in [configuration reference](/docs/api-reference/enrichment-components/configuration-reference/).

Also, we took this opportunity to make a few breaking changes. Here are the changes:

### Config Field Changes

In version 6.0.0, some of the config fields are renamed or moved to a different section. Here are these changes:

- `incomplete` stream config field is renamed to `failed`.

- `acceptInvalid` and `exitOnJsCompileError` fields under `featureFlags` section are moved under the `validation` section.

- `experimental.metadata` section is moved to the root level.

- In enrich-kafka, `output.good.headers` field is renamed to `output.good.attributes`.

- In enrich-kafka, `blobStorage.azureStorage.accounts` section is moved to the `blobClients.accounts`.

- In enrich-kafka, we are now using [static membership](https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances) for the consumer, to reduce rebalancing in case of pod restart or crash. The default value for `group.instance.id` is set to the host name.

### Feature Deprecations

- Output `pii` stream is removed as in our experience it is not used. There will no longer be an option to write `pii_transformation` events to an extra output stream.

- Remote adapters are removed. This was another feature with little to no usage that allowed Enrich to support custom payloads (which would be sent to a configured URL for translation into the expected format). In practice, most of this can be already achieved with [Iglu Webhooks](/docs/sources/webhooks/iglu-webhook/).

- Reading events from files and writing events to files is no longer supported. This has never been a viable option for production setups.

- In enrich-kinesis, passing enrichment configs to the application via DynamoDB is no longer possible.

- In enrich-kinesis, it is no longer possible to send KCL metrics to Cloudwatch.

## Metrics Changes

Existing metrics will continue to be emitted. Three new metrics are added:

- `failed`: Same value as the `incomplete` metric, for transition. The goal is to remove `incomplete` metric and to be consistent with the naming of streams/topics in configuration

- `e2e_latency_millis`: Same value as the `latency` metric, for transition. The goal is to remove `latency` metric so that the naming is consistent across applications

- `latency_millis`: Delay between the input record getting written to the stream and Enrich starting to process it

Furthermore, the old `latency` metric has changed subtly. Before, it represented the latency of the most recently processed event. Now it refers to the _maximum latency of all events_ since the previous metric was emitted.

---

# Upgrade guides for Enrich
> Step-by-step guides for upgrading Snowplow Enrich applications to newer versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/api-reference/enrichment-components/upgrade-guides/

This section contains information to help you upgrade to newer versions of Enrich.

## [📄️ 4.0.x upgrade guide](/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/)

[Upgrade guide for Snowplow Enrich 4.0.x covering license acceptance, atomic field limits, and deprecated assets.](/docs/api-reference/enrichment-components/upgrade-guides/4-0-x-upgrade-guide/)

## [📄️ 6.0.x upgrade guide](/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/)

[Upgrade guide for Snowplow Enrich 6.0.x covering common-streams refactoring, configuration changes, and deprecated features.](/docs/api-reference/enrichment-components/upgrade-guides/6-0-x-upgrade-guide/)

---

# Failed event types
> Reference guide for Snowplow failed event types including schema violations, enrichment failures, and loader errors, with recovery recommendations.
> Source: https://docs.snowplow.io/docs/api-reference/failed-events/

This page lists all the possible types of [failed events](/docs/fundamentals/failed-events/).

## Where do failed events originate?

While an event is being processed by the pipeline it is checked to ensure it meets the specific formatting or configuration expectations. These include checks like: does it match the schema it is associated with, were Enrichments successfully applied, and was the payload sent by the tracker acceptable.

Generally, the [Collector](/docs/api-reference/stream-collector/) tries to write any payload to the raw stream, no matter its content, and no matter whether it is valid. This explains why many of the failure types are filtered out by the [Enrich](/docs/api-reference/enrichment-components/) application, and not any earlier.

> **Note:** The Collector might receive events in batches. If something is wrong with the Collector payload as a whole (e.g. due to a [Collector payload format violation](#collector-payload-format-violation)), the generated failed event would represent an entire batch of Snowplow events.
>
> Once the Collector payload successfully reaches the validation and enrichment steps, it is split into its constituent events. Each of them would fail (or not fail) independently (e.g. due to an [enrichment failure](#enrichment-failure)). This means that each failed event generated at this stage represents a single Snowplow event.

## Schema violation

This failure type is produced during the process of [validation and enrichment](/docs/pipeline/enrichments/). It concerns the [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) which can be attached to your snowplow event.

**Details**

In order for an event to be processed successfully:

1. There must be a schema in an [Iglu repository](/docs/api-reference/iglu/iglu-repositories/) corresponding to each self-describing event or entity. The enrichment app must be able to look up the schema in order to validate the event.
2. Each self-describing event or entity must conform to the structure described in the schema. For example, all required fields must be present, and all fields must be of the expected type.

If your pipeline is generating schema violations, it might mean there is a problem with your tracking, or a problem with your [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/) which lists where schemas should be found. The error details in the schema violation JSON object should give you a hint about what the problem might be.

Snowplow customers should check in the Snowplow Console that all data structures are correct and have been [promoted to production](/docs/event-studio/data-structures/). Snowplow Self-Hosted users should check that the Enrichment app is configured with an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) that points to a repository containing the schemas.

Next, check the tracking code in your custom application, and make sure the entities you are sending conform to the schema definition.

Once you have fixed your tracking, you might want to also [recover the failed events](/docs/monitoring/recovering-failed-events/), to avoid any data loss.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Schema violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema).

## Enrichment failure

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, and it represents any failure to enrich the event by one of your configured enrichments.

**Details**

There are many reasons why an enrichment will fail, but here are some examples:

- You are using the [custom SQL enrichment](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/) but the credentials for accessing the database are wrong
- You are using the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) but have mis-configured the location of the MaxMind database
- You are using the [custom API request enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/) but the API server is not responding
- The raw event contained an unstructured event field or a context field which was not valid JSON
- An Iglu server responded with an unexpected error response, so the event schema could not be resolved

If your pipeline is generating enrichment failures, it might mean there is a problem with your enrichment configuration. The error details in the enrichment failure JSON object should give you a hint about what the problem might be.

Once you have fixed your enrichment configuration, you might want to also [recover the failed events](/docs/monitoring/recovering-failed-events/), to avoid any data loss.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Enrichment failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema).

## Collector payload format violation

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when Collector payloads from the raw stream are deserialized.

**Details**

Violations could be:

- Malformed HTTP requests
- Truncation
- Invalid query string encoding in URL
- Path not respecting `/vendor/version`

The most likely source of this failure type is bot traffic that has hit the Collector with an invalid HTTP request. Bots are prevalent on the web, so do not be surprised if your Collector receives some of this traffic. Generally you would ignore, and not try to recover, a Collector payload format violation, because it likely did not originate from a tracker or a webhook.

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Collector payload format violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/collector_payload_format_violation/jsonschema).

## Adapter failure

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when it tries to interpret a Collector payload from the raw stream as a HTTP request from a [3rd party webhook](/docs/sources/webhooks/).

> **Info:** Many adapter failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline.

**Details**

The failure could be:

1. The vendor/version combination in the Collector URL is not supported. For example, imagine an HTTP request sent to `/com.sandgrod/v3` which is a mis-spelling of the [sendgrid adapter](http://sendgrid.com) endpoint.
2. The webhook sent by the 3rd party does not conform to the expected structure and list of fields for this webhook. For example, imagine the 3rd party webhook payload is updated and stops sending a field that it was sending before.

Many adapter failures are caused by bot traffic, so do not be surprised to see some of them in your pipeline. However, if you believe you are missing data because of a misconfigured webhook, then you might try to fix the webhook and then [recover the failed events](/docs/monitoring/recovering-failed-events/).

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Adapter failure schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema).

## Tracker protocol violation

This failure type is produced by the [Enrich](/docs/pipeline/enrichments/) application, when an HTTP request does not conform to our [Snowplow Tracker Protocol](/docs/events/).

**Details**

Snowplow trackers send HTTP requests to the `/i` endpoint or the `/com.snowplowanalytics.snowplow/tp2` endpoint, and they are expected to conform to this protocol.

Many tracker protocol violations are caused by bot traffic, so do not be surprised to see some of them in your pipeline.

Another likely source is misconfigured query parameters if you are using the [pixel tracker](/docs/sources/pixel-tracker/). In this case you might try to fix your application sending events, and then [recover the failed events](/docs/monitoring/recovering-failed-events/).

Because this failure is handled during enrichment, events in the real time good stream are free of this violation type.

Tracker protocol violation schema can be found [here](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema).

## Size violation

This failure type can be produced either by the [Collector](/docs/api-reference/stream-collector/) or by the [Enrich](/docs/pipeline/enrichments/) application. It happens when the size of the raw event or enriched event is too big for the output message queue. In this case it will be truncated and wrapped in a size violation failed event instead.

**Details**

Failures of this type cannot be [recovered](/docs/monitoring/recovering-failed-events/). The best you can do is to fix any application that is sending over-sized events.

Because this failure is handled during collection or enrichment, events in the real time good stream are free of this violation type.

The size violation schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/size_violation/jsonschema/1-0-0).

## Loader parsing error

This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/), if the enriched event in the real time good stream cannot be parsed as a canonical TSV event format. For example, if the row does not have enough columns (131 are expected) or the `event_id` is not a UUID. This error type is uncommon and unexpected, because it can only be caused by an invalid message in the stream of validated enriched events.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

The loader parsing error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_parsing_error/jsonschema/2-0-0).

## Loader Iglu error

This failure type can be produced by [any loader](/docs/api-reference/loaders-storage-targets/) and describes an error using the [Iglu](/docs/api-reference/iglu/) subsystem.

**Details**

For example:

- A schema is not available in any of the repositories listed in the [Iglu resolver](/docs/api-reference/iglu/iglu-resolver/).
- Some loaders (e.g. [RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) and [Postgres loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)) make use of the "schema list" API endpoints, which are only implemented for an [Iglu server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) repository. A loader Iglu error will be generated if the schema is in a [static repo](/docs/api-reference/iglu/iglu-repositories/static-repo/) or [embedded repo](/docs/api-reference/iglu/iglu-repositories/jvm-embedded-repo/).
- The loader cannot auto-migrate a database table. If a schema version is incremented from `1-0-0` to `1-0-1` then it is expected to be [a non-breaking change](/docs/api-reference/iglu/common-architecture/schemaver/), and many loaders (e.g. RDB loader) attempt to execute a `ALTER TABLE` statement to facilitate the new schema in the warehouse. But if the schema change is breaking (e.g. string field changed to integer field) then the database migration is not possible.

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader Iglu error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_iglu_error/jsonschema/2-0-0).

## Loader recovery error (legacy)

Only the [BigQuery repeater](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/#snowplow-bigquery-repeater) generated this error. We call it "loader recovery error" because the purpose of the repeater was to recover from previously failed inserts. It represents the case when the software could not re-insert the row into the database due to a runtime failure or invalid data in a source.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader recovery error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_recovery_error/jsonschema/1-0-0)

## Loader runtime error

This failure type can be produced by any loader and describes generally any runtime error that we did not catch. For example, a DynamoDB outage, or a null pointer exception. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Loader runtime error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1).

## Relay failure

This failure type is only produced by relay jobs, which transfer Snowplow data into a 3rd party platform. This error type is uncommon and unexpected, and it probably indicates a mistake in the configuration or a bug in the software.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Relay failure schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/relay_failure/jsonschema/1-0-0).

## Generic error

This is a failure type for anything that does not fit into the other categories, and is unlikely enough that we have not created a special category. The failure error messages should give you a hint about what has happened.

**Details**

This failure type cannot be [recovered](/docs/monitoring/recovering-failed-events/).

Generic error schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/generic_error/jsonschema/1-0-0).

---

# Core Iglu components and data structures
> Platform-agnostic core components for Iglu with SchemaKey, SchemaVer, and SchemaCriterion data structures for consistent schema handling.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/iglu-core/

Iglu is designed to be not dependent on any particular programming language or platform. But there's growing set of applications beside clients and registries using different concepts originated from Iglu. To have consistent data structures and behavior among different applications, we're developing Iglu core libraries for different languages.

## Basic data structures

All languages have their own unique features and particular Iglu Core implementation may or may not use these features. One common rule for all Iglu core implementations is to minimize dependencies. Ideally Iglu core should have no external dependencies. Another rule is to implement the required basic data structures (in form of classes, structs, ADTs or any other appropriate form) and functions.

### SchemaKey

This data structure contains information about Self-describing datum, such as Snowplow unstructured event or context.

It also should have related `parse` functions, which can parse `SchemaKey` from most common representation - Iglu URI (string with form of `iglu:com.acme/someschema/format/1-0-0`) and Iglu path (same, but without `iglu:` protocol part). Reverse `asString` function required as well.

This also can include appropriate regular expressions to extract and validate schema key. Function for parsing `SchemaKey` from JSON Schemas is optional if there's no default JSON library like in JavaScript, but can be included within some interface.

More information can be found in [Self-describing JSON Schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) and [Self-describing JSONs](/docs/api-reference/iglu/common-architecture/self-describing-jsons/) wiki pages.

### SchemaMap

This is almost isomorphic entity to `SchemaKey`, which also contains same information: vendor, name, format and version. But unlike `SchemaKey` it supposed to be attached only to Schemas instead of datums. In schemas same information usually has different representation and also version is always _full_ opposed to datum's possibly _partial_.

### SchemaVer

This is a part of `SchemaKey` and `SchemaMap` with information about semantic Schema version, basically triplet of MODEL, REVISION, ADDITION.

Like, `SchemaKey` it should contain `parse` function with regular expressions as well as `asString` method.

It can either _full_ (e.g. `1-2-0`) or _partial_ (e.g. `1-?-?`) suited for schema inference.

More information can be found in dedicated wiki page: [SchemaVer](/docs/api-reference/iglu/common-architecture/schemaver/).

### SchemaCriterion

Last core data structure is `SchemaCriterion` which is a default way to filter Self-describing entities. Basically it represent `SchemaKey` divided into six parts, where last three (MODEL, REVISION, ADDITION) _can_ be unfilled, thus one can match all entities regardless parts which remain unfilled.

`SchemaCriterion` also must contain regular expression, `parse` and `asString` (unfilled parts replaced with asterisks) functions. One other required function is `matches` which accepts `SchemaCriterion` and `SchemaKey` and returning boolean value indicating if key was matched. Bear in mind that criterions matching versions like `.../*-1-*` or `.../*-*-0` are absolutely valid, they're useful if want to match all initial Schemas.

## Implementations

Currently we have only [Scala Iglu Core](https://github.com/snowplow/iglu/wiki/Scala-Iglu-Core) which can be considered as reference implementation. Among described above data structures it includes type classes and container classes to improve type-safety. These type classes and containers are completely optional in other implementations.

---

# Iglu architecture and design principles
> Technical design principles for the Iglu schema registry including self-describing JSON schemas, SchemaVer versioning, and schema resolution algorithms.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/

Iglu is built on a set of technical design decisions which are documented in this section. It is this set of design decisions that allow Iglu clients and repositories to interoperate.

## Common architecture aspects

Please review the following design documents:

- [Self-describing JSON Schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) - simple extensions to JSON Schema which **semantically identify** and version a given JSON Schema
- [Self-describing JSONs](/docs/api-reference/iglu/common-architecture/self-describing-jsons/) - a standardized JSON format which co-locates a reference to the instance's JSON Schema alongside the instance's data
- [SchemaVer](/docs/api-reference/iglu/common-architecture/schemaver/) - how we semantically version schemas
- [Schema resolution](/docs/api-reference/iglu/common-architecture/schema-resolution/) - our public algorithm for how we determine in which order we check Iglu repositories for a given schema

---

# Schema resolution algorithm for Iglu clients
> Standard schema resolution algorithm for Iglu clients with registry prioritization, caching, and lookup strategies.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/schema-resolution/

This page describes the Schema resolution algorithm which is standard for all Iglu clients. Currently only [Iglu Scala client](https://github.com/snowplow/iglu-scala-client) fully follow this algorithm, while other clients may miss some parts, but we're working on making their behavior consistent.

## 1. Prerequisites

Before going further it is important to understand basic Iglu client configuration and essential concepts like Resolver, Registry (or Repository), Schema. Here is a quick overview of these concepts, if you're familiar with them you may want to skip this section.

Iglu clients are configured via JSON object described in dedicated Schema called [resolver-config](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.iglu/resolver-config/jsonschema). Here we'll be using JSON resolver configuration which is platform independent and most wide-spread.

### 1.1 Resolver

Resolver is an primary object of Iglu Client library, which contains all logic necessary to fetch requested Schema from appropriate registry (repository) and cache it properly. Resolver has two main properties: cache size (`cacheSize`) and list of registries (`repositories`).

### 1.2 Registries

**NOTE:** term _repository_ was deprecated. _Registry_ is default term to use when referring to Schema storage. So far, we've not renamed all occurrences, so for now they can be used interchangeable.

Each registry in resolver configuration has several values common for all types of registries, such as `name`, `vendorPrefixes` and `priority`. Also each registry has type, which is defined inside `connection` property. The only one important thing here about type of repository is that each type has its own priority hardcoded inside client library. Below we'll refer to this hard-coded priority by `classPriority` and to user-defined priority by `instancePriority` Usually, the "safer" registry - the higher `classPriority` it has, so local repositories are more preferable than remote.

### 1.3 Cache

All Iglu clients use internal cache to store registry responses. By virtue of it, it is absolutely safe to launch Hadoop/Spark jobs with Iglu client embedded as it will not generate enormous amount of IO calls.

#### 1.3.1 Cache algorithm

Cache stores not just plain Schemas, but information about responses from each registry. It allows us to make different decisions depending on what exactly went wrong with particular request. Since Schema was successfuly fetched it will be stored until moment it get evicted by [LRU cache](https://en.wikipedia.org/wiki/Cache_replacement_policies#Least_Recently_Used_\(LRU\)) algorithm. This eviction it turn happens only if cache map reached its limit (defined in `cacheSize`) and particular Schema wasn't requested for longer time than all other.

#### 1.3.2 Cache TTL

Since version 0.5.0, Iglu Scala Client supports `cacheTtl` property. It is especially useful for real-time pipelines as they can store "failure" for very long time and TTL is a mechanism to ensure that day-long data won't go to bad stream. Note however that client also tries to re-resolve successfully fetched schemas, this allows operators to patch (re-upload) schemas without bringing pipeline down (although it is not recommended).

`cacheTtl` is available since [`1-0-2` version](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2) of resolver config.

## 2. Lookup algorithm

Overall, Schema Resolution algorithm can be described by following flowchart:

![](/assets/images/schema-resolution-flowchart-156afc49078500123a19aaf9fab7e4e7.png)

Few important things to note:

- If registry responded with "NotFound" error - "missing" value will be cached and this repository won't be queried again, until this "missing" value not evicted by LRU-algorithm
- If registry responded with error other than "NotFound", for example "TimeoutError", "NetworkError", "ServerFault" etc - "needToRetry" value will be cached and Resolver will give this registry 3 chances more. After three failed lookups - "missing" value will be cached
- These "missing" and "needToRetry" values in cache are per-registry, not per-schema, which means if `registryA` responded "NotFound" for Schema `iglu:com.acme/event/jsonschema/1-0-0` and `registryB` responded with TimeoutError - resolver will immediately abandon `registryA` and keep try to query `registryB` for 3 more times.

## 3. Registry priority

For each particular Schema lookup, registries will be prioritized. In other words they will be sorted according following input parameters (ordered by their significance):

- `vendorPrefix` - Resolver always will look first into those registries which `vendorPrefix`es matches `SchemaKey`'s vendor. It **does not** mean registries with unmatched `vendorPrefix` will be skipped, it means they will be queried last.
- `classPriority` - hardcoded in client library value for each type of registry. It means that whatever high priority (low integer value) was set up in configuration for a particular registry - it will be overridden by `classPriority`, so embedded repository will always be checked before HTTP (unless priority influenced by `vendorPrefix`)
- `instancePriority` - user-defined value. Influence only repositories within same `classPriority`.

One important thing to note is that both priorities (`classPriority` and `instancePriority`) order registries in ascending order. That means lower number means higher priority. Think of it as ascending list of number: `[1,2,3,4]` - smaller will be always first.

---

# SchemaVer semantic versioning for schemas
> Overview of a semantic versioning system for JSON schemas.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/schemaver/

_This page is adapted from the Snowplow Analytics blog post, [Introducing SchemaVer for semantic versioning of schemas](http://snowplowanalytics.com/blog/2014/05/13/introducing-schemaver-for-semantic-versioning-of-schemas/) ._

### Overview

With the advent of our new self-describing JSON Schemas, it became necessary to implement some kind of versioning to those JSON Schemas so they could evolve through time.

Our approach is based on [semantic versioning](http://semver.org/) (SemVer for short) which, as a reminder, looks like this: `MAJOR.MINOR.PATCH`

- `MAJOR` which you're supposed to use when you make backwards-incompatible API changes
- `MINOR` when you add backwards-compatible functionalities
- `PATCH` when you make backwards-compatible bug fixes

As is, SemVer does not suit schema versioning well. Indeed, there is no such thing as bug fixes for a JSON Schema and the idea of an API doesn't really translate to JSON Schemas either.

That's why we decided to introduce our own schema versioning notion: SchemaVer. SchemaVer is defined as follows: `MODEL-REVISION-ADDITION`

- `MODEL` when you make a breaking schema change which will prevent interaction with _any_ historical data
- `REVISION` when you introduce a schema change which _may_ prevent interaction with _some_ historical data
- `ADDITION` when you make a schema change that is compatible with _all_ historical data

### Addition example

By way of example, if we were to modify an existing JSON Schema representing an ad click with version `1-0-0` defined as follows:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": false
}
```

and introduce a new `impressionId` property to obtain the following JSON Schema:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        },
        "impressionId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": false
}
```

Because the new `impressionId` is **not** a required property and because the `additionalProperties` in our `1-0-0` version was set to `false`, any historical data following the `1-0-0` schema will work with this new schema.

According to our definition of SchemaVer, we are consequently looking at an `ADDITION` and the schema's version becomes `1-0-1`.

### Revision example

If we continue with the same example, but modify the `additionalProperties` property to true to get the following schema:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        },
        "impressionId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": true
}
```

We are now at version `1-0-2`. After a while, we decide to add a new `cost` property:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        },
        "impressionId": {
            "type": "string"
        },
        "cost": {
            "type": "number",
            "minimum": 0
        }
    },
    "required": ["bannerId"],
    "additionalProperties": true
}
```

The problem now is that since we modified the `additionalProperties` to true before adding the `cost` field, someone might have added another `cost` field in the meantime following a different set of rules (for example it could be an amount followed by the currency such as 1.00$, the effective type would be string and not number) and so we cannot be sure that this new schema validate all historical data.

As a result, this new JSON Schema is a `REVISION` of the previous one, its version becomes `1-1-0`.

### Model example

Times goes by and we choose to completely review our JSON Schema identifying an ad click only through a `clickId` property so our schema becomes:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "clickId": {
            "type": "string"
        },
        "cost": {
            "type": "number",
            "minimum": 0
        }
    },
    "required": ["clickId"],
    "additionalProperties": false
}
```

The change is so important that we cannot realistically expect our historical data to interact with this new JSON Schema, consequently, the `MODEL` is changed and the schema's version becomes `2-0-0`.

Another important thing to notice is that we switched the `additionalProperties` back to false in order to avoid unnecessary future revisions.

### Additional differences

There are a few additional differences between our own SchemaVer and SemVer:

- we use hyphens instead of periods to separate the components that make our SchemaVer
- the versioning starts with `1-0-0` instead of `0.1.0`

The design considerations behind those decisions can be found in the blog post on [SchemaVer](http://snowplowanalytics.com/blog/2014/05/13/introducing-schemaver-for-semantic-versioning-of-schemas/).

---

# Self-describing JSON schemas and vendor metadata
> JSON Schema extension with self property containing vendor, name, format, and version metadata for schema identification.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/

_This page is adapted from the Snowplow Analytics blog post, [Introducing self-describing JSONs](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/)._

With the explosion of possible event types due to Snowplow going from a web analytics to a general event analytics platform, it became necessary to give some coherence to the events sent in to Snowplow. Snowplow dealing only with JSON, we chose to rely on JSON Schemas.

In addition to the usual JSON Schema we decided to make it self-describing by adding information we already knew about the schema such as:

- `vendor` which tells us who created this JSON Schema
- `name` which is the JSON Schema's name
- `format` in our case this will be a JSON Schema
- `version` which is the JSON Schema's version (using [SchemaVer](/docs/api-reference/iglu/common-architecture/schemaver/))

We encapsulated all this information in a `self` property.

As an example, we would go from this JSON Schema:

```json
{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": false
}
```

to this one:

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.snowplowanalytics",
        "name": "ad_click",
        "format": "jsonschema",
        "version": "1-0-0"
    },
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": false
}
```

incorporating the aforementioned `self` property.

Notice that we also changed the `$schema` property to [our own JSON Schema](http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#) which enforces the `self` property.

To make our JSONs self-describing we still have to reference this JSON Schema in our JSONs. This process is described in [Self-describing JSONs](/docs/api-reference/iglu/common-architecture/self-describing-jsons/).

---

# Self-describing JSON format
> Standardized JSON format linking data instances to their schemas via Iglu URI references in a schema field.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/common-architecture/self-describing-jsons/

_This page is adapted from the Snowplow Analytics blog post, [Introducing self-describing JSONs](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/)._

In this section, we will be describing the approach we chose to link together a JSON with its JSON Schema in order to make it self-describing. Instead of embedding the JSON Schema directly into the JSON itself which would be very wasteful in terms of space, we chose only to store a reference to its JSON Schema.

For example, let's say we have a JSON representing a click on an ad like so:

```json
{
    "bannerId": "4acd518feb82"
}
```

which is supposed to conform to this [self-describing JSON Schema](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/):

```json
{
    "$schema": "http://json-schema.org/schema#",
    "self": {
        "vendor": "com.snowplowanalytics",
        "name": "ad_click",
        "format": "jsonschema",
        "version": "1-0-0"
    },
    "type": "object",
    "properties": {
        "bannerId": {
            "type": "string"
        }
    },
    "required": ["bannerId"],
    "additionalProperties": false
}
```

Our self-describing JSON will look like this:

```json
{
    "schema": "iglu:com.snowplowanalytics/ad_click/jsonschema/1-0-0",
    "data": {
        "bannerId": "4acd518feb82"
    }
}
```

Notice the two main differences compared to our original JSON:

1. There is a new `schema` field located at the root of the JSON which contains (in a space-efficient format) all the information required to uniquely identify the associated JSON Schema. The schema's URI follows the following pattern:

![](/assets/images/iglu-schema-key-bcb8f8d1b9814714ec9590690ebb4394.png)

1. The data contained in the original JSON has been encapsulated in a `data` field to prevent any accidental collisions should the JSON already have a `schema` field

This way, our JSON becomes de facto self-describing, embedding a link to its JSON Schema.

Back to [Common architecture](/docs/api-reference/iglu/common-architecture/).

---

# Set up an Iglu Central mirror
> Create public mirrors or private clones of Iglu Central schema registry for offline access or reduced latency.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-central-setup/

This guide is designed for Iglu users wanting to create a public mirror or private clone of [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/). There are a couple of reasons you may want to do this:

1. You may want to access Iglu Central from a software system that cannot access the open internet.
2. You may want a mirror of Iglu Central which has lower latency to your software system.

This guide is divided into two sections:

1. Create Iglu Central Mirror
2. Update your Iglu client configuration to point to your new Iglu Central

## Create Iglu Central Mirror

### Hosting an Iglu Server based mirror

You can mirror Iglu Central using `[igluctl](/docs/api-reference/iglu/igluctl-2/index.md)`:

```bash
git clone https://github.com/snowplow/iglu-central
cd iglu-central
igluctl static push --public schemas/ http://MY-IGLU-URL 00000000-0000-0000-0000-000000000000
```

For further information on Iglu Central, consult the [Iglu Central setup guide](/docs/api-reference/iglu/iglu-central-setup/).

### Hosting a Static Repository based mirror

Iglu Central is built on top of the Iglu static repo server, so the first step is to [setup a static repo](/docs/api-reference/iglu/iglu-repositories/static-repo/). You can give your copy of Iglu Central a name like:

```text
http://iglucentral.acme.com
```

Once you have completed this static repo setup, then copy into your `/schemas` sub-folder **all** of the schemas that you can find [in the Iglu Central GitHub Repo](https://github.com/snowplow/iglu-central/tree/master/schemas)

Once you have done this, check that your schemas are publically accessible, for example:

```text
http://iglucentral.acme.com/schemas/com.snowplowanalytics.self-desc/instance/jsonschema/1-0-2
```

## Update your Iglu client configuration

You now need to update your Iglu client configuration to point to your Iglu Central mirror, rather than the original.

Given a standard Iglu client configuration:

```json
{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2",
  "data": {
    "cacheSize": 500,
    "repositories": [
      {
        "name": "Iglu Central",
        "priority": 0,
        "vendorPrefixes": [ "com.snowplowanalytics" ],
        "connection": {
          "http": {
            "uri": "https://iglucentral.com"
          }
        }
      }
    ]
  }
}
```

Update it to point to your Iglu Central mirror:

```json
{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2",
  "data": {
    "cacheSize": 500,
    "repositories": [
      {
        "name": "Acme Corp's Iglu Central mirror",
        "priority": 0,
        "vendorPrefixes": [ "com.snowplowanalytics" ],
        "connection": {
          "http": {
            "uri": "http://iglucentral.acme.com"
          }
        }
      }
    ]
  }
}
```

And that's it - your Iglu client should now resolve to your Iglu Central mirror, rather than the original.

---

# Iglu client libraries
> Client libraries for resolving schemas from Iglu repositories in Scala and Objective-C with embedded and remote repository support.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-clients/

Iglu clients are used for interacting with Iglu server repos and for resolving schemas in embedded and remote Iglu schema repositories.

## Technical architecture

In this diagram we show an Iglu client resolving a schema from Iglu Central, one embedded repository and a further two remote HTTP repositories:

![](/assets/images/iglu-clients-2a639a6f765d5146f869eb947a42f15c.png)

For more information on the rules governing resolving schemas from multiple repositories, see [Schema resolution](/docs/api-reference/iglu/common-architecture/schema-resolution/).

## Available Iglu clients

There are currently two Iglu client libraries implemented:

| **Repo server**                                               | **Description**                       | **Status**       |
| ------------------------------------------------------------- | ------------------------------------- | ---------------- |
| [Scala client](https://github.com/snowplow/iglu-scala-client) | An Iglu client and resolver for Scala | Production-ready |
| [Objc client](https://github.com/snowplow/iglu-objc-client)   | An Iglu client and resolver for OSX   | Unsupported      |

---

# Objective-C Iglu client
> Iglu client library for Objective-C with JSON schema resolution and validation for iOS 7.0+ and macOS 10.9+.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-clients/objc-client/

The [Iglu Objc client](https://github.com/snowplow/iglu-objc-client) allows you to resolve JSON Schemas from embedded and remote repositories. It does not yet let you write to repositories in any way (e.g. you can't publish new schemas to an Iglu repository).

This client library should be straightforward to use if you are comfortable with Objective-C development.

## Client compatibility

The Obj-C client is compatible with OSX 10.9+ and iOS 7.0+.

## Dependencies

The library is dependant on [KiteJSONValidator](https://github.com/samskiter/KiteJSONValidator) for all JSONSchema validation.

## Setup

### CocoaPods

We support installing the Obj-C Client via CocoaPods since it's the easiest way to install the client. Doing so is simple:

1. Install CocoaPods using `gem install cocoapods`
2. Create the file `Podfile` in the root of your XCode project directory, if you don't have one already
3. Add the following line into it:

```ruby
pod 'SnowplowIgluClient'
```

4. Run `pod install` in the same directory

### Manual Setup

If you prefer not to use CocoaPods, you can grab the client from our [GitHub repo](https://github.com/snowplow/iglu-objc-client.git) and import it into your project.

#### Clone the client

First, git clone the latest version of the client to your local machine:

```bash
git clone https://github.com/snowplow/iglu-objc-client.git
```

If you don't have git installed locally, [install it](http://git-scm.com/downloads) first.

#### Copy the client into your project

You first need to copy the client's `SnowplowIgluClient` sub-folder into your XCode project's folder. The command will look something like this:

```bash
cp -r iglu-objc-client/SnowplowIgluClient MyObjcApp/MyObjcApp/
```

- Replace `MyObjcApp` with the name of your own app, and tweak the source code sub-folder accordingly.
- Next, drag and drop the sub-folder `MyObjcApp/MyObjcApp/SnowplowIgluClient` into your XCode project's workspace.
- Make sure that the suggested options for adding `SnowplowIgluClient` are set **Create groups**, then click **Finish**.

#### Copy required resources (Optional)

The client requires two schemas for initial operation; the first for validating that a JSON is a correct self-describing JSON and the second for validating the resolver-config JSON passed to it in startup. The client will look for these in a resource bundle named `SnowplowIgluResources`.

To get this bundle you will need to:

- Open the `SnowplowIgluClient.xcworkspace` in XCode.
- Build the `SnowplowIgluResources` schema.
- In your `Products` folder within XCode you should now see a `SnowplowIgluResources.bundle`.
- Copy this bundle to your project.

Alternatively you can also include the standard Snowplow repository in your resolver-config:

```json
{
  "name": "Iglu Central",
  "vendorPrefixes": [
    "com.snowplowanalytics"
  ],
  "connection": {
    "http": {
      "uri": "https://iglucentral.com"
    }
  },
  "priority": 0
}
```

This will allow the client to download the required schemas at runtime.

## Initialization

Assuming you have completed the setup for your Objective-C project, you are now ready to initialize the Obj-C client.

### Importing the library

All interactions are handled through the ObjC client's `IGLUClient` class.

Import the header for the client like so:

```objc
#import "IGLUClient.h"
```

You are now ready to create your Obj-C Client.

### JSON-based initialization

You will need to supply either a resolver-config as an `NSString` or the URL to your resolver-config as an argument for the client. If a valid resolver-config is not passed in the client will throw an **exception**.

To make this step a touch easier we have included several utility functions for getting `NSString's` from URLs and File Paths.

For example; grabbing your resolver-config from a local source and creating the client could look like this:

```objc
#import "IGLUUtilities.h"

// Create Client
NSString * resolverAsString = [IGLUUtilities
  getStringWithFilePath:@"your_iglu_resolver.json"
  andDirectory:@"Your_Directory"
  andBundle:[NSBundle bundleForClass:[self class]]];

IGLUClient * client = [[IGLUClient alloc]
  initWithJsonString:resolverAsString
  andBundles:nil];
```

To create a client from a URL:

```objc
// The URL is passed as an NSString
IGLUClient * client = [[IGLUClient alloc] initWithUrlPath:@"https://raw.githubusercontent.com/snowplow/snowplow/master/3-enrich/config/iglu_resolver.json" andBundles:nil];
```

The `andBundle:` argument of the client init accepts an `NSMutableArray` of bundle objects. These objects will be used to search for files for any embedded repositories you include.

To add to the available bundles you can use:

```objc
[client addToBundles:yourBundleObject];
```

## Validating JSON

Once you have successfully created a client you can start validating your self-describing JSON.

**NOTE:** All JSONs must first be parsed into an NSDictionary before they can be validated.

To parse your JSON String as an `NSDictionary` you can use `IGLUUtilities` like so:

```objc
NSDictionary * jsonDictionary = [IGLUUtilities parseToJsonWithString:yourStringHere];
```

To validate your JSON:

```objc
BOOL result = [client validateJson:jsonDictionary];
```

The above command is telling the client to:

- Check the JSON is a valid self-describing JSON
- Check the JSON validates against it's own schema

Currently the only output from the client will be a `YES` or `NO` response as the underlying library does not support error printing as of yet.

---

# Scala Iglu client
> Production-ready Scala Iglu client and schema resolver for JVM applications with SBT and Gradle integration.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-clients/scala-client-setup/

The [Scala client](https://github.com/snowplow/iglu-scala-client) is an Iglu client and schema resolver implemented in Scala.

Setting up the Scala client to use from your own code is straightforward. For actual examples of initialization you can look at [Scala client](https://github.com/snowplow/iglu-scala-client) page.

## Integration options

To minimize jar bloat, we have tried to keep external dependencies to a minimum. The main dependencies are on Jackson and JSON Schema-related libraries.

## Setup

### Hosting

The Scala client is published to Snowplow's [hosted Maven repository](http://maven.snplow.com), which should make it easy to add it as a dependency into your own Java app.

The current version of the Scala client is 4.0.3.

### SBT

Add this to your SBT config:

```scala
// Dependency
val igluClient = "com.snowplowanalytics" %% "iglu-scala-client"  % "4.0.3"
```

### Gradle

Add into `build.gradle`:

```gradle
dependencies {
  ...
  // Iglu client
  compile 'com.snowplowanalytics:iglu-scala-client:"4.0.3"'
}
```

Now read the [Scala client API](https://github.com/snowplow/iglu-scala-client) to start using the Scala client.

---

# Iglu Central public schema repository
> Public machine-readable repository of Snowplow JSON schemas hosted on Amazon S3 with self-hosting support via igluctl.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-central/

[Iglu Central](https://iglucentral.com/) is a public repository of JSON Schemas hosted by Snowplow Analytics.

As far as we know, Iglu Central is the first public **machine-readable** schema repository - all prior efforts we have seen are human-browsable directories of articles about schemas (e.g. [schema.org](http://schema.org/)).

Think of Iglu Central as like [RubyGems.org](http://rubygems.org/) or [Maven Central](http://central.maven.org/) but for storing publically-available JSON Schemas.

## Technical architecture

Under the hood, Iglu Central is built and run as a static Iglu repository, which is simply an Iglu repository server structured as a static website serving its whole content over http, and is hosted on Amazon S3.

![iglu-central-img](/assets/images/iglu-central-c0427b712c8c80ad53d1a8a2b7e6871d.png)

The [deployment process](/docs/api-reference/iglu/iglu-central-setup/) for Iglu Central is documented on this wiki in case a user wants to setup a public mirror or private instance of Iglu Central.

Iglu Central is available for view at [https://iglucentral.com](https://iglucentral.com/). Although Iglu Central is primarily designed to be consumed by [Iglu clients](/docs/api-reference/iglu/iglu-clients/), the root index page for Iglu Central links to all schemas currently hosted on Iglu Central.

## Self Hosting Iglu Central schemas

The schemas for Iglu Central are stored in GitHub, in [snowplow/iglu-central](https://github.com/snowplow/iglu-central). You can mirror Iglu Central using `[igluctl](/docs/api-reference/iglu/igluctl-2/index.md)`:

```bash
git clone https://github.com/snowplow/iglu-central
cd iglu-central
igluctl static push --public schemas/ http://CHANGE-TO-MY-IGLU-URL.elb.amazonaws.com 00000000-0000-0000-0000-000000000000
```

For further information on Iglu Central, consult the [Iglu Central setup guide](/docs/api-reference/iglu/iglu-central-setup/).

---

# Iglu Server
> RESTful interface for publishing, testing, and serving Iglu schemas with comprehensive API endpoints for schema management, validation, and authentication.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-server/

The [Iglu Server](https://github.com/snowplow-incubator/iglu-server/) is an Iglu schema registry which allows you to publish, test and serve schemas via an easy-to-use RESTful interface. It is split into a few services which will be detailed in the following sections.

## Setup an Iglu Server

Information on setting up an instance of the Iglu Server can be found in [the setup guide](/docs/api-reference/iglu/iglu-repositories/iglu-server/setup/).

## 1. The schema service (`/api/schemas`)

The schema service allows you to interact with Iglu schemas using simple HTTP requests.

### 1.1 POST requests

Sending a `POST` request to the schema service allows you to publish a new self-describing schema to the repository.

As an example, let's assume you own the `com.acme` vendor prefix (more information on that can be found in the API authentication section) and have a JSON schema defined as follows:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    "clickId": {
      "type": "string"
    },
    "targetUrl": {
      "type": "string",
      "minLength": 1
    }
  },
  "required": ["targetUrl"],
  "additionalProperties": false
}
```

The schema can be added to your repository by making a `POST` request to the following endpoint with the schema included in the request's body:

```text
HOST/api/schemas/
```

By default, the schema will not be public (available to others) - this can be changed by adding an `isPublic` query parameter and setting its value to `true`.

For example, the following request:

```bash
curl HOST/api/schemas -X POST -H "apikey: YOUR_APIKEY" -d @myschema.json
```

will produce a response like this one, if no errors are encountered:

```json
{
  "message": "Schema created",
  "updated": false,
  "location": "iglu:com.acme/ad_click/jsonschema/1-0-0",
  "status":201
}
```

_Please note:_ This endpoint must be used with an API key with a `schema_action` permission of `CREATE`.

### 1.2 PUT requests

Another way of adding schemas is using a `PUT` request. Just like a `POST` request, it allows you to publish a schema to the Iglu Server by including it in the request's body, with an optional `isPublic` parameter used to set the visibility of a schema. Unlike `POST` requests, `PUT` requests require a schema's Iglu URI to be included in the request URI (i.e. `HOST/api/schemas/vendor/name/format/version`.

However, this means that a schema included in the request's body can be non-self-describing as well as self-describing. Note that in the latter case the URL path must match the schema's metadata, so a schema described as `iglu:com.snplow/foo/jsonschema/1-0-0` cannot be published using the `/api/schemas/com.acme/bar/jsonschema/1-0-0` PUT endpoint.

For example:

```bash
curl HOST/api/schemas/com.acme/ad_click/jsonschema/1-0-0 -X PUT -H "apikey: YOUR_APIKEY" -d @myschema.json
```

_Please note:_ This endpoint must be used with an API key with a `schema_action` permission of `CREATE`.

### 1.3 Single-schema GET requests

A schema previously added to the repository can be retrieved by making a `GET` request:

```bash
curl HOST/api/schemas/com.acme/ad_click/jsonschema/1-0-0 -X GET -H "apikey: YOUR_APIKEY"
```

The JSON response should look like this:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
  "clickId": {
    "type": "string"
  },
  "targetUrl": {
    "type": "string",
    "minLength": 1
  }
  },
  "required": ["targetUrl"],
  "additionalProperties": false
}
```

GET requests support a `repr` URL parameter, allowing you to specify three different ways of representing an Iglu schema. This can have values of either `canonical`, `meta` or `uri`. `repr=canonical` returns the schema as a self-describing Iglu event - it is also the default representation method if no query parameter is specified. (An example of this representation can be seen above.) `repr=meta` adds an additional `metadata` field to the schema, containing some meta information about it - this would make the JSON response look like this:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
  "clickId": {
    "type": "string"
  },
  "targetUrl": {
    "type": "string",
    "minLength": 1
  }
  },
  "required": ["targetUrl"],
  "additionalProperties": false,
  "metadata": {
    "createdAt":"2019-04-01T14:23:45.173728Z",
    "updatedAt":"2019-04-01T14:23:45.173728Z",
    "isPublic":true
  }
}
```

`repr=uri` simply returns a schema's Iglu URI - this is used internally in the Iglu Server, but public requests are also supported. A response with this URL parameter set would look like this:

```text
"iglu:com.acme/ad_click/jsonschema/1-0-0"
```

_Please note:_ While `metadata`/`body` query parameters used in previous versions of the Iglu Server are supported, they have been deprecated in favor of the single `repr` parameter.

### 1.4 Multiple GET requests

You can also retrieve multiple schemas in a single `GET` request using a few different endpoints

#### Vendor-based requests

Every schema belonging to a single vendor can be retrieved by sending a `GET` request to the following endpoint:

```text
HOST/api/schemas/vendor
```

```bash
curl HOST/api/schemas/com.acme -X GET -H "apikey: YOUR_APIKEY"
```

You will get back an array of every schema belonging to this vendor. You can use the same approach to get a list of schemas by vendor and name:

```text
HOST/api/schemas/vendor/name
```

```bash
curl HOST/api/schemas/com.acme/ad_click -X GET -H "apikey: YOUR_APIKEY"
```

Or simply retrieve every single public schema accessible to you:

```text
HOST/api/schemas
```

or `/api/schemas/public` in pre-0.5.0 releases.

```bash
curl HOST/api/schemas -X GET -H "apikey: YOUR_APIKEY"
```

_Please note:_ you can only retrieve schemas that can be read by your API key. This means that if you do not own a vendor you're requesting schemas for, you will only be able to retrieve the vendor's public schemas (if any exist).

### 1.5 Swagger support

We have added [Swagger](https://swagger.io/) support to our API so you can explore the repository server’s API interactively. The Swagger UI is available at the following URL:

```text
http://$HOST/static/swagger-ui/index.html
```

## 2. Schema validation and the validation service (`/api/validation`)

When adding a schema to the repository, the repository will validate that the provided schema is self-describing - an overview of this concept can be found in the [Self describing JSON schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) wiki page. In practice this means your schema should contain a `self` property, which itself contains the following properties:

- `vendor`
- `name`
- `format`
- `version`

Non-self-describing schemas can only be added to a repository using a PUT call to the schemas API that describes its vendor, name, format and version in the URL itself rather than the schema.

The Iglu Server's Validation service can also be used to validate that a schema is valid without adding it to the repository using the following endpoint:

```text
POST HOST/api/schemas/validation/validate/schema/format
```

```bash
curl HOST/api/validation/validate/schema/jsonschema -X POST -d @myevent.json
```

The response received will be a detailed report containing information about the schema's validity, as well as potential errors or warnings:

```json
{
  "message": "The schema has some issues",
  "report": [
    {
      "message": "The schema is missing the \"description\" property",
      "level": "INFO",
      "pointer": "/properties/targetUrl"
    },
    {
      "message": "A string type in the schema doesn't contain \"maxLength\" or format which is required",
      "level": "WARNING",
      "pointer": "/properties/targetUrl"
    },
    {
      "message": "The schema is missing the \"description\" property",
      "level": "INFO",
      "pointer": "/properties/clickId"
    },
    {
      "message": "A string type in the schema doesn't contain \"maxLength\" or format which is required",
      "level": "WARNING",
      "pointer": "/properties/clickId"
    },
    {
      "message": "Use \"type: null\" to indicate a field as optional for properties clickId",
      "level": "INFO",
      "pointer": "/"
    }
  ]
}
```

Another endpoint in the validation service allows you to validate self-describing _data_ against a schema already located in the Iglu Server repository, if it is accessible to your API key:

```text
POST HOST/api/schemas/validation/validate/instance
```

```bash
curl HOST/api/validation/validate/instance -X POST -H "apikey: YOUR_APIKEY" -d @myevent.json
```

The service will then either confirm the schema's validity:

```json
{
  "message": "Instance is valid iglu:com.acme/ad_click/jsonschema/1-0-0"
}
```

Or, if it has some issues, return a detailed report about its problems:

```json
{
  "message":"The instance is invalid against its schema",
  "report":[
    {
      "message": "$.targetUrl: must be at least 1 characters long",
      "path": "$.targetUrl",
      "keyword": "minLength",
      "targets": [ "1" ]
    }
  ]
}
```

Like the schema service, the validation service is also accessible through Swagger UI.

## 3. Drafts service (`/api/drafts`)

The draft schema service lets you interact with draft schemas using simple HTTP requests. Draft schemas are variants of Iglu schemas with simple versions that start with `1` and can be increased as needed.

### 3.1 POST requests

Sending a `POST` request to the draft service allows you to publish a new self-describing schema to the repository.

As an example, let's assume you own the `com.acme` vendor prefix (more information on that can be found in the API authentication section) and have a JSON schema defined as follows:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    "clickId": {
      "type": "string"
    },
    "targetUrl": {
      "type": "string",
      "minLength": 1
    }
  },
  "required": ["targetUrl"],
  "additionalProperties": false
}
```

The schema can be added to your repository as a draft by making a `POST` request to the following endpoint with the schema included in the request's body, and its vendor, name, format and desired draft number added to the request's URL:

```text
HOST/api/drafts/vendor/name/format/draftNumber
```

By default, the schema draft will not be public (available to others) - this can be changed by adding an `isPublic` query parameter and setting its value to `true`.

For example, the following request:

```bash
curl HOST/api/drafts/com.acme/ad_click/jsonschema/1 -X POST -H "apikey: YOUR_APIKEY" -d @myschema.json
```

will produce a response like this one, if no errors are encountered:

```json
{
  "message": "Schema created",
  "updated": false,
  "location": "iglu:com.acme/ad_click/jsonschema/1",
  "status":201
}
```

_Please note:_ This endpoint must be used with an API key with a `schema_action` permission of `CREATE`.

### 3.2 Single-draft GET requests

A schema draft previously added to the repository can be retrieved by making a `GET` request:

```bash
curl HOST/api/drafts/com.acme/ad_click/jsonschema/1 -X GET -H "apikey: YOUR_APIKEY"
```

The JSON response should look like this:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
  "clickId": {
    "type": "string"
  },
  "targetUrl": {
    "type": "string",
    "minLength": 1
  }
  },
  "required": ["targetUrl"],
  "additionalProperties": false
}
```

GET requests support a `repr` URL parameter, allowing you to specify three different ways of representing an Iglu schema. This can have values of either `canonical`, `meta` or `uri`. `repr=canonical` returns the schema as a self-describing Iglu event - it is also the default representation method if no query parameter is specified. (An example of this representation can be seen above.) `repr=meta` adds an additional `metadata` field to the schema, containing some meta information about it - this would make the JSON response look like this:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an Acme Inc ad click event",
  "self": {
    "vendor": "com.acme",
    "name": "ad_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
  "clickId": {
    "type": "string"
  },
  "targetUrl": {
    "type": "string",
    "minLength": 1
  }
  },
  "required": ["targetUrl"],
  "additionalProperties": false,
  "metadata": {
    "createdAt":"2019-04-01T14:23:45.173728Z",
    "updatedAt":"2019-04-01T14:23:45.173728Z",
    "isPublic":true
  }
}
```

`repr=uri` simply returns a schema's Iglu URI - this is used internally in the Iglu Server, but public requests are also supported. A response with this URL parameter set would look like this:

```text
"iglu:com.acme/ad_click/jsonschema/1-0-0"
```

_Please note:_ While `metadata`/`body` query parameters used in previous versions of the Iglu Server are supported, they have been deprecated in favor of the single `repr` parameter.

## 4. Debug (`/api/debug`) and metadata (`/api/meta`) services

The Iglu Server includes several endpoints for inspecting its own state.

The `/api/debug` endpoint is only active when `debug` is set to true in the Iglu Server's configuration file, and returns the Iglu Server's internal state if in-memory storage is used instead of PostgreSQL. **This endpoint should be used for internal development and testing only!**

The `/api/meta/health` endpoint will respond with a simple OK string if the server is available:

```bash
curl HOST/api/meta/health
OK
```

The `/api/meta/health/db` endpoint is similar, but will also check if the database is accessible if PostgreSQL storage is used (in-memory storage is always accessible):

```bash
curl HOST/api/meta/health/db
OK
```

The `/api/meta/server` endpoint returns information about the server - its version, database type, certain configuration settings etc. If an `apiKey` header is included in the request, it will also return information about the key's permissions and the number of schemas accessible to it:

```bash
curl HOST/api/meta/server -H 'apikey: YOUR_APIKEY'
{
  "version": "0.6.0",
  "authInfo": {
    "vendor": "",
    "schema": "CREATE_VENDOR",
    "key": [ "CREATE", "DELETE" ]
  },
  "database": "postgres",
  "schemaCount": 18,
  "debug": true,
  "patchesAllowed": false
}
```

## 5. API keys and the authentication service (`/api/auth`)

In order to use the routes of the Iglu Server's API that require either write access to the repository or readaccess to non-public schemas, you will need an API key, passed as an `apiKey` HTTP header to relevant calls of those services.

The Iglu Server's administrator is responsible for distributing API keys to the repository's clients. To do so, they will need a super API key which will let them generate other keys - this key will have to be manually added to the database:

```sql
INSERT INTO permissions
VALUES ('api_key_here', '', TRUE, 'CREATE_VENDOR'::schema_action, '{"CREATE", "DELETE"}'::key_action[])
```

Thanks to this super API key the server's administrator be able to use the API key generation service, which lets them create and revoke API keys. A pair of API keys for a given vendor can be generated by submitting a POST request to the keygen endpoint, with the prefix included in the request's body:

```text
POST HOST/api/auth/keygen
```

```bash
curl HOST/api/auth/keygen -X POST -H 'apikey: ADMIN_APIKEY' -H "Content-Type: application/json" -d '{"vendorPrefix": "com.acme"}'
```

If no errors occur, the method should return two UUIDs that act as API keys for the given vendor:

```json
{
  "read":"bfa90866-aa14-4b92-b6ef-d421fd688b54",
  "write":"6175aa41-d3b7-4e4f-9fb4-3a170f3c6c16"
}
```

One of those API keys will have read access and will let you retrieve private schemas or drafts (or other vendors' public schemas) through `GET` requests. The other will have both read and write access - you will therefore be able to publish and modify schemas or drafts through `POST` and `PUT` requests in addition to being able to retrieve them. It is then up to you on to distribute those two keys however you want. The keys grant access to every schema whose vendor starts with `com.acme`, though wildcard (`*`) vendor keys can also be generated.

Existing API keys can be revoked by sending a `DELETE` request to the same endpoint:

```text
DELETE HOST/api/auth/keygen
```

```bash
curl HOST/api/auth/keygen?key=APIKEY_TO_DELETE -X DELETE -H 'apikey: ADMIN_APIKEY'
```

If the operation succeeds, it will return a simple JSON response:

```json
{
  "message":"Keys have been deleted"
}
```

---

# Iglu Server configuration reference
> Complete reference of all configuration options for Iglu Server, including database, networking, webhooks, and advanced settings.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/

This is a complete list of the options that can be configured in the Iglu Server HOCON config file. The [example configs in github](https://github.com/snowplow-incubator/iglu-server/tree/master/config) show how to prepare an input file.

## License

Iglu Server is released under the [Snowplow Limited Use License](https://docs.snowplow.io/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Iglu Server, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

## Common options

| parameter                                          | description                                                                                                                                                                                                                                                                              |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `repoServer.interface`                             | Optional. Default: `0.0.0.0`. Address on which the server listens to http connections.                                                                                                                                                                                                   |
| `repoServer.port`                                  | Optional. Default: `8080`. Port on which the server listens.                                                                                                                                                                                                                             |
| `repoServer.idleTimeout`                           | Default: `30 seconds`. TCP connections are dropped after this timeout expires. In case Iglu Server runs behind a load balancer, this should slightly exceed the load balancer's idle timeout.                                                                                            |
| `repoServer.hsts.enable` _(since 0.12.0)_          | Default: `false`. Whether to send an [HSTS header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security).                                                                                                                                                 |
| `repoServer.hsts.maxAge` _(since 0.12.0)_          | Default: `365 days`. The maximum age for the HSTS header.                                                                                                                                                                                                                                |
| `database.type`                                    | Optional. Default: `postgres`. Can be changed to `dummy` during development for in-memory only storage.                                                                                                                                                                                  |
| `database.host`                                    | Required. Host name for Postgres database.                                                                                                                                                                                                                                               |
| `database.port`                                    | Optional. Default: `5432`. Port for Postgres database.                                                                                                                                                                                                                                   |
| `database.dbname`                                  | Required. Name of Postgres database.                                                                                                                                                                                                                                                     |
| `database.username`                                | Required. Username for connecting to Postgres.                                                                                                                                                                                                                                           |
| `database.password`                                | Required. Password for connecting to Postgres.                                                                                                                                                                                                                                           |
| `swagger.baseUrl`                                  | Optional. Example: `/custom/prefix`. Customise the api base url in Swagger. Helpful for when running iglu-server behind a proxy server.                                                                                                                                                  |
| `debug`                                            | Optional. Default: `false`. Enable additional debug api endpoint to respond with all internal state.                                                                                                                                                                                     |
| `patchesAllowed`                                   | Optional. Default: `false`. If `true`, allows overwriting a given version of a schema with new content. See [amending schemas](/docs/fundamentals/schemas/versioning/).                                                                                                                  |
| `webhooks.schemaPublished`                         | Optional. Array with the list of webhooks that will be called when a schema is published or updated with a vendor that matches the specified prefixes. See the [examples in github](https://github.com/snowplow-incubator/iglu-server/blob/0.8.7/config/config.reference.hocon#L81-L99). |
| `webhooks.schemaPublished.uri`                     | Required. URI of the HTTP server that will receive the webhook event.                                                                                                                                                                                                                    |
| `webhooks.schemaPublished.vendorPrefixes`          | Optional. Example: `["com", "org.acme", "org.snowplow"]`. List of schema prefixes (regexes) that should be sent via the webhook.                                                                                                                                                         |
| `webhooks.schemaPublished.usePost` (since _0.8.7_) | Optional. Default: `false`. Whether to use `POST` to send request via the webhook.                                                                                                                                                                                                       |
| `superApiKey`                                      | Optional. Set a super api key with permission to read/write any schema, and add other api keys.                                                                                                                                                                                          |

## Advanced options

We believe these advanced options are set to sensible defaults, and hopefully you won’t need to ever change them.

| parameter                                 | description                                                                                                                                                                                                                 |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `repoServer.threadPool.type`              | Default: `fixed` for a fixed thread pool. Can be `cached` for a cached thread pool. Type of the thread pool used by the underlying BlazeServer for executing Futures.                                                       |
| `repoServer.threadPool.size`              | Optional. Default: `4`. Size of the thread pool if the type is `fixed`.                                                                                                                                                     |
| `repoServer.maxConnections`               | Optional. Default: `1024`. Maximum number of client connections that may be active at any time.                                                                                                                             |
| `database.pool.type`                      | Optional. Default: `hikari` (recommended for production). Can be changed to `nopool` to remove the upper bound on the number of connections.                                                                                |
| `database.pool.maximumPoolSize`           | Optional. Default: `5`. Maximum number of connections in the Hikari pool.                                                                                                                                                   |
| `database.pool.connectionTimeout`         | Optional. Default: `30 seconds`. Timeout on the Hikari connection pool.                                                                                                                                                     |
| `database.pool.maxLifetime`               | Optional. Default: `1800 seconds`. Maximum lifetime of a connection in the Hikari pool.                                                                                                                                     |
| `database.pool.minimumIdle`               | Optional. Default: `5`. Minimum number of idle connections in the Hikari pool.                                                                                                                                              |
| `database.pool.connectionPool.type`       | Optional. Default: `fixed` for a fixed thread pool (recommended in production). Type of the thread pool used for awaiting connection to the database.                                                                       |
| `database.pool.connectionPool.size`       | Optional. Default: `4`. Number of threads to use when the connection pool has type `fixed`.                                                                                                                                 |
| `database.pool.transactionPool.type`      | Optional. Default: `cached` (recommended for production). Type of the thread pool used for blocking JDBC operations.                                                                                                        |
| `preTerminationPeriod` (since _0.8.0_)    | Optional. Default: `1 second`. How long the server should pause after receiving a sigterm before starting the graceful shutdown. During this period the server continues to accept new connections and respond to requests. |
| `preTerminationUnhealthy` (since _0.8.0_) | Optional. Default: `false`. During the `preTerminationPeriod`, the server can be configured to return 503s on the `/health` endpoint. Can be helpful for removing the server from a load balancer’s targets.                |

---

# Setup guide for Iglu Server
> Deploy Iglu Server with Docker or Terraform for PostgreSQL-backed schema repository with RESTful API and authentication.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/iglu-server/setup/

For more information on the architecture of the Iglu server, please read [the technical documentation](/docs/api-reference/iglu/iglu-repositories/iglu-server/).

## Available on Terraform Registry

A Terraform module is available which deploys an Iglu Server on AWS EC2 without the need for this manual setup.

## 1. Run the Iglu server

Iglu Server is [published on Docker Hub](https://hub.docker.com/repository/docker/snowplow/iglu-server).

```bash
$ docker pull snowplow/iglu-server:0.14.1
```

The application is configured by passing a hocon file on the command line:

```bash
$ docker run --rm \
-v $PWD/config.hocon:/iglu/config.hocon \
snowplow/iglu-server:0.14.1 --config /iglu/config.hocon
```

Alternatively, you can download and run [a jar file from the github release](https://github.com/snowplow-incubator/iglu-server/releases).

```bash
$ java -jar iglu-server-0.14.1.jar --config /path/to/config.hocon
```

Here is an example of a minimal configuration file:

```json
{
  "database": {
    "host": "postgres"
    "dbname": "igludb"
    "username": "postgres"
    "password": "mysecret"
  }

  "superApiKey": "bb7b7503-40d3-459c-943a-f8d31a6f5638"
}
```

See [the configuration reference](/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/) for a complete description of all parameters.

We also provide a [docker-compose.yml](https://github.com/snowplow-incubator/iglu-server/blob/master/docker/docker-compose.yml) to help you get started.

## 2. Initialize the database

> **Note:** Iglu Server has been successfully tested with PostgreSQL 16.3, but should work with PostgreSQL 8.2 or newer.

With a fresh install you need to manually create the database:

```bash
$ psql -U postgres -c "CREATE DATABASE igludb"
```

And then use the `setup` command of the iglu server to create the database tables:

```bash
$ docker run --rm \
-v $PWD/config.hocon:/iglu/config.hocon \
snowplow/iglu-server:0.14.1 setup --config /iglu/config.hocon
```

## 3. Use the API key generation service

The super API key you put in the configuration file is able to generate further API keys for your clients through the API key generation service.

To generate a pair of read and write API keys for a specific vendor prefix, simply send a `POST` request to this URL using your super API key in an `apikey` HTTP header:

```text
HOST/api/auth/keygen
```

For example:

```bash
curl \
  HOST/api/auth/keygen \
  -X POST \
  -H 'apikey: your_super_apikey' \
  -d '{"vendorPrefix":"com.acme"}'
```

**Note:** From 0.6.0+ the vendor prefix should be `vendorPrefix` within a JSON body however prior to this it was `vendor_prefix` as a query parameter.

You should receive a JSON response like this one:

```json
{
  "read": "an-uuid",
  "write": "another-uuid"
}
```

If you need to revoke a specific API key, you can do so by sending a `DELETE` request to the following endpoint:

```text
HOST/api/auth/keygen?key=some-uuid
```

For example:

```bash
curl \
  HOST/api/auth/keygen \
  -X DELETE \
  -H 'apikey: your_super_apikey' \
  -d 'key=some-uuid'
```

You should now be all set up to use the Iglu server, if you would like to know more about the Iglu server, please read the [technical documentation](/docs/api-reference/iglu/iglu-repositories/iglu-server/).

## Dummy mode

Since 0.6.0 Iglu Server supports new dummy DB mode. In this mode, Server does not require persistent storage as PostgreSQL and stores all data in memory. Use this for debug purposes only, all your data will be lost after restart.

To enable dummy mode, you need to set `database.type` setting to `"dummy"`.

Dummy Iglu Server works with single hardcoded master API key - `48b267d7-cd2b-4f22-bae4-0f002008b5ad`, which you can use to upload your schemas and create new api keys.

## Logging

Iglu Server uses [SLF4J Simple Logger](https://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html) underneath. Which can be configured via system properties.

For example:

```bash
$ iglu-server-0.14.1.jar \
-Dorg.slf4j.simpleLogger.logFile=server.log                                   # In order to redirect logs \
-Dorg.slf4j.simpleLogger.log.org.http4s.blaze.channel.nio1.SelectorLoop=warn  # To suppress very verbose SelectorLoop output
```

On debug loglevel `SchemaService` will print all HTTP requests and responses.

---

# Introduction to Iglu repositories for schema storage
> Remote and embedded Iglu repositories for storing and serving JSON schemas via HTTP or embedded in JVM applications.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/

An Iglu repository acts as a store of data schemas (currently JSON Schemas only). Hosting JSON Schemas in an Iglu repository allows you to use those schemas in Iglu-capable systems such as Snowplow.

## Technical architecture

So far we support two types of Iglu repository:

1. **Remote repositories** - essentially websites containing schemas which an Iglu client can query over HTTP
2. **Embedded repositories** - which are embedded in a piece of software (typically alongside an Iglu client)

In this diagram we show an Iglu client resolving a schema from Iglu Central, one embedded repository and a further two remote HTTP repositories:

![iglu-repos-img](/assets/images/iglu-repos-82e5f47255a46f97fe0b46b8abe90934.png)

## Available Iglu repositories

We currently have two Iglu "repo" technologies available for deploying your Iglu repository - follow the links to find out more:

| **Repository**    | **Category** | **Description**                                            | **Status**       |
| ----------------- | ------------ | ---------------------------------------------------------- | ---------------- |
| Iglu Server       | Remote       | An Iglu repository server structured as a RESTful API      | Production-ready |
| Static repo       | Remote       | An Iglu repository server structured as a static website   | Production-ready |
| JVM-embedded repo | Embedded     | An Iglu repository embedded in a Java or Scala application | Production-ready |

## Iglu Central

[Iglu Central](https://iglucentral.com/) is a public repository of JSON Schemas hosted by [Snowplow Analytics](http://snowplowanalytics.com/). For more information on its technical architecture, see [Iglu Central](/docs/api-reference/iglu/iglu-central-setup/).

---

# JVM-embedded Iglu repository for applications
> Embed JSON schemas in Java or Scala application resources for bootstrap schema resolution without remote dependencies.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/jvm-embedded-repo/

A JVM-embedded repo is an Iglu repository **embedded** inside a Java or Scala application, typically alongside the [Scala client](/docs/api-reference/analytics-sdk/analytics-sdk-scala/).

## Technical architecture

A JVM-embedded repo is simply a set of schemas stored in an Iglu-compatible path inside the `resources` folder of a Java or Scala application.

As an embedded repo, there is a no mechanism for updating the schemas stored in the repository following the release of the host application.

## Example

For an example of a JVM-embedded repo, check out the repository embedded in the Iglu Scala client itself:

<https://github.com/snowplow/iglu-scala-client/tree/0.1.0/src/main/resources/iglu-client-embedded>

This embedded repository is used to bootstrap the Iglu Scala client with JSON Schemas that it needs before it can access any remote repositories.

## Setup

### 1. Prepare your files

You need to create a file structure for your JSON Schemas. Please check out the template we provide here:

<https://github.com/snowplow/iglu/tree/master/2-repositories/jvm-embedded-repo/template>

Make the following changes:

- Replace `com.myvendor` with your company domain, reverse-ordered
- Replace `myschema` with the name of your first JSON Schema
- Leave `jsonschema` as-is (we only support JSON Schemas for now)
- Replace `1-0-0` with the schema specification of your first JSON Schema

Writing JSON Schemas is out of scope for this setup guide - see [Self-describing-JSONs-and-JSON-Schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) for details.

Done? Now you are ready to embed your files.

### 2. Embed your files

You now need to embed your JSON Schema files into your Java or Scala application.

The Iglu Scala client will expect to find these JSON Schema files included in the application as resources. Therefore, you should store the files in a path something like this:

```text
myapp/src/main/resources/my-repo/schemas
```

### 3. Update your Iglu client configuration

Finally, update your Iglu client configuration so that it can resolve your new repository.

For details on how to do this, check out the page on [Iglu client configuration](/docs/api-reference/iglu/iglu-resolver/). In the case above, the `path` you would specify for your embedded Iglu repository would be simply `/my-repo`.

---

# Static Iglu repository on web servers
> Host Iglu schemas as a static website on S3, Apache, Nginx, or IIS for HTTP-accessible schema repositories.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-repositories/static-repo/

A static repo is simply an Iglu repository server structured as a static website. [Iglu Central](/docs/api-reference/iglu/iglu-central-setup/) can be used as an example, [serving](https://iglucentral.com/) its whole content over HTTP.

## 1. Choose a hosting partner

We host static Iglu registry using Amazon S3, but you can choose any existing webserver your company is already using, such as Apache, IIS or Nginx.

## 2. Prepare your files

You need to create a file structure for your JSON Schemas. Please check out the template we provide here:

<https://github.com/snowplow/iglu/tree/master/2-repositories/static-registry/template>

Make the following changes:

- Replace `com.myvendor` with your company domain, reverse-ordered
- Replace `myschema` with the name of your first JSON Schema
- Leave `jsonschema` as-is (we only support JSON Schemas for now)
- Replace `1-0-0` with the schema specification of your first JSON Schema

Writing JSON Schemas is out of scope for this setup guide - see [Self-describing-JSONs-and-JSON-Schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) for details.

Done? Now you are ready to host your files.

## 3. Host the files in your schema registry

To host your static schema registry, follow the AWS guide, [Host a Static Website on Amazon Web Services](http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html).

To host your static schema on an alternative webserving platform, please consult the appropriate webserver documentation or talk to your Systems team.

## 4. Update your Iglu client configuration

Finally, update your Iglu client configuration so that it can resolve your new registry.

For details on how to do this, check out the page on [Iglu client configuration](/docs/api-reference/iglu/iglu-resolver/).

---

# Iglu Resolver configuration for Snowplow applications
> Configure Iglu Resolver for schema fetching and validation in Snowplow enrichers and loaders with cache and repository settings.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/iglu-resolver/

Iglu Resolver is a component embedded into many Snowplow applications, including enrichers and loaders. It's responsible for fetching schemas from Iglu registries and validating data against these schemas.

Most of the time, configuring Iglu Resolver (or Client) means adding following JSON file:

```json
{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-3",
  "data": {
    "cacheSize": 500,
    "cacheTtl": 600,
    "repositories": [
      {
        "name": "Iglu Central",
        "priority": 0,
        "vendorPrefixes": [ "com.snowplowanalytics" ],
        "connection": {
          "http": {
            "uri": "https://iglucentral.com"
          }
        }
      },
      {
        "name": "Custom Iglu Server",
        "priority": 0,
        "vendorPrefixes": [ "com.snowplowanalytics" ],
        "connection": {
          "http": {
            "uri": "https://${iglu_server_hostname}/api",
            "apikey": "${iglu_server_apikey}"
          }
        }
      }
    ]
  }
}
```

The above configuration assumes Snowplow-authored schemas (Iglu Central) will be used in a pipeline, and that you have your own registry (Iglu Server) being hosted at `https://${iglu_server_hostname}/` with an API Key, `${iglu_server_apikey}`, with read rights.

### Configuration parameters

- `cacheSize` determines how many individual schemas we will keep cached in our Iglu client (to save additional lookups)
- `cacheTtl` determines how long a schema can live in the cache before being reloaded (in seconds)
- `repositories` is a JSON array of repositories to look up schemas in
- `priority` and `vendorPrefixes` help the resolver to know which repository to check first for a given schema. For details see Iglu's [repository resolution algorithm](/docs/api-reference/iglu/common-architecture/schema-resolution/#3-registry-priority)

---

# Igluctl CLI for schema management
> Command-line tool for validating, publishing, and managing JSON schemas in Iglu registries with DDL generation and verification.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/igluctl-2/

Iglu is a schema repository for JSON Schema. A schema repository (sometimes called a registry) is like npm or Maven or git but holds data schemas instead of software or code. Iglu is used extensively in Snowplow.

This document is for version 0.13.0.

## Igluctl

Iglu provides a CLI application, called igluctl which allows you to perform most common tasks on Iglu registry. So far, the overall structure of igluctl commands looks like the following:

- `lint` - validate set of JSON Schemas for syntax and consistency of their properties

- `static` - work with static Iglu registry

  - `generate` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for Redshift and Postgres warehouses. Generate DDLs and migrations from set of JSON Schemas. If the schema is not evolved correctly and backward incompatible data is sent within transformer's aggregation window, loading would fail for all events.
  - `push` - push set of JSON Schemas from static registry to full-featured (Scala Registry for example) one
  - `pull` - pull set of JSON Schemas from registry to local folder
  - `deploy` - run entire schema workflow using a config file. This could be used to chain multiple commands, i.e. `lint` followed by `push` and `s3cp`.
  - `s3cp` - copy JSONPaths or schemas to S3 bucket

- `server` - work with an Iglu server
  - `keygen` - generate read and write API keys on Iglu Server

- `table-check` - will check a given Redshift or Postgres tables against iglu server.

- `verify` (since 0.13.0) - work with schemas to check their evolution

  - `redshift` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for loading into Redshift. It reports the major schema versions within which schema evolution rules were broken.
  - `parquet` - verify that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for parquet transformation (for loading into Databricks). It reports the breaking schema versions.

## Downloading and running Igluctl

Download the latest Igluctl from GitHub releases and unzip the file:

```bash
$ wget https://github.com/snowplow/igluctl/releases/download/0.13.0/igluctl_0.13.0.zip
$ unzip igluctl_0.13.0.zip
```

To run Igluctl you can, for example, can pass the `--help` option to see information on the different commands and flags like this:

```bash
$ ./igluctl --help
```

> **Note:** If you are on Windows, then you'll need to run Igluctl like this:
>
> ```bash
> $ java -jar igluctl --help
> ```
>
> Below and everywhere in documentation you'll find example commands without this `java -jar` prefix, so please remember to add it when running Igluctl.

Note that Igluctl expects [JRE 8](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html) or later, and [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or later to run.

## lint

`igluctl lint` validates JSON Schemas.

It is designed to be run against file-based schema registries with the standard Iglu folder structure:

```text
schemas
└── com.example
    └── my-schema
        └── jsonschema
            ├── 1-0-0
            └── 1-0-1
```

You can validate _all_ the schemas in the registry:

```bash
$ /path/to/igluctl lint /path/to/schema/registry/schemas
```

Alternatively you can validate an individual schema e.g.:

```bash
$ /path/to/igluctl lint /path/to/schema/registry/schemas/com.example_company/example_event/jsonschema/1-0-0
```

Examples of errors that are identified:

- JSON Schema has inconsistent self-describing information and path on filesystem
- JSON Schema has invalid `$schema` keyword. It should be always set to [iglu-specific](http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#), while users tend to set it to Draft v4 or even to self-referencing Iglu URI
- JSON Schema is invalid against its standard (empty `required`, string `maximum` and similar)
- JSON Schema contains properties which contradict each other, like `{"type": "integer", "maxLength": 0}` or `{"maximum": 0, "minimum": 10'}`. These schemas are inherently useless as for some valiators there is no JSON instance they can validate

The above cases can very hard to spot without a specialized tool as they are still valid JSONs and in last case it is even valid JSON Schemas - so will validate against a standard JSON schema validator.

`lint` has two options:

- `--skip-checks` which will lint without specified linters, given comma separated. To see available linters and their explanations, `$ /path/to/igluctl --help`
- `--skip-schemas` which will lint all the schemas except the schemas passed to this option as a comma separated list. For example running: `/path/to/igluctl lint /path/to/schema/registry/schemas --skip-schemas iglu:com.acme/click/jsonschema/1-0-1,iglu:com.acme/scroll/jsonschema/1-0-1` will lint all schemas in `/path/to/schema/registry/schemas` except the two schemas passed via `--skip-schemas`.

Note: `--severityLevel` option is deprecated and removed as of version 0.4.0.

Below are two groups of linters; allowed to be skipped and not allowed to be skipped. By default, all of them are enabled but igluctl users can skip any combination of `rootObject`, `unknownFormats`, `numericMinMax`, `stringLength`, `optionalNull`, `description` through `--skip-checks`.

Igluctl let you skip below checks:

| NAME             | DEFINITION                                                                                         |
| ---------------- | -------------------------------------------------------------------------------------------------- |
| `rootObject`     | Check that root of schema has object type and contains properties                                  |
| `unknownFormats` | Check that schema doesn’t contain unknown formats                                                  |
| `numericMinMax`  | Check that schema with numeric type contains both minimum and maximum properties                   |
| `stringLength`   | Check that schema with string type contains maxLength property or other ways to extract max length |
| `optionalNull`   | Check that non-required fields have null type                                                      |
| `description`    | Check that property contains description                                                           |

A sample usage could be as following:

```bash
$ ./igluctl lint --skip-checks description,rootObject /path/to/schema/registry/schemas
```

Note that linter names are case sensitive

Igluctl also includes many checks proving that schemas doesn’t have conflicting expectations (such as `minimum` value bigger than `maximum`). Schemas with such expectations are valid according to specification, but do not make any sense in real-world use cases. These checks are mandatory and cannot be disabled.

`igluctl lint` will exit with status code 1 if encounter at least one error.

## static generate

`igluctl static generate` generates corresponding [Redshift](http://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html) DDL files (`CREATE TABLE` statements) and migration scripts (`ALTER TABLE` statements).

As of version 0.11.0 this command will also validate the compatibility of schema family and display warnings if there is an incompatible evolution.

```bash
$ ./igluctl static generate $INPUT
```

You also can specify directory for output (current dir is used as default):

```bash
$ ./igluctl static generate --output $DDL_DIR $INPUT
```

### Generating migration Redshift table scripts to accommodate updated schema versions

If an input directory is specified with several self-describing JSON schemas with a single REVISION, Igluctl will generate migration scripts to update (`ALTER`) Redshift tables for older schema versions to support the latest schema version.

For example, having the following Self-describing JSON Schemas as an input:

- schemas/com.acme/click\_event/1-0-0
- schemas/com.acme/click\_event/1-0-1
- schemas/com.acme/click\_event/1-0-2

Igluctl will generate the following migration scripts:

- sql/com.acme/click\_event/1-0-0/1-0-1 to alter table from 1-0-0 to 1-0-1
- sql/com.acme/click\_event/1-0-0/1-0-2 to alter table from 1-0-0 to 1-0-2
- sql/com.acme/click\_event/1-0-1/1-0-2 to alter table from 1-0-1 to 1-0-2

This migrations (and all subsequent table definitions) are aware of column order and will ensure that new columns are added at the end of the table definition. This means that the tables can be updated in-place with single `ALTER TABLE` statements.

### Handling union types

One of the more problematic scenarios to handle when generating Redshift table definitions is handling `UNION` field types e.g. `["integer", "string"]`. Union types will be transformed as most general. In the above example (union of an integer and string type) the corresponding Redshift column will be a `VARCHAR(4096)`.

### Missing schema versions

`static generate` command will check versions of schemas inside `input` as following:

- If user specified folder and one of schemas has no 1-0-0 or misses any other schemas in between (like it has 1-0-0 and 1-0-2) - refuse to do anything (but proceed with –force option)
- If user specified full path to file with schema and this file is not 1-0-0 - just print a warning
- If user specified full path to file with schema and it is 1-0-0 - all good

## static push

`igluctl static push` publishes schemas stored locally to a remote [Iglu Server](https://github.com/snowplow/iglu-server).

It accepts three required arguments:

- `host` - Iglu Server host name or IP address with optional port and endpoint. It should conform to the pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys
- `path` - path to your static registry (local folder containing schemas)

Also it accepts optional `--public` argument which makes schemas available without `apikey` header.

```bash
$ ./igluctl static push /path/to/static/registry iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## static pull

`igluctl static pull` downloads schemas stored on a remote [](https://github.com/snowplow/iglu/tree/master/2-repositories/iglu-server)[Iglu Server](https://github.com/snowplow/iglu-server) to a local folder.

It accepts three required arguments:

- `host` - Scala Iglu Registry host name or IP address with optional port and endpoint. It should conform to the pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys
- `path` - path to your static registry (local folder to download to)

```bash
$ ./igluctl static pull /path/to/static/registry iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## static s3cp

`igluctl static s3cp` enables you to upload JSON Schemas to chosen S3 bucket. This is helpful for generating a remote iglu registry which can be served from S3 over http(s).

`igluctl static s3cp` accepts two required arguments and several options:

- `input` - path to your files. Required.
- `bucket` - S3 bucket name. Required.
- `s3path` - optional S3 path to prepend your input root. Usually you don’t need it.
- `accessKeyId` - your AWS Access Key Id. This may or or may not be required, depending on your preferred authentication option.
- `secretAccessKey` - your AWS Secret Access Key. This may or or may not be required, depending on your preferred authentication option.
- `profile` - your AWS profile name. This may or or may not be required, depending on your preferred authentication option.
- `region` - AWS S3 region. Default: `us-west-2`
- `skip-schema-lists` - Do not generate and upload schema list objects. If using a static registry for all Snowplow applications, don’t enable this setting as some components still require lists to function correctly.

`igluctl static s3cp` tries to closely follow AWS CLI authentication process. First it checks if profile name or `accessKeyId`/`secretAccessKey` pair provided and uses it. If neither of above provided - it looks into `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` environment variables. If above aren’t available as well - it `~/.aws/config` file. If all above failed - it exits with error.

## static deploy

`igluctl static deploy` performs whole schema workflow at once.

It accepts one required arguments:

- `config` - Path to configuration file

```bash
$ ./igluctl static deploy /path/to/config/file
```

Your configuration file should be a hocon file, following the [reference example](https://github.com/snowplow/igluctl/blob/0.8.0/config/deploy.reference.hocon). For backwards compatibility with previous versions, you could also provide a [self-describing json](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.iglu/igluctl_config/jsonschema/1-0-0).

Example:

```json
  {
    "lint": {
      "skipWarnings": true
      "includedChecks": [
        "rootObject"
        "unknownFormats"
        "numericMinMax"
        "stringLength"
        "optionalNull"
        "description"
        "stringMaxLengthRange"
      ]
    }

    "generate": {
      "dbschema": "atomic"
      "force": false
    }


    "actions": [
      {
        "action": "push"
        "isPublic": true
        "apikey": "bd96b5ff-7eb7-4085-83e0-97ac4954b891"
        "apikey": ${APIKEY_1}
      }
      {
        "action": "s3cp"
        "uploadFormat": "jsonschema"
        "profile": "profile-1"
        "region": "eu-east-2"
      }
    ]

}
```

## server keygen

`igluctl server keygen` generates read and write API keys on Iglu Server.

It accepts two required arguments:

- `host` - Scala Iglu Registry host name or IP address with optional port and endpoint. It should conform pattern `host:port/path` (or just `host`) **without** http\:// prefix.
- `apikey` - master API key, used to create temporary write and read keys

Also it accepts `--vendor-prefix` argument which will be associated with generated key.

```bash
$ ./igluctl server keygen --vendor-prefix com.acme iglu.acme.com:80/iglu-server f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

## table-check

`igluctl table-check` will check given RedShift or Postgres schema against iglu repository. As of version 0.11.0 it would cross verify the column types as well as names.

It supports two interfaces:

- `igluctl table-check --server <uri>` to check all tables
- `igluctl table-check --resolver <path> --schema <schemaKey>` to check particular table

It also accepts a number of arguments:

```bash
--resolver <path>
    Iglu resolver config path
--schema <schemaKey>
    Schema to check against. It should have iglu:<URI> format
--server <uri>
    Iglu Server URL
--apikey <uuid>
    Iglu Server Read ApiKey (non master)
--dbschema <string>
    Database schema
--host <string>
    Database host address
--port <integer>
    Database port
--dbname <string>
    Database name
--username <string>
    Database username
--password <string>
    Database password
```

```bash
$ ./igluctl table-check --resolver <path> --schema <schemaKey> ...connection parameters
```

or

```bash
$ ./igluctl table-check --server <uri> ...connection params
```

## verify parquet

`igluctl verify parquet` verifies that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for parquet transformation (for loading into Databricks). It reports the breaking schema versions.

It accepts one required arguments:

- `input` - path to your schema files.

Example command:

```bash
$ ./igluctl verify parquet /path/to/static/registry
```

Example output:

```text
Breaking change introduced by 'com.acme/product/jsonschema/1-0-2'. Changes: Incompatible type change Long to Double at /item/price
Breaking change introduced by 'com.acme/user/jsonschema/1-0-1'. Changes: Incompatible type change Long to Integer at /id
Breaking change introduced by 'com.acme/item/jsonschema/1-1-0'. Changes: Incompatible type change String to Json at /metadata
```

## verify redshift

`igluctl verify redshift` verifies that schema is evolved correctly within the same major version (e.g. from `1-a-b` to `1-c-d`) for loading into Redshift. It reports the major schema versions within which schema evolution rules were broken.

It accepts two required and one optional arguments:

- `server` - Iglu Server URL.
- `apikey` - Iglu Server Read ApiKey
- `--verbose/-v` - emit detailed report or not (disabled by default)

Example command:

```bash
$ ./igluctl verify redshift --server iglu.acme.com --apikey f81d4fae-7dec-11d0-a765-00a0c91e6bf6
```

Example output:

```text
iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-*-*
iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-*-*
iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-*-*
```

Example command with verbose output:

```bash
$ ./igluctl verify redshift --server iglu.acme.com --apikey f81d4fae-7dec-11d0-a765-00a0c91e6bf6 --verbose
```

Example output:

```text
iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-*-*:
  iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2: [Incompatible types in column cache_size old RedshiftBigInt new RedshiftDouble]
  iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-3: [Incompatible types in column cache_size old RedshiftBigInt new RedshiftDouble]
iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-*-*:
  iglu:com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-0-1: [Incompatible types in column parameters.hash_algorithm old RedshiftChar(3) new RedshiftVarchar(6)]
iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-*-*:
  iglu:com.snowplowanalytics.snowplow.badrows/loader_runtime_error/jsonschema/1-0-1: [Making required column nullable error]
```

---

# Introduction to the Iglu schema registry
> Machine-readable schema registry for JSON and Thrift schemas with self-describing JSON support, schema versioning, and distributed repositories.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/

**Iglu** is a machine-readable schema registry for [JSON](http://json-schema.org/) and Thrift schema. A schema registry is like npm, Maven, or Git but holds data schemas instead of software or code.

Iglu consists of three key technical aspects:

1. A [common architecture](/docs/api-reference/iglu/common-architecture/) that informs all aspects of Iglu
2. [Iglu registries](/docs/api-reference/iglu/iglu-repositories/) that can host a set of [self-describing JSON schemas](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/)
3. [Iglu clients](/docs/api-reference/iglu/iglu-clients/) that can resolve schemas from one or more Iglu registries

## Iglu explained

**Iglu** is built on a set of technical design decisions that allow Iglu clients and registries to interoperate. The key design components are:

- [Self-describing JSON schema](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/): extensions to JSON schema that semantically identify and version a given JSON schema
- [Self-describing JSON](/docs/api-reference/iglu/common-architecture/self-describing-jsons/): a standardized JSON format which co-locates a reference to the instance's JSON schema alongside the instance's data
- [SchemaVer](/docs/api-reference/iglu/common-architecture/schemaver/): how we semantically version schemas
- [Schema resolution](/docs/api-reference/iglu/common-architecture/schema-resolution/): our public algorithm for how we determine in which order we check Iglu registries for a given schema

**Iglu clients** are used for interacting with Iglu server repos and for resolving schemas in embedded and remote Iglu schema registries.

In the below diagram we show an Iglu client resolving a schema from Iglu Central, one embedded registry and a further two remote HTTP registries:

![Iglu client](/assets/images/iglu-clients-2a639a6f765d5146f869eb947a42f15c.png)

An **Iglu registry** acts as a store of data schemas. Hosting JSON schemas in an Iglu registry allows you to use those schemas in Iglu-capable systems such as Snowplow.

So far we support two types of Iglu registry:

- **Remote registries** - essentially websites containing schemas which an Iglu client can query over HTTP
- **Embedded registries** - which are embedded in a piece of software (typically alongside an Iglu client)

In the below diagram we show an Iglu client resolving a schema from Iglu Central, one embedded registry and a further two remote HTTP registries:

![Iglu repositories](/assets/images/iglu-repos-82e5f47255a46f97fe0b46b8abe90934.png)

**Iglu Central** ([https://iglucentral.com](https://iglucentral.com/)) is a public registry of Snowplow JSON schemas.

Under the covers, Iglu Central is built and run as a **static Iglu registry**, hosted on Amazon S3.

> A **static repo** is simply an Iglu registry server structured as a static website.

![Iglu Central](/assets/images/iglu-central-c0427b712c8c80ad53d1a8a2b7e6871d.png)

The **deployment process** for Iglu Central is documented in [Iglu Central setup](/docs/api-reference/iglu/iglu-central-setup/) in case you want to set up a public mirror or private instance of Iglu Central.

---

# Manage schemas using Iglu Server
> Manage schemas with Iglu Server or host a static Iglu registry in Amazon S3 or Google Cloud Storage for self-hosted Snowplow deployments.
> Source: https://docs.snowplow.io/docs/api-reference/iglu/manage-schemas/

To manage your [schemas](/docs/fundamentals/schemas/), you will need to have an [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) installed (you will already have one if you followed the [Snowplow Self-Hosted Quick Start](/docs/get-started/self-hosted/)).

Alternatively, you can host a [static Iglu registry](/docs/api-reference/iglu/iglu-repositories/static-repo/) in Amazon S3 or Google Cloud Storage.

## Create a schema

First, design the schema for your custom event (or entity). For example:

```json
{
     "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
     "description": "Schema for a button click event",
     "self": {
         "vendor": "com.snowplowanalytics",
         "name": "button_click",
         "format": "jsonschema",
         "version": "1-0-0"
     },
     "type": "object",
     "properties": {
         "id": {
             "type": "string",
             "minLength": 1
         },
         "target": {
             "type": "string"
         },
         "content": {
             "type": "string"
         }
     },
     "required": ["id"],
     "additionalProperties": false
 }
```

Next, save this schema in the following folder structure, with a filename of `1-0-0` (without any extension):

```text
schemas
└── com.snowplowanalytics
    └── button_click
        └── jsonschema
            └── 1-0-0
```

> **Tip:** If you update the `vendor` or the `name` in the example, you should update the above path too.

Finally, to upload your schema to your Iglu registry, you can use [igluctl](/docs/api-reference/iglu/igluctl-2/):

```bash
igluctl static push --public <local path to schemas> <Iglu server endpoint> <iglu_super_api_key>
```

See the [Igluctl reference page](/docs/api-reference/iglu/igluctl-2/#static-push) for more information on the `static push` command.

## Versioning schemas

When evolving your [schema](/docs/fundamentals/schemas/) and [uploading](/docs/api-reference/iglu/manage-schemas/) it to your [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/), you will need to choose how to increment its version.

There are two kinds of schema changes:

- **Non-breaking** - a non-breaking change is backward compatible with historical data and increments the `patch` number i.e. `1-0-0` -> `1-0-1`, or the middle digit i.e. `1-0-0` -> `1-1-0`.
- **Breaking** - a breaking change is not backwards compatible with historical data and increments the `model` number i.e. `1-0-0` -> `2-0-0`.

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Warning:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

---

# JSON Schema reference
> Complete reference for JSON Schema draft 4 features supported in Snowplow data structures, including validation keywords and type definitions.
> Source: https://docs.snowplow.io/docs/api-reference/json-schema-reference/

[Snowplow schemas](/docs/fundamentals/schemas/) are based on the [JSON Schema](https://json-schema.org/) standard ([draft 4](https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00)). This reference provides comprehensive documentation for all JSON Schema features that are supported in Snowplow.

Understanding the full capabilities of JSON Schema allows you to create more precise and robust [data structures](/docs/fundamentals/schemas/) that ensure your data quality and provide clear documentation for your tracking implementation.

## Schema structure

Every Snowplow schema must follow this basic structure with Snowplow-specific metadata and JSON Schema validation rules:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Human-readable description of the schema purpose",
  "self": {
    "vendor": "com.example",
    "name": "schema_name",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    // Field definitions go here
  },
  "additionalProperties": false,
  "required": ["required_field_name"]
}
```

### Required components

Every Snowplow schema must include these components:

- **`$schema`**: Must be `"http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#"`

- **`self`** object containing:

  - **`vendor`**: Your organization identifier (e.g., `"com.example"`)
  - **`name`**: The schema name
  - **`format`**: Must be `"jsonschema"`
  - **`version`**: Semantic version (e.g., `"1-0-0"`)

- **`type`**: Must be `"object"` for the root level

### Optional components

These components are optional but commonly used:

- **`description`**: Human-readable description of the schema purpose (highly recommended)
- **`properties`**: Object defining the fields and their validation rules
- **`additionalProperties`**: Whether additional properties are allowed (commonly set to `false`)
- **`required`**: Array of required field names
- **`minProperties`** / **`maxProperties`**: Constraints on number of properties

## Core validation keywords

### Type validation

The `type` keyword specifies the expected data type for a value. Snowplow supports all JSON Schema primitive types:

#### String type

```json
{
  "user_name": {
    "type": "string",
    "description": "The user's display name"
  }
}
```

#### Number and integer types

```json
{
  "price": {
    "type": "number",
    "description": "Product price in USD"
  },
  "quantity": {
    "type": "integer",
    "description": "Number of items purchased"
  }
}
```

#### Boolean type

```json
{
  "is_premium": {
    "type": "boolean",
    "description": "Whether the user has a premium account"
  }
}
```

#### Array type

```json
{
  "tags": {
    "type": "array",
    "description": "Product tags",
    "items": {
      "type": "string"
    }
  }
}
```

#### Object type

```json
{
  "address": {
    "type": "object",
    "description": "User's shipping address",
    "properties": {
      "street": {"type": "string"},
      "city": {"type": "string"},
      "postal_code": {"type": "string"}
    }
  }
}
```

#### Null type

```json
{
  "middle_name": {
    "type": ["string", "null"],
    "description": "User's middle name (optional)"
  }
}
```

### Multiple types

You can specify multiple acceptable types using an array:

```json
{
  "user_id": {
    "type": ["string", "integer"],
    "description": "User identifier (string or numeric)"
  },
  "optional_field": {
    "type": ["string", "null"],
    "description": "Optional text field"
  }
}
```

## String validation

### Length constraints

Control the minimum and maximum length of string values:

```json
{
  "username": {
    "type": "string",
    "minLength": 3,
    "maxLength": 20,
    "description": "Username between 3-20 characters"
  },
  "password": {
    "type": "string",
    "minLength": 8,
    "description": "Password must be at least 8 characters"
  }
}
```

### Enumeration

Restrict values to a specific set of allowed strings:

```json
{
  "status": {
    "type": "string",
    "enum": ["active", "inactive", "pending", "suspended"],
    "description": "Account status"
  },
  "color": {
    "type": "string",
    "enum": ["red", "green", "blue", "yellow"],
    "description": "Primary color selection"
  }
}
```

### Pattern matching

Use regular expressions to validate string format:

```json
{
  "product_code": {
    "type": "string",
    "pattern": "^[A-Z]{2}-\\d{4}$",
    "description": "Product code format (e.g., AB-1234)"
  },
  "phone_number": {
    "type": "string",
    "pattern": "^\\+?[1-9]\\d{1,14}$",
    "description": "International phone number format"
  }
}
```

> **Tip:** For common formats like email addresses, URLs, and dates, prefer using the `format` keyword instead of regular expressions for better readability and standardized validation.

### Format validation

Use the `format` keyword to validate common string formats:

```json
{
  "email": {
    "type": "string",
    "format": "email",
    "description": "Valid email address"
  },
  "website": {
    "type": "string",
    "format": "uri",
    "description": "Website URL"
  },
  "server_ip": {
    "type": "string",
    "format": "ipv4",
    "description": "IPv4 address of the server"
  },
  "created_at": {
    "type": "string",
    "format": "date-time",
    "description": "ISO 8601 timestamp"
  },
  "user_id": {
    "type": "string",
    "format": "uuid",
    "description": "UUID identifier"
  }
}
```

#### Supported format values

- **`uri`**: Uniform Resource Identifier
- **`ipv4`**: IPv4 address (e.g., "192.168.1.1")
- **`ipv6`**: IPv6 address
- **`email`**: Email address
- **`date-time`**: ISO 8601 date-time (e.g., "2023-12-25T10:30:00Z")
- **`date`**: ISO 8601 date (e.g., "2023-12-25")
- **`hostname`**: Internet hostname
- **`uuid`**: UUID string

## Numeric validation

### Range constraints

Set minimum and maximum values for numbers and integers:

```json
{
  "age": {
    "type": "integer",
    "minimum": 0,
    "maximum": 150,
    "description": "Person's age in years"
  },
  "discount_rate": {
    "type": "number",
    "minimum": 0,
    "maximum": 1,
    "description": "Discount rate between 0 and 1"
  }
}
```

### Multiple constraints

Combine multiple numeric validations:

```json
{
  "rating": {
    "type": "number",
    "minimum": 1,
    "maximum": 5,
    "multipleOf": 0.5,
    "description": "Star rating in half-point increments"
  }
}
```

## Array validation

### Length constraints

Control the size of arrays:

```json
{
  "favorite_colors": {
    "type": "array",
    "minItems": 1,
    "maxItems": 5,
    "description": "User's favorite colors (1-5 selections)",
    "items": {
      "type": "string",
      "enum": ["red", "blue", "green", "yellow", "purple", "orange"]
    }
  }
}
```

### Item validation

Define validation rules for array items:

```json
{
  "purchase_items": {
    "type": "array",
    "description": "Items in the purchase",
    "items": {
      "type": "object",
      "properties": {
        "product_id": {"type": "string"},
        "quantity": {"type": "integer", "minimum": 1},
        "price": {"type": "number", "minimum": 0}
      },
      "required": ["product_id", "quantity", "price"],
      "additionalProperties": false
    }
  }
}
```

### Unique items

Ensure all array items are unique:

```json
{
  "user_tags": {
    "type": "array",
    "uniqueItems": true,
    "description": "Unique tags assigned to user",
    "items": {
      "type": "string"
    }
  }
}
```

## Object validation

### Property requirements

Specify which object properties are required:

```json
{
  "user_profile": {
    "type": "object",
    "properties": {
      "first_name": {"type": "string"},
      "last_name": {"type": "string"},
      "email": {"type": "string"},
      "phone": {"type": ["string", "null"]}
    },
    "required": ["first_name", "last_name", "email"],
    "additionalProperties": false
  }
}
```

### Additional properties

Control whether additional properties are allowed:

```json
{
  "strict_object": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "value": {"type": "number"}
    },
    "additionalProperties": false,
    "description": "Only name and value properties allowed"
  },
  "flexible_object": {
    "type": "object",
    "properties": {
      "core_field": {"type": "string"}
    },
    "additionalProperties": true,
    "description": "Additional properties are permitted"
  }
}
```

### Property count constraints

Limit the number of properties in an object:

```json
{
  "metadata": {
    "type": "object",
    "minProperties": 1,
    "maxProperties": 10,
    "additionalProperties": {"type": "string"},
    "description": "Metadata with 1-10 string properties"
  }
}
```

## Advanced validation patterns

### Schema composition

Use `oneOf` and `anyOf` to create flexible validation rules:

#### Using oneOf

Validate that data matches exactly one of several schemas:

```json
{
  "contact_info": {
    "type": "object",
    "oneOf": [
      {
        "properties": {
          "type": {"enum": ["email"]},
          "email": {"type": "string", "format": "email"}
        },
        "required": ["type", "email"],
        "additionalProperties": false
      },
      {
        "properties": {
          "type": {"enum": ["phone"]},
          "phone": {"type": "string", "pattern": "^\\+?[1-9]\\d{1,14}$"}
        },
        "required": ["type", "phone"],
        "additionalProperties": false
      },
      {
        "properties": {
          "type": {"enum": ["address"]},
          "street": {"type": "string"},
          "city": {"type": "string"},
          "postal_code": {"type": "string"}
        },
        "required": ["type", "street", "city", "postal_code"],
        "additionalProperties": false
      }
    ]
  }
}
```

#### Using anyOf

Validate that data matches one or more of several schemas:

```json
{
  "user_permissions": {
    "type": "object",
    "anyOf": [
      {
        "properties": {
          "can_read": {"type": "boolean"}
        },
        "required": ["can_read"]
      },
      {
        "properties": {
          "can_write": {"type": "boolean"}
        },
        "required": ["can_write"]
      },
      {
        "properties": {
          "can_admin": {"type": "boolean"}
        },
        "required": ["can_admin"]
      }
    ]
  }
}
```

## Limitations and unsupported features

While Snowplow supports most JSON Schema Draft 4 features, there are some limitations to be aware of:

- **`$ref`**: Schema references are not supported in property definitions
- **`allOf`**: Schema intersection is not supported
- **`not`**: Negation validation is not supported
- **`dependencies`**: Property dependencies are not supported
- **`exclusiveMinimum`** and **`exclusiveMaximum`**: Exclusive bounds are not supported

Instead of unsupported features, use these approaches:

```json
{
  // Instead of $ref, define inline schemas
  "address": {
    "type": "object",
    "properties": {
      "street": {"type": "string"},
      "city": {"type": "string"},
      "country": {"type": "string", "enum": ["US", "CA", "UK", "DE"]}
    },
    "required": ["street", "city", "country"],
    "additionalProperties": false
  },

  // Instead of exclusiveMinimum/exclusiveMaximum, use minimum/maximum with adjusted values
  "percentage": {
    "type": "number",
    "minimum": 0,
    "maximum": 99.99,
    "description": "Percentage value (0 to less than 100)"
  },

  // Use format validation for common patterns
  "created_date": {
    "type": "string",
    "format": "date-time",
    "description": "ISO 8601 timestamp"
  }
}
```

---

# BigQuery Loader configuration reference
> Configure BigQuery Streaming Loader with BigQuery, Kinesis, and Pub/Sub settings for streaming enriched Snowplow events.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/

The configuration reference in this page is written for BigQuery Loader `2.1.0`

### License

The BigQuery Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {
  "accept": true
}
```

### BigQuery configuration

| Parameter                 | Description                                                                                                                                                                                               |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.project`     | Required. The GCP project to which the BigQuery dataset belongs                                                                                                                                           |
| `output.good.dataset`     | Required. The BigQuery dataset to which events will be loaded                                                                                                                                             |
| `output.good.table`       | Optional. Default value `events`. Name to use for the events table                                                                                                                                        |
| `output.good.credentials` | Optional. Service account credentials (JSON). If not set, default credentials will be sourced from the usual locations, e.g. file pointed to by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-bigquery-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                       |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 2.1.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                               | Description                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `input.subscription`                                    | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                           |
| `input.durationPerAckExtension`                         | Optional. Default value `15 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                                  |
| `input.minRemainingAckDeadline`                         | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `15 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `1.5 seconds` left of the remaining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`                              | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                     |
| `input.debounceRequests`                                | Optional. Default value `100 millis`. Adds an artifical delay between consecutive requests to Pub/Sub for more messages. Under some circumstances, this was found to slightly alleviate a problem in which Pub/Sub might re-deliver the same messages multiple times.                                                                                                    |
| `input.retries.transientErrors.delay` (since 2.1.0)     | Optional. Default value `100 millis`. Backoff delay for follow-up attempts after transient errors.                                                                                                                                                                                                                                                                       |
| `input.retries.transientErrors.attempts` (since 2.1.0)  | Optional. Default value `10`. Maximum number of attempts, after which the loader will crash and exit.                                                                                                                                                                                                                                                                    |
| `output.bad.topic`                                      | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                      |
| `output.bad.batchSize`                                  | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                        |
| `output.bad.requestByteThreshold`                       | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                           |
| `output.bad.maxRecordSize`                              | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                            |
| `output.retries.transientErrors.delay` (since 2.1.0)    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts after transient errors.                                                                                                                                                                                                                                                                       |
| `output.retries.transientErrors.attempts` (since 2.1.0) | Optional. Default value `10`. Maximum number of attempts, after which the loader will crash and exit.                                                                                                                                                                                                                                                                    |

***

## Other configuration options

| Parameter | Description |
| --------- | ----------- |

|                                       |                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                   | Optional. Default value `10000000`. Events are emitted to BigQuery when the batch reaches this size in bytes                                                                                                                                                                                                                                                     |
| `batching.maxDelay`                   | Optional. Default value `1 second`. Events are emitted to BigQuery after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                                            |
| `batching.writeBatchConcurrency`      | Optional. Default value 2. How many batches can we send simultaneously over the network to BigQuery                                                                                                                                                                                                                                                              |
| `cpuParallelism.parseBytesFactor`     | Optional. Default value `0.1`.Controls how many batches of bytes we can parse into enriched events simultaneously.E.g. If there are 2 cores and `parseBytesFactor = 0.1` then only one batch gets processed at a time.Adjusting this value can cause the app to use more or less of the available CPU.                                                           |
| `cpuParallelism.transformFactor`      | Optional. Default value `0.75`.Controls how many batches of enriched events we can transform into BigQuery format simultaneously.E.g. If there are 4 cores and `transformFactor = 0.75` then 3 batches gets processed in parallel.Adjusting this value can cause the app to use more or less of the available CPU.                                               |
| `retries.setupErrors.delay`           | Optional. Default value `30 seconds`.Configures exponential backoff on errors related to how BigQuery is set up for this loader.Examples include authentication errors and permissions errors.This class of errors are reported periodically to the monitoring webhook.                                                                                          |
| `retries.transientErrors.delay`       | Optional. Default value `1 second`.Configures exponential backoff on errors that are likely to be transient.Examples include server errors and network errors.                                                                                                                                                                                                   |
| `retries.transientErrors.attempts`    | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                             |
| `skipSchemas`                         | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`.A list of schemas that won't be loaded to BigQuery.This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                    |
| `legacyColumnMode`                    | Optional. Default value `false`.When this mode is enabled, the loader uses the legacy column style used by the v1 BigQuery loader.For example, an entity for a `1-0-0` schema is loaded into a column ending in `_1_0_0`, instead of a column ending in `_1`.This feature could be helpful when migrating from the v1 loader to the v2 loader.                   |
| `legacyColumns`                       | Optional, e.g. `["iglu:com.example/legacy/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/legacy/jsonschema/1--"]`.Schemas for which to use the legacy column style used by the v1 BigQuery loader, even when `legacyColumnMode` is disabled.                                                                                                          |
| `exitOnMissingIgluSchema`             | Optional. Default value `true`.Whether the loader should crash and exit if it fails to resolve an Iglu Schema.We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing.Change to `false` so events go the failed events stream instead of crashing the loader. |
| `monitoring.metrics.statsd.hostname`  | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                        |
| `monitoring.metrics.statsd.port`      | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                     |
| `monitoring.metrics.statsd.tags.*`    | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.period`    | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.prefix`    | Optional. Default `snowplow.bigquery-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                            |
| `monitoring.webhook.endpoint`         | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how BigQuery is set up for this loader.                                                                                                                                                                                   |
| `monitoring.webhook.tags.*`           | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                  |
| `monitoring.webhook.heartbeat.*`      | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                            |
| `monitoring.sentry.dsn`               | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.tags.*`            | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                               |
| `telemetry.disable`                   | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                        |
| `telemetry.userProvidedId`            | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                              |
| `http.client.maxConnectionsPerServer` | Optional. Default value 4. Configures the internal HTTP client used for iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                      |

---

# BigQuery Streaming Loader
> Stream Snowplow events to BigQuery from Kinesis or Pub/Sub with real-time loading and schema evolution.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/

The BigQuery Streaming Loader is an application that loads Snowplow events to BigQuery.

**AWS:**

On AWS, the BigQuery Streaming Loader continually pulls events from Kinesis and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>BigQuery Loader</b>"}}
subgraph bigquery [BigQuery]
  table[("<b>Events table</b>")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```

The BigQuery Loader is published as a Docker image which you can run on any AWS VM.

```bash
docker pull snowplow/bigquery-loader-kinesis:2.1.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/bigquery-loader-kinesis:2.1.0 \
--config=/myconfig/loader.hocon \
--iglu-config /myconfig/iglu.hocon
```

Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**GCP:**

On GCP, the BigQuery Streaming Loader continually pulls events from Pub/Sub and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>BigQuery Loader</b>"}}
subgraph bigquery [BigQuery]
  table[("<b>Events table</b>")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```

The BigQuery Loader is published as a Docker image which you can run on any GCP VM.

```bash
docker pull snowplow/bigquery-loader-pubsub:2.1.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/bigquery-loader-pubsub:2.1.0 \
--config=/myconfig/loader.hocon \
--iglu-config /myconfig/iglu.hocon
```

Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

***

> **Tip:** For more information on how events are stored in BigQuery, check the [mapping between Snowplow schemas and the corresponding BigQuery column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=bigquery).

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/master/config/config.kinesis.minimal.hocon)

**GCP:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/master/config/config.pubsub.minimal.hocon)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for all possible configuration parameters.

### Iglu

The BigQuery Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

## Metrics

The BigQuery Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric               | Definition                                                                                                                                                     |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_good`        | A count of events that are successfully written to BigQuery.                                                                                                   |
| `events_bad`         | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                             |
| `latency_millis`     | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                  |
| `e2e_latency_millis` | The end-to-end latency of the snowplow pipeline. The time in milliseconds from when an event was received by the collector, until it is written into BigQuery. |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for BigQuery Loader (since version 2.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# BigQuery Loader 1.0.x upgrade guide
> Upgrade BigQuery Loader from 0.6.x to 1.0.x with HOCON config, new load_tstamp field, and StreamLoader migration steps.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/1-0-x-upgrade-guide/

## Configuration

The only breaking change from the 0.6.x series is the new format of the configuration file. That used to be a self-describing JSON but is now HOCON. Additionally, some app-specific command-line arguments have been incorporated into the config, such as Repeater's `--failedInsertsSub` option. For more details, see the [setup guide](/docs/api-reference/loaders-storage-targets/bigquery-loader/) and [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/configuration-reference/).

Using Repeater as an example, if your configuration for 0.6.x looked like this:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow.storage/bigquery_config/jsonschema/1-0-0",
  "data": {
    "name": "Alpha BigQuery test",
    "id": "31b1559d-d319-4023-aaae-97698238d808",
    "projectId": "com-acme",
    "datasetId": "snowplow",
    "tableId": "events",
    "input": "enriched-sub",
    "typesTopic": "types-topic",
    "typesSubscription": "types-sub",
    "badRows": "bad-topic",
    "failedInserts": "failed-inserts-topic",
    "load": {
      "mode": "STREAMING_INSERTS",
      "retry": false
     },
     "purpose": "ENRICHED_EVENTS"
  }
}
```

it will now look like this:

```json
{
  "projectId": "com-acme"

  "loader": {
    "input": {
      "subscription": "enriched-sub"
    }

    "output": {
      "good": {
        "datasetId": "snowplow"
        "tableId": "events"
      }

      "bad": {
        "topic": "bad-topic"
      }

      "types": {
        "topic": "types-topic"
      }

      "failedInserts": {
        "topic": "failed-inserts-topic"
      }
    }
  }

  "mutator": {
    "input": {
      "subscription": "types-sub"
    }

    "output": {
      "good": ${loader.output.good} # will be automatically inferred
    }
  }

  "repeater": {
    "input": {
      "subscription": "failed-inserts-sub"
    }

    "output": {
      "good": ${loader.output.good} # will be automatically inferred

      "deadLetters": {
        "bucket": "gs://dead-letter-bucket"
      }
    }
  }

  "monitoring": {} # disabled
}
```

And instead of running it like this:

```bash
$ ./snowplow-bigquery-repeater \
    --config=$CONFIG \
    --resolver=$RESOLVER \
    --failedInsertsSub="failed-inserts-sub" \
    --deadEndBucket="gs://dead-letter-bucket"
    --desperatesBufferSize=20 \
    --desperatesWindow=20 \
    --backoffPeriod=900 \
    --verbose
```

you will run it like this:

```bash
$ docker run \
  -v /path/to/resolver.json:/resolver.json \
  snowplow/snowplow-bigquery-repeater:1.0.1 \
    --config=$CONFIG \
    --resolver=/resolver.json \
    --bufferSize=20 \
    --timeout=20 \
    --backoffPeriod=900 \
    --verbose
```

## New events table field

The first time you deploy Mutator 1.0.0 it will add a new column to your events table: `load_tstamp`. This represents the exact moment when the row was inserted into BigQuery. It shows you when events have arrived in the warehouse, which makes it possible to use incremental processing of newly arrived data in your downstream data modeling.

Depending on your traffic volume and pattern, there might be a short time period in which the loader app cannot write to BigQuery because the new column hasn't propagated and is not yet visible to all workers. For that reason, **we recommend that you upgrade Mutator first**.

## Migrating to StreamLoader

StreamLoader has been built as a standalone application, replacing Apache Beam and no longer requires you to use Dataflow.

Depending on your data volume and traffic patterns, this might lead to significant cost reductions. However, by migrating away from Dataflow, you no longer benefit from its exactly-once processing guarantees. As such, there could be a slight increase in the number of duplicate events loaded into BigQuery.

Duplicate events generally are to be expected in a Snowplow pipeline, which provides an at-least-once guarantee.

In our tests, we found that duplicates arise only during extreme autoscaling of the loader, eg if your pipeline has a sudden extreme spike in events. Aside from autoscaling events, we found the number of duplicate rows to be very low, however this depends on the type of worker infrastructure you use.

---

# BigQuery Loader 2.0.0 upgrade guide
> Guide for upgrading to BigQuery Loader 2.0.0, including configuration changes, new column naming strategy, and recovery columns for schema evolution.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/2-0-0-upgrade-guide/

## Configuration

BigQuery Loader 2.0.0 brings changes to the loading setup. It is no longer necessary to configure and deploy three independent applications (Loader, Repeater and Mutator in [1.X](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/)) in order to load your data to BigQuery. Starting from 2.0.0, only one application is needed, which naturally introduces some breaking changes to the configuration file structure.

See the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for all possible configuration parameters and the minimal [configuration samples](https://github.com/snowplow-incubator/snowplow-bigquery-loader/blob/v2/config) for each of supported cloud environments.

## Infrastructure

Apart from Repeater and Mutator, other infrastructure components have become obsolete:

- The `types` PubSub topic connecting Loader and Mutator.
- The `failedInserts` PubSub topic connecting Loader and Repeater.
- The `deadLetter` GCS bucket used by Repeater to store data that repeatedly failed to be inserted into BigQuery.

This means that failed events are now written to the failed events PubSub topic, configured as `output.bad.topic`, rather than directly to the GCS bucket as before. This change was made to consolidate all types of event failures into a single place.

## Events table format

Starting from 2.0.0, BigQuery Loader changes its output column naming strategy. For example, for [ad\_click event](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_click_event/jsonschema/1-0-0):

- Before an upgrade, the corresponding column would be named `unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1_0_0`.
- After an upgrade, new column will be named `unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1`.

All self-describing events and entities will be loaded to new "major version"-oriented columns. Old "full version"-oriented columns would remain unchanged, but no new data would be loaded into them (the 2.0.0 loader just ignores these columns).

The new column naming scheme has several advantages:

- Fewer columns created (BigQuery has a limit on the total number of columns)
- No need to update data models (or use complex macros) every time a new minor version of a schema is created

The catch is that you have to follow the rules of schema evolution more strictly to ensure data from different schema versions can fit in the same column — see below.

> **Tip:** If you are using [Snowplow dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), they will automatically consolidate the data between `_1_0_0` and `_1` style columns, because they look at the major version prefix (e.g. `_1`), which is common to both.
>
> If you are not using Snowplow dbt models but still use dbt, you can employ [this macro](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) to manually aggregate the data across old and new columns.

### Enable legacy mode for the old table format

To simplify migration to the new table format, it is possible to run the 2.x loader in legacy mode, so it loads self-describing events and entities using the old column names of the 1.x loader.

**Option 1:** In the configuration file, set `legacyColumnMode` to `true`. When this mode is enabled, the loader uses the legacy column style for all self-describing events and entities.

**Option 2:** In the configuration file, set `legacyColumns` to list specific schemas for which to use the legacy column style. This list is used when `legacyColumnMode` is `false` (the default).

For example:

```json
"legacyColumns": [
  "iglu:com.example/legacy_a/jsonschema/1-0-0",
  "iglu:com.example/legacy_b/jsonschema/1-*-*"
]
```

## Recovery columns

### What is schema evolution?

One of Snowplow’s key features is the ability to [define custom schemas and validate events against them](/docs/fundamentals/schemas/). Over time, users often evolve the schemas, e.g. by adding new fields or changing existing fields. To accommodate these changes, BigQuery Loader 2.0.0 automatically adjusts the database tables in the warehouse accordingly.

There are two main types of schema changes:

**Breaking**: The schema version has to be changed in a major way (`1-2-3` → `2-0-0`). As of BigQuery Loader 2.0.0, each major schema version has its own column (`..._1`, `..._2`, etc, for example: `contexts_com.snowplowanalytics_ad_click_1`).

**Non-breaking**: The schema version can be changed in a minor way (`1-2-3` → `1-3-0` or `1-2-3` → `1-2-4`). Data is stored in the same database column.

Loader tries to format the incoming data according to the latest version of the schema it saw (for a given major version, e.g. `1-*-*`). For example, if a batch contains events with schema versions `1-0-0`, `1-0-1` and `1-0-2`, the loader derives the output schema based on version `1-0-2`. Then the loader instructs BigQuery to adjust the database column and load the data.

### Recovering from invalid schema evolution

Let's consider these two schemas as an example of breaking schema evolution (changing the type of a field from `integer` to `string`) using the same major version (`1-0-0` and `1-0-1`):

```json
{
   // 1-0-0
   "properties": {
      "a": {"type": "integer"}
   }
}
```

```json
{
   // 1-0-1
   "properties": {
      "a": {"type": "string"}
   }
}
```

With BigQuery Loader 1.x, data for each version would go to its own column — no issue. With BigQuery Loader 2.x, there is only one column. But strings and integers can’t coexist!

To avoid crashing or losing data, BigQuery Loader 2.0.0 proceeds by creating a new column for the data with schema `1-0-1`, e.g. `contexts_com_snowplowanalytics_ad_click_1_0_1_recovered_9999999`, where:

- `1_0_1` is the version of the offending schema;
- `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one).

If you create a new schema `1-0-2` that reverts the offending changes and is again compatible with `1-0-0`, the data for events with that schema will be written to the original column as expected.

> **Tip:** You might find that some of your schemas were evolved incorrectly in the past, which results in the creation of these “recovery” columns after the upgrade. To address this for a given schema, create a new _minor_ schema version that reverts the breaking changes introduced in previous versions. (Or, if you want to keep the breaking change, create a new _major_ schema version.) You can set it to [supersede](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded) the previous version(s), so that events are automatically validated against the new schema.

> **Note:** If events with incorrectly evolved schemas never arrive, then the recovery column would not be created.

You can read more about schema evolution and how recovery columns work [here](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=bigquery#versioning).

---

# Upgrade guides for the BigQuery Loader
> Upgrade guides for BigQuery Loader with breaking changes, migration steps, and compatibility notes for major versions.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of the BigQuery Loader.

---

# Databricks Streaming Loader configuration reference
> Configure Databricks Streaming Loader with Unity Catalog, Kinesis, Pub/Sub, and Kafka settings for lakehouse streaming.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/

The configuration reference in this page is written for Databricks Streaming Loader `0.4.0`

### License

The Databricks Streaming Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {
  "accept": true
}
```

### Databricks configuration

| Parameter                        | Description                                                                                                                                                                                |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.host`               | Required, e.g. `https://{workspace-id}.cloud.databricks.com`. URL of the Databricks workspace.                                                                                             |
| `output.good.catalog`            | Required. Name of the Databricks catalog containing the volume.                                                                                                                            |
| `output.good.schema`             | Required. Name of the Databricks schema containing the volume.                                                                                                                             |
| `output.good.volume`             | Required. Name of the Databricks volume to which this loader will upload staging files. This must be an [external](https://docs.databricks.com/aws/en/volumes/managed-vs-external) volume. |
| `output.good.token`              | Required if using PAT authentication. A Databricks [personal access token](https://docs.databricks.com/aws/en/dev-tools/auth).                                                             |
| `output.good.oauth.clientId`     | Required if using OAUTH authentication. The client ID for a Databricks [service principal](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m).                                   |
| `output.good.oauth.clientSecret` | Required if using OAUTH authentication. The client secret for a Databricks [service principal](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m).                               |
| `output.good.compression`        | Optional. Default value `snappy`. Compression algorithm for the uploaded staging parquet files.                                                                                            |
| `output.good.httpTimeout`        | Optional. Default value `20 seconds`. Timeout duration of Databricks SDK's HTTP client.                                                                                                    |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-databricks-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                     |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 0.4.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                      | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                        |
| `input.durationPerAckExtension`           | Optional. Default value `15 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                               |
| `input.minRemainingAckDeadline`           | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`                | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                  |
| `input.debounceRequests`                  | Optional. Default value `100 millis`. Adds an artifical delay between consecutive requests to Pub/Sub for more messages. Under some circumstances, this was found to slightly alleviate a problem in which Pub/Sub might re-deliver the same messages multiple times.                                                                                                 |
| `input.retries.transientErrors.delay`     | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                            |
| `input.retries.transientErrors.attempts`  | Optional. Default value `10`. Max number of attempts, after which Enrich will crash and exit                                                                                                                                                                                                                                                                          |
| `output.bad.topic`                        | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                   |
| `output.bad.batchSize`                    | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                     |
| `output.bad.requestByteThreshold`         | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                        |
| `output.bad.maxRecordSize`                | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                         |
| `output.retries.transientErrors.delay`    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                            |
| `output.retries.transientErrors.attempts` | Optional. Default value `10`. Max number of attempts, after which Enrich will crash and exit                                                                                                                                                                                                                                                                          |

**Azure:**

| Parameter                           | Description                                                                                                                                                                                  |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                   | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                         |
| `input.bootstrapServers`            | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                                |
| `input.consumerConf.*`              | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html).      |
| `input.debounceCommitOffsets`       | Optional. Default value 10 seconds. How frequently to commit our progress back to kafka. By increasing this value, we decrease the number of requests made to the kafka broker.              |
| `input.commitTimeout` (since 0.4.0) | Optional. Default value 15 seconds. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `output.bad.topicName`              | Required. Name of the Kafka topic that will receive failed events.                                                                                                                           |
| `output.bad.bootstrapServers`       | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                                 |
| `output.bad.producerConf.*`         | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html).      |
| `output.bad.maxRecordSize`          | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                                  |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>     "security.protocol": "SASL_SSL"
>     "sasl.mechanism": "PLAIN"
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
> }
> ```

***

## Other configuration options

| Parameter                             | Description                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                   | Optional. Default value `16000000`. Events are uploaded to the Databricks volume when the batch reaches this size in bytes                                                                                                                                                                                                                                          |
| `batching.maxDelay`                   | Optional. Default value `1 second`. Events are uploaded to the Databricks volume after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                                 |
| `batching.uploadParallelismFactor`    | Optional. Default value 3. Controls how many batches can we send simultaneously over the network to Databricks. E.g. If there are 4 available processors, and `uploadParallelismFactor` is 3.5, then the loader sends up to 14 batches in parallel. Adjusting this value can cause the app to use more or less of the available CPU.                                |
| `cpuParallelismFactor`                | Optional. Default value 0.75. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and `cpuParallelismFactor` is 0.75, then the loader processes 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU.                       |
| `retries.setupErrors.delay`           | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how Databricks is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                                        |
| `retries.transientErrors.delay`       | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                                    |
| `retries.transientErrors.attempts`    | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                |
| `skipSchemas`                         | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to Databricks. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                   |
| `exitOnMissingIgluSchema`             | Optional. Default value `true`. Whether the loader should crash and exit if it fails to resolve an Iglu Schema. We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing. Change to `false` so events go the failed events stream instead of crashing the loader. |
| `monitoring.metrics.statsd.hostname`  | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                           |
| `monitoring.metrics.statsd.port`      | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                        |
| `monitoring.metrics.statsd.tags.*`    | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.period`    | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.prefix`    | Optional. Default `snowplow.databricks-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                             |
| `monitoring.webhook.endpoint`         | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Databricks is set up for this loader.                                                                                                                                                                                    |
| `monitoring.webhook.tags.*`           | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                     |
| `monitoring.webhook.heartbeat.*`      | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                               |
| `monitoring.sentry.dsn`               | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                              |
| `monitoring.sentry.tags.*`            | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                  |
| `telemetry.disable`                   | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                           |
| `telemetry.userProvidedId`            | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                 |
| `http.client.maxConnectionsPerServer` | Optional. Default value 4. Configures the internal HTTP client used for iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                         |

---

# Databricks Streaming Loader
> Load Snowplow events to Databricks with low latency using Lakeflow Declarative Pipelines and Unity Catalog volumes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/

> **Info:** You will need a premium Databricks plan to use Lakeflow Declarative Pipelines.

The Databricks Streaming Loader is an application that integrates with a Databricks [Lakeflow Declarative Pipeline](https://docs.databricks.com/aws/en/dlt/) to load Snowplow events into Databricks with low latency.

**AWS:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any AWS VM.

```bash
docker pull snowplow/databricks-loader-kinesis:0.4.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \
--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \
snowplow/databricks-loader-kinesis:0.4.0 \
--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**GCP:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any GCP VM.

```bash
docker pull snowplow/databricks-loader-pubsub:0.4.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \
--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \
snowplow/databricks-loader-pubsub:0.4.0 \
--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

**Azure:**

There are two parts to how the Databricks Streaming Loader works.

In the first part, you use Snowplow's Databricks Streaming Loader to push staging files into a [Unity Catalog volume](https://docs.databricks.com/aws/en/volumes/).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Databricks Streaming Loader</b>"}}
subgraph databricks [Databricks]
  volume[("<b>Volume</b>")]
end
stream-->loader-->|REST API|databricks
```

In the second part, you use a Databricks Lakeflow Declarative Pipeline to load the staging files into a Streaming Live Table.

```mermaid
flowchart LR
subgraph databricks [Databricks]
    direction LR
    volume[("<b>Volume</b>")]
    pipeline{{"<b>Lakeflow Declarative Pipeline</b>"}}
    table[("<b>Streaming Live Table</b>")]
    volume-->pipeline-->table
end
```

The Databricks Streaming Loader is published as a Docker image which you can run on any Azure VM.

```bash
docker pull snowplow/databricks-loader-kafka:0.4.0
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting your Databricks credentials via environment variables, e.g. `DATABRICKS_CLIENT_SECRET`, so that you can refer to them in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env DATABRICKS_CLIENT_ID="${DATABRICKS_CLIENT_ID}" \
--env DATABRICKS_CLIENT_SECRET="${DATABRICKS_CLIENT_SECRET}" \
snowplow/databricks-loader-kafka:0.4.0 \
--config=/myconfig/loader.hocon
```

Where `loader.hocon` is the loader's [configuration file](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/#configuring-the-loader) and `iglu.hocon` is the [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.

***

> **Tip:** For more information on how events are stored in Databricks, check the [mapping between Snowplow schemas and the corresponding Databricks column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=databricks).

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.kinesis.minimal.hocon)

**GCP:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.pubsub.minimal.hocon)

**Azure:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/databricks-loader/blob/develop/config/config.kafka.minimal.hocon)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/) for all possible configuration parameters.

### Iglu

The Databricks Streaming Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

## Configuring the Databricks Lakeflow Declarative Pipeline

Create a Pipeline in your Databricks workspace, and copy the following SQL into the associated `.sql` file:

```sql
CREATE STREAMING LIVE TABLE events
CLUSTER BY (load_tstamp, event_name)
TBLPROPERTIES (
  'delta.dataSkippingStatsColumns' =
      'load_tstamp,collector_tstamp,derived_tstamp,dvce_created_tstamp,true_tstamp,event_name'
)
AS SELECT
  *,
  current_timestamp() as load_tstamp
FROM cloud_files(
  "/Volumes/<CATALOG_NAME>/<VOLUME_NAME>/<SCHEMA_NAME>/events",
  "parquet",
  map(
    "cloudfiles.inferColumnTypes", "false",
    "cloudfiles.includeExistingFiles", "false", -- set to true to load files already present in the volume
    "cloudfiles.schemaEvolutionMode", "addNewColumns",
    "cloudfiles.partitionColumns", "",
    "cloudfiles.useManagedFileEvents", "true",
    "datetimeRebaseMode", "CORRECTED",
    "int96RebaseMode", "CORRECTED",
    "mergeSchema", "true"
  )
)
```

Replace `/Volumes/<CATALOG_NAME>/<VOLUME_NAME>/<SCHEMA_NAME>/events` with the correct path to your volume.

Note that the volume must be an [external volume](https://docs.databricks.com/aws/en/volumes/) in to order to use the cloud files option `cloudfiles.useManagedFileEvents`, which is highly recommended for this integration.

## Metrics

The Databricks Streaming Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric               | Definition                                                                                                                                                                |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_good`        | A count of events that are successfully written to the Databricks volume.                                                                                                 |
| `events_bad`         | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                                        |
| `latency_millis`     | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                             |
| `e2e_latency_millis` | The end-to-end latency of the snowplow pipeline. The time in milliseconds from when an event was received by the collector, until it is written to the Databricks volume. |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for Databricks Streaming Loader (since version 0.1.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Elasticsearch Loader for Kinesis and NSQ
> Load Snowplow enriched and bad events from Kinesis or NSQ streams into Elasticsearch or OpenSearch clusters.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/elasticsearch/

If you are using [Enrich](/docs/api-reference/enrichment-components/) to write enriched Snowplow events to one stream and bad events to another, you can use the Elasticsearch Loader to read events from either of those streams and write them to [Elasticsearch](http://www.elasticsearch.org/overview/). It works with either Kinesis or NSQ streams.

> **Warning:** We only offer this loader on AWS or as part of [Snowplow Mini](/docs/api-reference/snowplow-mini/).

## What the data looks like

There are a few changes compared to the [standard structure of Snowplow data](/docs/fundamentals/canonical-event/).

### Boolean fields reformatted

All boolean fields like `br_features_java` are normally either `"0"` or `"1"`. In Elasticsearch, these values are converted to `false` and `true`.

### New `geo_location` field

The `geo_latitude` and `geo_longitude` fields are combined into a single `geo_location` field of Elasticsearch's ["geo\_point" type](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html).

### Self-describing events

Each [self-describing event](/docs/fundamentals/events/#self-describing-events) gets its own field (same [naming rules](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=snowflake#location) as for Snowflake). For example:

```json
{
  "unstruct_com_snowplowanalytics_snowplow_link_click_1": {
    "targetUrl": "http://snowplow.io",
    "elementId": "action",
    "elementClasses": [],
    "elementTarget": ""
  }
}
```

### Entities

Each [entity](/docs/fundamentals/entities/) type attached to the event gets its own field (same [naming rules](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=snowflake#location) as for Snowflake). The field contains an array with the data for all entities of the given type. For example:

```json
{
  "contexts_com_acme_user_1": [
    {
      "name": "Alice"
    }
  ],
  "contexts_com_acme_product_1": [
    {
      "name": "Apple"
    },
    {
      "name": "Orange"
    }
  ]
}
```

## Setup guide

### Configuring Elasticsearch

#### Getting started

First off, install and set up Elasticsearch. For more information, check out the [installation guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html) for installation information and [Supported versions of OpenSearch and Elasticsearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version) for the latest information of ElasticSearch/OpenSearch supported versions by AWS.

> **Note:** We support ElasticSearch v6.x and v7.x. We also support OpenSearch v1.x and v2.x. We do not support ElasticSearch v8.x currently.

#### Raising the file limit

Elasticsearch keeps a lot of files open simultaneously so you will need to increase the maximum number of files a user can have open. To do this:

```bash
sudo vim /etc/security/limits.conf
```

Append the following lines to the file:

```bash
{{USERNAME}} soft nofile 32000
{{USERNAME}} hard nofile 32000
```

Where `{{USERNAME}}` is the name of the user running Elasticsearch. You will need to logout and restart Elasticsearch before the new file limit takes effect.

To check that this new limit has taken effect you can run the following command from the terminal:

```bash
curl localhost:9200/_nodes/process?pretty
```

If the `max_file_descriptors` equals 32000 it is running with the new limit.

#### Defining the mapping

Use the following request to create the mapping with Elasticsearch 7+:

```bash
curl -XPUT 'http://localhost:9200/snowplow' -d '{
  "settings": {
      "analysis": {
          "analyzer": {
              "default": {
                  "type": "keyword"
              }
          }
      }
  },
  "mappings": {
      "properties": {
          "geo_location": {
              "type": "geo_point"
          }
      }
  }
}'
```

Note that Elasticsearch 7+ [no longer uses mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html). If you have an older version, you might need to include mapping types in the above snippet.

This initialization sets the default analyzer to "keyword". This means that string fields will not be split into separate tokens for the purposes of searching. This saves space and ensures that URL fields are handled correctly.

If you want to tokenize specific string fields, you can change the "properties" field in the mapping like this:

```bash
curl -XPUT 'http://localhost:9200/snowplow' -d '{
  "settings": {
      "analysis": {
          "analyzer": {
              "default": {
                  "type": "keyword"
              }
          }
      }
  },
  "mappings": {
      "properties": {
          "geo_location": {
              "type": "geo_point"
          },
          "field_to_tokenize": {
              "type": "string",
              "analyzer": "english"
          }
      }
  }
}'
```

### Installing the Elasticsearch Loader

The Elasticsearch Loader is published on Docker Hub:

```bash
docker pull snowplow/snowplow-elasticsearch-loader:2.1.3
```

The container can be run with the following command:

```bash
docker run \
-v /path/to/config.hocon:/snowplow/config.hocon \
snowplow/snowplow-elasticsearch-loader:2.1.3 \
--config /snowplow/config.hocon
```

Alternatively you can download and run a [jar file from the github release](https://github.com/snowplow/snowplow-elasticsearch-loader/releases):

```bash
java -jar snowplow-elasticsearch-loader-2.1.3.jar --config /path/to/config.hocon
```

### Using the Elasticsearch Loader

#### Configuration

The sink is configured using a HOCON file, for which you can find examples [here](https://github.com/snowplow/snowplow-elasticsearch-loader/tree/master/config). These are the fields:

| Name                               | Description                                                                                                                                                                                                                                                                               |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| purpose                            | Required. "ENRICHED\_EVENTS" for a stream of successfully enriched events "BAD\_ROWS" for a stream of bad events "JSON" for writing plain json                                                                                                                                            |
| input.type                         | Required. Configures where input events will be read from. Can be “kinesis”, “stdin” or “nsq”                                                                                                                                                                                             |
| input.streamName                   | Required when `input.type` is kinesis or nsq. Name of the stream to read from.                                                                                                                                                                                                            |
| input.initialPosition              | Required when `input.type` is kinesis. Used when `input.type` is Kinesis. Specifies where to start reading from the stream the first time the app is run. "TRIM\_HORIZON" for as far back as possible, "LATEST" for as recent as possibly, "AT\_TIMESTAMP" for after specified timestamp. |
| input.initialTimestamp             | Used when `input.type` is kinesis. Required when `input.initialTimestamp` is "AT\_TIMESTAMP". Specifies the timestamp to start read.                                                                                                                                                      |
| input.maxRecords                   | Used when `input.type` is kinesis. Optional. Maximum number of records fetched in a single request. Default value 10000.                                                                                                                                                                  |
| input.region                       | Used when `input.type` is kinesis. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html). Region where the Kinesis stream is located.                    |
| input.customEndpoint               | Used when `input.type` is kinesis. Optional. Custom endpoint to override AWS Kinesis endpoints, this can be used to specify local endpoints when using localstack.                                                                                                                        |
| input.dynamodbCustomEndpoint       | Used when `input.type` is kinesis. Optional. Custom endpoint to override AWS DynamoDB endpoints for Kinesis checkpoints lease table, this can be used to specify local endpoints when using Localstack.                                                                                   |
| input.appName                      | Used when `input.type` is kinesis. Optional. Used by a DynamoDB table to maintain stream state. Default value "snowplow-elasticsearch-loader".                                                                                                                                            |
| input.buffer.byteLimit             | Used when `input.type` is kinesis. Optional. The limit of the buffer in terms of bytes. When this value is exceeded, events will be sent to Elasticsearch. Default value 1000000.                                                                                                         |
| input.buffer.recordLimit           | Used when `input.type` is kinesis. Optional. The limit of the buffer in terms of record count. When this value is exceeded, events will be sent to Elasticsearch. Default value 500.                                                                                                      |
| input.buffer.timeLimit             | Used when `input.type` is kinesis. Optional. The time limit in milliseconds to wait to send the buffer to Elasticsearch. Default value 500.                                                                                                                                               |
| input.channelName                  | Required when `input.type` is nsq. Channel name for NSQ source stream. If more than one application reading from the same NSQ topic at the same time, all of them must have unique channel name to be able to get all the data from the same topic.                                       |
| input.nsqlookupdHost               | Required when `input.type` is nsq. Host name for nsqlookupd                                                                                                                                                                                                                               |
| input.nsqlookupdPort               | Required when `input.type` is nsq. HTTP port for nsqd.                                                                                                                                                                                                                                    |
| output.good.type                   | Required. Configure where to write good events. Can be "elasticsearch" or "stdout".                                                                                                                                                                                                       |
| output.good.client.endpoint        | Required. The Elasticsearch cluster endpoint.                                                                                                                                                                                                                                             |
| output.good.client.port            | Optional. The port the Elasticsearch cluster can be accessed on. Default value 9200.                                                                                                                                                                                                      |
| output.good.client.username        | Optional. HTTP Basic Auth username. Can be removed if not active.                                                                                                                                                                                                                         |
| output.good.client.password        | Optional. HTTP Basic Auth password. Can be removed if not active.                                                                                                                                                                                                                         |
| output.good.client.shardDateFormat | Optional. Formatting used for sharding good stream, i.e. \_yyyy-MM-dd. Can be removed if not needed.                                                                                                                                                                                      |
| output.good.client.shardDateField  | Optional. Timestamp field for sharding good stream. If not specified derived\_tstamp is used.                                                                                                                                                                                             |
| output.good.client.maxRetries      | Optional. The maximum number of request attempts before giving up. Default value 6.                                                                                                                                                                                                       |
| output.good.client.ssl             | Optional. Whether to use ssl or not. Default value false.                                                                                                                                                                                                                                 |
| output.good.aws.signing            | Optional. Whether to activate AWS signing or not. It should be activated if AWS OpenSearch service is used. Default value false.                                                                                                                                                          |
| output.good.aws.region             | Optional. Region where the AWS OpenSearch service is located.                                                                                                                                                                                                                             |
| output.good.cluster.index          | Required. The Elasticsearch index name.                                                                                                                                                                                                                                                   |
| output.good.cluster.documentType   | Optional. The Elasticsearch index type. Index types are deprecated in ES >=7.x Therefore, it shouldn't be set with ES >=7.x                                                                                                                                                               |
| output.good.chunk.byteLimit        | Optional. Bulk request to Elasticsearch will be splitted to chunks according given byte limit. Default value 1000000.                                                                                                                                                                     |
| output.good.chunk.recordLimit      | Optional. Bulk request to Elasticsearch will be splitted to chunks according given record limit. Default value 500.                                                                                                                                                                       |
| output.bad.type                    | Required. Configure where to write failed events. Can be "kinesis", "nsq", "stderr" or "none".                                                                                                                                                                                            |
| output.bad.streamName              | Required. Stream name for events which are rejected by Elasticsearch.                                                                                                                                                                                                                     |
| output.bad.region                  | Used when `output.bad.type` is kinesis. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html). Region where the bad Kinesis stream is located.           |
| output.bad.customEndpoint          | Used when `output.bad.type` is kinesis. Optional. Custom endpoint to override AWS Kinesis endpoints, this can be used to specify local endpoints when using localstack.                                                                                                                   |
| output.bad.nsqdHost                | Required when `output.bad.type` is nsq. Host name for nsqd.                                                                                                                                                                                                                               |
| output.bad.nsqdPort                | Required when `output.bad.type` is nsq. HTTP port for nsqd.                                                                                                                                                                                                                               |
| monitoring.snowplow\.collector     | Optional. Snowplow collector URI for monitoring. Can be removed together with monitoring section.                                                                                                                                                                                         |
| monitoring.snowplow\.appId         | Optional. The app id used in decorating the events sent for monitoring. Can be removed together with monitoring section.                                                                                                                                                                  |
| monitoring.metrics.cloudWatch      | Optional. Whether to enable Cloudwatch metrics or not. Default value true.                                                                                                                                                                                                                |

#### Document count

To check the number of documents in an Elasticsearch or OpenSearch cluster, use the [Count API](https://docs.opensearch.org/latest/api-reference/search-apis/count/) provided by Elasticsearch/OpenSearch. For example, to get the total number of documents in the cluster, use `GET _count`.

---

# Google Cloud Storage Loader
> Archive Snowplow events from Pub/Sub to Google Cloud Storage buckets using Dataflow with windowing and partitioning.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/

[Cloud Storage Loader](https://github.com/snowplow-incubator/snowplow-google-cloud-storage-loader/) is a [Dataflow](https://cloud.google.com/dataflow/) job which dumps event from an input [PubSub](https://cloud.google.com/pubsub/) subscription into a [Cloud Storage](https://cloud.google.com/storage/) bucket.

Cloud Storage loader is built on top of [Apache Beam](https://beam.apache.org/) and its Scala wrapper [SCIO](https://github.com/spotify/scio).

## Running

Cloud Storage Loader comes both as a Docker image and a ZIP archive.

### Docker

Docker image can be found on [Docker Hub](https://hub.docker.com/r/snowplow/snowplow-google-cloud-storage-loader).

A container can be run as follows:

```bash
docker run \
-v $PWD/config:/snowplow/config \ # if running outside GCP
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/config/credentials.json \ # if running outside GCP
snowplow/snowplow-google-cloud-storage-loader:0.5.6 \
--runner=DataFlowRunner \
--jobName=[JOB-NAME] \
--project=[PROJECT] \
--streaming=true \
--workerZone=[ZONE] \
--inputSubscription=projects/[PROJECT]/subscriptions/[SUBSCRIPTION] \
--outputDirectory=gs://[BUCKET]/YYYY/MM/dd/HH/ \ # partitions by date
--outputFilenamePrefix=output \ # optional
--shardTemplate=-W-P-SSSSS-of-NNNNN \ # optional
--outputFilenameSuffix=.txt \ # optional
--windowDuration=5 \ # optional, in minutes
--compression=none \ # optional, gzip, bz2 or none
--numShards=1 # optional
```

To display the help message:

```bash
docker run snowplow/snowplow-google-cloud-storage-loader:0.5.6 \
--help
```

To display documentation about Cloud Storage Loader-specific options:

```bash
docker run snowplow/snowplow-google-cloud-storage-loader:0.5.6 \
--help=com.snowplowanalytics.storage.googlecloudstorage.loader.Options
```

### ZIP archive

Archive is hosted on GitHub at this URI:

```bash
https://github.com/snowplow-incubator/snowplow-google-cloud-storage-loader/releases/download/0.5.6/snowplow-google-cloud-storage-loader-0.5.6.zip
```

Once unzipped the artifact can be run as follows:

```bash
./bin/snowplow-google-cloud-storage-loader \
--runner=DataFlowRunner \
--project=[PROJECT] \
--streaming=true \
--workerZone=[ZONE] \
--inputSubscription=projects/[PROJECT]/subscriptions/[SUBSCRIPTION] \
--outputDirectory=gs://[BUCKET]/YYYY/MM/dd/HH/ \ # partitions by date
--outputFilenamePrefix=output \ # optional
--shardTemplate=-W-P-SSSSS-of-NNNNN \ # optional
--outputFilenameSuffix=.txt \ # optional
--windowDuration=5 \ # optional, in minutes
--compression=none \ # optional, gzip, bz2 or none
--numShards=1 # optional
```

To display the help message:

```bash
./bin/snowplow-google-cloud-storage-loader --help
```

To display documentation about Cloud Storage Loader-specific options:

```bash
./bin/snowplow-google-cloud-storage-loader --help=com.snowplowanalytics.storage.googlecloudstorage.loader.Options
```

## Configuration

### Cloud Storage Loader specific options

- `--inputSubscription=String` The Cloud Pub/Sub subscription to read from, formatted like projects/\[PROJECT]/subscriptions/\[SUB]. Required.
- `--outputDirectory=gs://[BUCKET]/` The Cloud Storage directory to output files to, ending in /. Required.
- `--outputFilenamePrefix=String` The prefix for output files. Default: output. Optional.
- `--shardTemplate=String` A valid shard template as described [here](https://javadoc.io/static/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all/1.7.0/com/google/cloud/dataflow/sdk/io/ShardNameTemplate.html), which will be part of the filenames. Default: `-W-P-SSSSS-of-NNNNN`. Optional.
- `--outputFilenameSuffix=String` The suffix for output files. Default: .txt. Optional.
- `--windowDuration=Int` The window duration in minutes. Default: 5. Optional.
- `--compression=String` The compression used (gzip, bz2 or none). Note that bz2 can't be loaded into BigQuery. Default: no compression. Optional.
- `--numShards=Int` The maximum number of output shards produced when writing. Default: 1. Optional.
- `--dateFormat=YYYY/MM/dd/HH/` A date format string used for partitioning via date in `outputDirectory` and `partitionedOutputDirectory`. Default: `YYYY/MM/dd/HH/`. Optional. For example, the date format `YYYY/MM/dd/HH/` would produce a directory structure like this:
  ```bash
  gs://bucket/
    └── 2022
      └── 12
        └── 15
          ├── ...
          ├── 18
          ├── 19
          ├── 20
          └── ...
  ```
- `--partitionedOutputDirectory=gs://[BUCKET]/` The Cloud Storage directory to output files to, partitioned by schema, ending with /. Unpartitioned data will be sent to `outputDirectory`. Optional.

### Dataflow options

To run the Cloud Storage Loader on Dataflow, it is also necessary to specify additional configuration options. None of these options have default values, and they are all required.

- `--runner=DataFlowRunner` Passing the string `DataFlowRunner` specifies that we want to run on Dataflow.
- `--jobName=[NAME]` Specify a name for your Dataflow job that will be created.
- `--project=[PROJECT]` The name of your GCP project.
- `--streaming=true` Pass `true` to notify Dataflow that we're running a streaming application.
- `--workerZone=[ZONE]` The [zone](https://cloud.google.com/compute/docs/regions-zones) where the Dataflow nodes (effectively [GCP Compute Engine](https://cloud.google.com/compute/) nodes) will be launched.
- `--region=[REGION]` The [region](https://cloud.google.com/compute/docs/regions-zones) where the Dataflow job will be launched.
- `--gcpTempLocation=gs://[BUCKET]/` The GCS bucket where temporary files necessary to run the job (e.g. JARs) will be stored.

The list of all the options can be found at <https://cloud.google.com/dataflow/pipelines/specifying-exec-params#setting-other-cloud-pipeline-options>.

---

# Warehouse and lake data loaders
> Load Snowplow enriched events into data warehouses and lakes including BigQuery, Redshift, Snowflake, Databricks, and S3.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/

Snowplow provides loader applications for several different warehouse and lakes, across different clouds.

Choose a loader based on your use case and data needs.

---

# Lake Loader configuration reference
> Configure Lake Loader for Delta Lake and Iceberg tables with Kinesis, Pub/Sub, and Kafka stream settings for data lakes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/

The configuration reference in this page is written for Lake Loader `0.9.1`

### License

The Lake Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {
  "accept": true
}
```

### Table configuration

**Delta Lake:**

| Parameter                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.location`               | Required, e.g. `gs://mybucket/events`. URI of the bucket location to which to write Snowplow enriched events in Delta format. The URI should start with the following prefix:* `s3a://` on AWS
* `gs://` on GCP
* `abfs://` on Azure                                                                                                                                                                                                                                               |
| `output.good.deltaTableProperties.*` | Optional. A map of key/value strings corresponding to Delta's table properties. These can be anything [from the Delta table properties documentation](https://docs.delta.io/latest/table-properties.html). The default properties include configuring Delta's [data skipping feature](https://docs.delta.io/latest/optimizations-oss.html#data-skipping) for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |

**Iceberg / Glue:**

| Parameter                              | Description                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.type`                     | Required, set this to `Iceberg`                                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.catalog.type`             | Required, set this to `Glue`                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.location`                 | Optional, e.g. `s3a://mybucket/events`. URI of the bucket location to which to write Snowplow enriched events in Iceberg format. The URI should start with `s3a://`. If not provided, the catalog's default warehouse location will be used.                                                                                                                                                                             |
| `output.good.database`                 | Required. Name of the database in the Glue catalog                                                                                                                                                                                                                                                                                                                                                                       |
| `output.good.table`                    | Required. The name of the table in the Glue database                                                                                                                                                                                                                                                                                                                                                                     |
| `output.good.icebergTableProperties.*` | Optional. A map of key/value strings corresponding to Iceberg's table properties. These can be anything [from the Iceberg table properties documentation](https://iceberg.apache.org/docs/latest/configuration/). The default properties include configuring Iceberg's column-level statistics for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |
| `output.good.catalog.options.*`        | Optional. A map of key/value strings which are passed to the catalog configuration. These can be anything [from the Iceberg catalog documentation](https://iceberg.apache.org/docs/latest/aws/) e.g. `"glue.id": "1234567"`                                                                                                                                                                                              |

**Iceberg / REST:**

> **Note:** The REST catalog integration has been tested with Snowflake Open Catalog.

| Parameter                              | Description                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `output.good.type`                     | Required, set this to `Iceberg`                                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.catalog.type`             | Required, set this to `Rest`                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.catalog.uri`              | Required. URI of the REST catalog server, e.g. `http://localhost:8080`                                                                                                                                                                                                                                                                                                                                                   |
| `output.good.catalog.name`             | Required. Name of the catalog                                                                                                                                                                                                                                                                                                                                                                                            |
| `output.good.location`                 | Optional. URI of the bucket location to which to write Snowplow enriched events in Iceberg format. The URI should start with `s3a://`, `gs://`, or `abfs://` depending on your cloud provider. If not provided, the catalog's default warehouse location will be used.                                                                                                                                                   |
| `output.good.database`                 | Required. Name of the database in the catalog                                                                                                                                                                                                                                                                                                                                                                            |
| `output.good.table`                    | Required. The name of the table in the database                                                                                                                                                                                                                                                                                                                                                                          |
| `output.good.icebergTableProperties.*` | Optional. A map of key/value strings corresponding to Iceberg's table properties. These can be anything [from the Iceberg table properties documentation](https://iceberg.apache.org/docs/latest/configuration/). The default properties include configuring Iceberg's column-level statistics for the important Snowplow timestamp columns: `load_tstamp`, `collector_tstamp`, `derived_tstamp`, `dvce_created_tstamp`. |
| `output.good.catalog.options.*`        | Optional. A map of key/value strings which are passed to the catalog configuration. These can be anything [from the Iceberg REST catalog documentation](https://iceberg.apache.org/docs/latest/rest-catalog/), such as authentication credentials.                                                                                                                                                                       |

***

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-lake-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                           |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `input.maxRetries` (since 0.9.0)                    | Optional. Default value 10. Maximum number of retries for AWS SDK operations when reading from Kinesis.                                                                                                                                                                                                                                      |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |
| `output.bad.maxRetries` (since 0.8.0)               | Optional. Default value 10. Maximum number of retries by Kinesis Client.                                                                                                                                                                                                                                                                     |

**GCP:**

| Parameter                                                   | Description                                                                                                                                                                                                                                                                                                                                                             |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                        | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                          |
| `input.parallelPullFactor`                                  | Optional. Default value 0.5. `parallelPullFactor * cpu count` will determine the number of threads used internally by the Pub/Sub client library for fetching events                                                                                                                                                                                                    |
| `input.durationPerAckExtension`                             | Optional. Default value `600 seconds`. Pub/Sub ack deadlines are extended for this duration when needed. A sensible value is double the size of the `windowing` config parameter, but no higher than 10 minutes.                                                                                                                                                        |
| `input.minRemainingAckDeadline`                             | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `600 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `60 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`                                  | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                    |
| `input.retries.transientErrors.delay` (since 0.8.0)         | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                              |
| `input.retries.transientErrors.attempts` (since 0.8.0)      | Optional. Default value `10`. Max number of attempts, after which Loader will crash and exit                                                                                                                                                                                                                                                                            |
| `output.bad.topic`                                          | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                     |
| `output.bad.batchSize`                                      | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                       |
| `output.bad.requestByteThreshold`                           | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                          |
| `output.bad.maxRecordSize`                                  | Optional. Default value 10000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                          |
| `output.bad.retries.transientErrors.delay` (since 0.8.0)    | Optional. Default value `100 millis`. Backoff delay for follow-up attempts                                                                                                                                                                                                                                                                                              |
| `output.bad.retries.transientErrors.attempts` (since 0.8.0) | Optional. Default value `10`. Max number of attempts, after which Loader will crash and exit                                                                                                                                                                                                                                                                            |

**Azure:**

| Parameter                           | Description                                                                                                                                                                                  |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                   | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                         |
| `input.bootstrapServers`            | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                                |
| `input.consumerConf.*`              | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html).      |
| `input.commitTimeout` (since 0.8.0) | Optional. Default value 15 seconds. The time to wait for offset commits to complete. If an offset commit doesn't complete within this time, a CommitTimeoutException will be raised instead. |
| `output.bad.topicName`              | Required. Name of the Kafka topic that will receive failed events.                                                                                                                           |
| `output.bad.bootstrapServers`       | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                                 |
| `output.bad.producerConf.*`         | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html).      |
| `output.bad.maxRecordSize`          | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                                  |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>     "security.protocol": "SASL_SSL"
>     "sasl.mechanism": "PLAIN"
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
> }
> ```

***

## Other configuration options

| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `windowing`                               | Optional. Default value `5 minutes`. Controls how often the loader writes/commits pending events to the lake.                                                                                                                                                                                                                                                                                                                                              |
| `exitOnMissingIgluSchema`                 | Optional. Default value `true`. Whether the loader should crash and exit if it fails to resolve an Iglu Schema. We recommend `true` because Snowplow enriched events have already passed validation, so a missing schema normally indicates an error that needs addressing. Change to `false` so events go the failed events stream instead of crashing the loader.                                                                                        |
| `respectIgluNullability`                  | Optional. Default value `true`. Whether the output parquet files should declare nested fields as non-nullable according to the Iglu schema. When `true`, nested fields are nullable only if they are not required fields according to the Iglu schema. When `false`, all nested fields are defined as nullable in the output table's schemas. Set this to `false` if you use a query engine that dislikes non-nullable nested fields of a nullable struct. |
| `skipSchemas`                             | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to the lake. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded.                                                                                                                                                         |
| `spark.conf.*`                            | Optional. A map of key/value strings which are passed to the internal spark context.                                                                                                                                                                                                                                                                                                                                                                       |
| `spark.taskRetries`                       | Optional. Default value 3. How many times the internal spark context should be retry a task in case of failure                                                                                                                                                                                                                                                                                                                                             |
| `retries.setupErrors.delay`               | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how the lake is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                                                                                                                                 |
| `retries.transientErrors.delay`           | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                                                                                                                           |
| `retries.transientErrors.attempts`        | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.hostname`      | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.port`          | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                                                                                                                               |
| `monitoring.metrics.statsd.tags.*`        | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.period`        | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.prefix`        | Optional. Default `snowplow.lakeloader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                                                                                                                                           |
| `monitoring.webhook.endpoint`             | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Snowflake is set up for this loader.                                                                                                                                                                                                                                                                            |
| `monitoring.webhook.tags.*`               | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                                                                                                                            |
| `monitoring.webhook.heartbeat.*`          | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.healthProbe.port`             | Optional. Default value `8000`. Open a HTTP server that returns OK only if the app is healthy.                                                                                                                                                                                                                                                                                                                                                             |
| `monitoring.healthProbe.unhealthyLatency` | Optional. Default value `15 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time.                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.dsn`                   | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                                     |
| `monitoring.sentry.tags.*`                | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                                                                                                                         |
| `telemetry.disable`                       | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                                                                  |
| `telemetry.userProvidedId`                | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                                                                        |
| `inMemBatchBytes`                         | Optional. Default value 50000000. Controls how many events are buffered in memory before saving the batch to local disk. The default value works well for reasonably sized VMs. For smaller VMs (e.g. less than 2 cpu core, 8 GG memory) consider decreasing this value.                                                                                                                                                                                   |
| `cpuParallelismFraction`                  | Optional. Default value 0.75. Controls how the app splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and cpuParallelismFraction = 0.75, then we process 3 batches concurrently. The default value works well for most workloads.                                                                                                                                                         |
| `numEagerWindows`                         | Optional. Default value 1. Controls how eagerly the loader starts processing the next timed window even when the previous timed window is still finalizing (committing into the lake). By default, we start processing a timed windows if the previous 1 window is still finalizing, but we do not start processing a timed window if any more older windows are still finalizing. The default value works well for most workloads.                        |
| `http.client.maxConnectionsPerServer`     | Optional. Default value 4. Configures the internal HTTP client used for Iglu resolver, alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time. For Iglu Server in particular, this avoids overwhelming the server with multiple concurrent requests.                                                                                                                                                          |

---

# Open Table Format Lake Loader
> Load Snowplow events to data lakes using Delta or Iceberg table formats on S3, GCS, or Azure ADLS Gen2.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/

The Lake Loader is an application that loads Snowplow events to a cloud storage bucket using Open Table Formats.

> **Info:** The Lake Loader supports the two major Open Table Formats: [Delta](https://delta.io/) and [Iceberg](https://iceberg.apache.org/).
>
> For Iceberg tables, the loader supports [AWS Glue](https://docs.aws.amazon.com/glue/) and [Iceberg REST](https://iceberg.apache.org/docs/latest/rest-catalog/) as catalogs. The REST catalog integration has been tested with Snowflake Open Catalog.

**AWS:**

On AWS the Lake Loader continually pulls events from Kinesis and writes to S3.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["S3"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any AWS VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-aws:0.9.1
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-aws:0.9.1 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**GCP:**

On GCP the Lake Loader continually pulls events from Pub/Sub and writes to GCS.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["GCS"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any GCP VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-gcp:0.9.1
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-gcp:0.9.1 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**Azure:**

On Azure the Lake Loader continually pulls events from Kafka and writes to ADLS Gen 2.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["ADLS Gen 2"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any Azure VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-azure:0.9.1
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-azure:0.9.1 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

***

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.aws.minimal.hocon)

**GCP:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.gcp.minimal.hocon)

**Azure:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.azure.minimal.hocon)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) for all possible configuration parameters.

### Windowing

"Windowing" is an important config setting, which controls how often the Lake Loader commits a batch of events to the data lake. If you adjust this config setting, you should be aware that data lake queries are most efficient when the size of the parquet files in the lake are relatively large.

- If you set this to a **low** value, the loader will write events to the lake more frequently, reducing latency. However, the output parquet files will be smaller, which will make querying the data less efficient.
- Conversely, if you set this to a **high** value, the loader will generate bigger output parquet files, which are efficient for queries — at the cost of events arriving to the lake with more delay.

The default setting is `5 minutes`. For moderate to high volumes, this value strikes a nice balance between the need for large output parquet files and the need for reasonably low latency data.

```text
{
  "windowing": "5 minutes"
}
```

If you tune this setting correctly, then your lake can support efficient analytic queries without the need to run an `OPTIMIZE` job on the files.

### Iglu

The Lake Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

### Metrics

The Lake Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:

| Metric                      | Definition                                                                                                                                                                                                                                      |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_committed`          | A count of events that are successfully written and committed to the lake. Because the loader works in timed windows of several minutes, this metric has a "spiky" value, which is often zero and then periodically spikes up to larger values. |
| `events_received`           | A count of events received by the loader. Unlike `events_committed` this is a smooth varying metric, because the loader is constantly receiving events throughout a timed window.                                                               |
| `events_bad`                | A count of failed events that could not be loaded, and were instead sent to the bad output stream.                                                                                                                                              |
| `latency_millis`            | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader.                                                                                                   |
| `processing_latency_millis` | For each window of events, the time in milliseconds from when the first event is read from the stream, until all events are written and committed to the lake.                                                                                  |
| `e2e_latency_millis`        | The end-to-end latency of the snowplow pipeline. For each window of events, the time in milliseconds from when the first event was received by the collector, until all events are written and committed to the lake.                           |
| `table_data_files_total`    | The total number of data files in the table after a commit. This metric helps monitor table growth and the effectiveness of file compaction strategies.                                                                                         |
| `table_snapshots_retained`  | The number of snapshots retained in the table metadata. This metric helps monitor snapshot accumulation and the effectiveness of snapshot expiration policies.                                                                                  |

See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) for how to configure the StatsD receiver.

**Telemetry notice**

By default, Snowplow collects telemetry data for Lake Loader (since version 0.1.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Lake maintenance jobs
> Maintain Delta Lake and Iceberg tables with OPTIMIZE operations and snapshot expiration to manage file size and storage costs.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/

**Delta Lake:**

The [Delta documentation](https://docs.delta.io/latest/best-practices.html#-delta-compact-files) makes recommendations for running regular maintenance jobs to get the best performance from your lake. This guide expands on those recommendations specifically for your Snowplow events lake.

The Snowplow Lake Loader **does not** automatically run the maintenance tasks described below.

## Compact data files

We recommend that you schedule a compaction job to run once per day.

Your daily compaction job should operate on files that were loaded in the previous calendar day, i.e. the last completed timestamp partition. For example, if you run the job via SQL, then use a `WHERE` clause on `load_tstamp_date < current_date()`:

```sql
OPTIMIZE <your_table> WHERE load_tstamp_date < current_date()
```

Data compaction has two benefits for Snowplow data:

1. Queries are more efficient when the underlying parquet files are large. After you compact your files, you will benefit from this whenever you run queries over your historic data, i.e. not just the most recently loaded events.
2. When there are fewer data files, the size of the table's delta log files is smaller. This reduces the overhead of creating a new delta log file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.

## Vacuum data files

We recommend that you schedule a Vacuum job to run once per week.

A vacuum is needed to clean up unreferenced data files that were logically deleted by the daily compaction jobs.

```sql
VACUUM <your_table>;
```

Unreferenced data files do not negatively impact query performance or write performance. But they do contribute to storage costs.

Vacuum jobs need to list every file in the lake directory. In large lakes, there might be a lot of files, requiring the job to use a lot of Spark compute resources. This is why we recommend to run it infrequently to offset this impact.

**Iceberg:**

> **Note:** We support different catalog options for Iceberg lakes. The instructions below are not necessary when using Snowflake Open Catalog.

The [Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/) makes recommendations for running regular maintenance jobs to get the best performance from your lake. This guide expands on those recommendations specifically for your Snowplow events lake.

The Snowplow Lake Loader **does not** automatically run the maintenance tasks described below.

## Expire snapshots

We recommend that you schedule the `expireSnapshots` Iceberg action to run once per day.

The Snowplow Lake Loader is a continuously-running streaming loader, so it creates new snapshots very frequently, each time it commits more events into the lake. So it is especially important to manage the total number of snapshots held by the Iceberg metadata.

There are two benefits of expiring snapshots in your Snowplow lake:

1. The snapshot metadata files can be much smaller, because the list of metadata files to track is much smaller. This reduces the overhead of creating a new snapshot file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.
2. If you regularly run compaction jobs (see below) then you will amass lots of small parquet files, which have since been rewritten into larger parquet files. By expiring snapshots, you will delete the redundant small data files, which will save you some storage cost.

For example, if you run the action via a Spark SQL procedure:

```sql
CALL catalog_name.system.expire_snapshots(
    table => 'snowplow.events',
    stream_results => true
)
```

## Compact data files

We recommend that you schedule the `rewriteDataFiles` Iceberg action to run once per day.

Your daily compaction job should operate on files that were loaded in the previous calendar day, i.e. the last completed timestamp partition. For example, if you run the action via a Spark SQL procedure, then use a `where` clause on `load_tstamp < current_date()`:

```sql
CALL catalog_name.system.rewrite_data_files(
    table => 'snowplow.events',
    where => 'load_tstamp < current_date()'
);
```

The `rewriteDataFiles` action has two benefits for Snowplow data:

1. Queries are more efficient when the underlying parquet files are large. After you compact your files, you will benefit from this whenever you run queries over your historic data, i.e. not just the most recently loaded events.
2. When there are fewer data files, the size of the table's manifest files is smaller. This reduces the overhead of creating a new manifest file, and thus improves the performance of the Lake Loader when committing new events into the lake. This becomes especially important as your lake grows in size over time.

## Remove orphan files

We recommend that you schedule the `removeOrphanFiles` Iceberg action to run once per month.

> **Tip:** Your Iceberg table should have `write.metadata.delete-after-commit.enabled=true` set in the table properties. If your Iceberg table was originally created by a Lake Loader older than version 0.7.0, then please run:
>
> ```sql
> ALTER TABLE <your_table>
> SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled'='true')
> ```

As long as `delete-after-commit` is enabled in the table properties, the Snowplow Lake Loader should not create orphan files under normal circumstances. But it is technically still possible for the loader to create orphan files under rare exceptional circumstances, e.g. transient network errors, or if the loader exits without completing a graceful shutdown. Orphan files do not negatively impact query performance or write performance. But they do contribute to storage costs.

This action needs to list every file in the lake directory. In large lakes that might be a very large number of files. In large lakes, there might be a lot of files, requiring the job to use a lot of Spark compute resources. This is why we recommend to run it infrequently to offset this impact.

```sql
CALL catalog_name.system.remove_orphan_files(
    table => 'snowplow.events'
);
```

***

---

# Partitioning for data lakes
> Data lake partitioning by load_tstamp and event_name for efficient querying and incremental processing in Delta and Iceberg tables.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/partitions/

A lake created by the Lake Loader has two levels of partitioning:

1. By the date that the event is loaded to the lake. For Iceberg, we use a [partition transform](https://iceberg.apache.org/spec/#partitioning) `date(load_tstamp)`. For Delta, we create a [generated column](https://delta.io/blog/2023-04-12-delta-lake-generated-columns/) called `date_load_tstamp` defined as `generatedAlwaysAs(CAST(load_tstamp AS DATE))`.
2. By the `event_name` field.

This structure of partitioning works very well with queries that filter on `load_tstamp` and/or `event_name`. It works especially well with incremental models, which only ever process the most recently loaded events.

> **Note:** If you are using Snowplow's DBT packages, then set the `session_timestamp` variable to `load_stamp` to match the table's partitioning.

If you run a query with a clause like `WHERE load_tstamp > ?`, then your query engine can go directly to the partition containing the relevant files. Even better, because Delta and Iceberg collect file-level statistics, such a query can go directly to the relevant files within the partition, matching exactly the `load_tstamp` of interest.

If you often write queries over a single type of event, e.g. `WHERE event_name = 'add_to_cart'` then the query engine can do a very efficient query over the parquet files for the specific event.

> **Note:** The Lake Loader has been optimized for writing into a lake with the default partitioning, and the loader will not perform so well with any other partitioning. For these reasons, we strongly advise that you do not change the partitioning structure of your lake.

---

# S3 loader configuration reference
> Configure S3 Loader with Kinesis input, S3 output, compression, partitioning, and monitoring settings for event archiving.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/

S3 Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run S3 Loader, configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

This is a complete list of the options that can be configured in the S3 loader HOCON config file. The [example configs in github](https://github.com/snowplow/snowplow-s3-loader/tree/master/config) show how to prepare an input file.

| parameter                                                         | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                                | Required. Name of the kinesis stream from which to read                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `input.appName`                                                   | Optional. Default: `snowplow-s3-loader`. Kinesis Client Lib app name (corresponds to DynamoDB table name)                                                                                                                                                                                                                                                                                                                                                                   |
| `input.initialPosition.type` (since 3.0.0)                        | Optional. Default: `TRIM_HORIZON`. Set the initial position to consume the Kinesis stream. Possible values: `LATEST` (most recent data), `TRIM_HORIZON` (oldest available data), `AT_TIMESTAMP` (start from the record at or after the specified timestamp)                                                                                                                                                                                                                 |
| `input.initialPosition.timestamp` (since 3.0.0)                   | Required for `AT_TIMESTAMP`. E.g. `2020-07-17T10:00:00Z`                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `input.retrievalMode.type` (since 3.0.0)                          | Optional. Default: `Polling`. Set the mode for retrieving records. Possible values: `Polling` or `FanOut`                                                                                                                                                                                                                                                                                                                                                                   |
| `input.retrievalMode.maxRecords` (since 3.0.0)                    | Required for `Polling`. Default: `750`. Maximum size of a batch returned by a call to `getRecords`. Records are checkpointed after a batch has been fully processed, thus the smaller `maxRecords`, the more often records can be checkpointed into DynamoDb, but possibly reducing the throughput                                                                                                                                                                          |
| `input.retrievalMode.idleTimeBetweenReads` (since 3.1.0)          | Optional for `Polling`. Default: `1500 millis`. Idle time between `getRecords` requests.                                                                                                                                                                                                                                                                                                                                                                                    |
| `input.workerIdentifier` (since 3.0.0)                            | Optional. Default: host name. Name of this KCL worker used in the DynamoDB lease table                                                                                                                                                                                                                                                                                                                                                                                      |
| `input.leaseDuration` (since 3.0.0)                               | Optional. Default: `10 seconds`. Duration of shard leases. KCL workers must periodically refresh leases in the DynamoDB table before this duration expires                                                                                                                                                                                                                                                                                                                  |
| `input.maxLeasesToStealAtOneTimeFactor` (since 3.0.0)             | Optional. Default: `2.0`. Controls how to pick the max number of leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the KCL to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency                                                                                                                                               |
| `input.checkpointThrottledBackoffPolicy.minBackoff` (since 3.0.0) | Optional. Default: `100 millis`. Minimum backoff before retrying when DynamoDB provisioned throughput exceeded                                                                                                                                                                                                                                                                                                                                                              |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` (since 3.0.0) | Optional. Default: `1 second`. Maximum backoff before retrying when DynamoDB provisioned throughput limit exceeded                                                                                                                                                                                                                                                                                                                                                          |
| `input.debounceCheckpoints` (since 3.0.0)                         | Optional. Default: `10 seconds`. How frequently to checkpoint our progress to the DynamoDB table. By increasing this value, we can decrease the write-throughput requirements of the DynamoDB table                                                                                                                                                                                                                                                                         |
| `input.customEndpoint`                                            | Optional. Override the default endpoint for kinesis client api calls                                                                                                                                                                                                                                                                                                                                                                                                        |
| `input.maxRetries` (since 3.1.0)                                  | Optional. Default: `10`. Maximum number of times the Kinesis client will retry AWS API calls in case of failure                                                                                                                                                                                                                                                                                                                                                             |
| `input.apiCallAttemptTimeout` (since 3.1.0)                       | Optional. Default: `15 seconds`. Maximum time of a single attempt of an AWS API operation.                                                                                                                                                                                                                                                                                                                                                                                  |
| `output.good.path`                                                | Required. Full path to output data, e.g. `s3://acme-snowplow-output/`                                                                                                                                                                                                                                                                                                                                                                                                       |
| `output.good.partitionFormat` (since 2.1.0)                       | Optional. Configures how files are partitioned into S3 directories. When loading self describing jsons, you might choose to partition by `{vendor}.{name}/model={model}/date={yyyy}-{MM}-{dd}`. Valid substitutions are `{vendor}`, `{name}`, `{format}`, `{model}` for self-describing jsons; and `{yyyy}`, `{MM}`, `{dd}`, `{HH}` for year, month, day and hour. Defaults to `{vendor}.{schema}` when loading self-describing JSONs or blank when loading enriched events |
| `output.good.filenamePrefix`                                      | Optional. Add a prefix to files                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `output.good.compression`                                         | Optional. Has to be `GZIP` (default)                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.streamName`                                           | Required. Name of a kinesis stream to output failures                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 3.0.0)      | Optional. Default: `100 milliseconds`. Minimum backoff before retrying when writing fails with exceeded kinesis write throughput                                                                                                                                                                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 3.0.0)      | Optional. Default: `1 second`. Maximum backoff before retrying when writing fails with exceeded kinesis write throughput                                                                                                                                                                                                                                                                                                                                                    |
| `output.bad.recordLimit` (since 3.0.0)                            | Optional. Default: `500`. Maximum allowed to records we are allowed to send to Kinesis in 1 PutRecords request                                                                                                                                                                                                                                                                                                                                                              |
| `output.bad.byteLimit` (since 3.0.0)                              | Optional. Default: `5242880`. Maximum allowed to bytes we are allowed to send to Kinesis in 1 PutRecords request                                                                                                                                                                                                                                                                                                                                                            |
| `output.bad.maxRetries` (since 3.1.0)                             | Optional. Default: `10`. Maximum number of times the Kinesis client will retry AWS API calls in case of failure                                                                                                                                                                                                                                                                                                                                                             |
| `purpose`                                                         | Required. `ENRICHED_EVENTS` for enriched events or `SELF_DESCRIBING` for self-describing data                                                                                                                                                                                                                                                                                                                                                                               |
| `batching.maxBytes` (since 3.0.0)                                 | Optional. Default: `67108864`. After this amount of compressed bytes have been added to the buffer it gets written to a file (unless `maxDelay` is reached before)                                                                                                                                                                                                                                                                                                          |
| `batching.maxDelay` (since 3.0.0)                                 | Optional. Default: `2 minutes`. After this delay has elapsed the buffer gets written to a file (unless `maxBytes` is reached before)                                                                                                                                                                                                                                                                                                                                        |
| `cpuParallelismFactor` (since 3.0.0)                              | Optional. Default: `1`. Controls how the app splits the workload into concurrent batches which can be run in parallel, e.g. if there are 4 available processors and `cpuParallelismFactor = 0.75` then we process 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU                                                                                                                                                   |
| `uploadParallelismFactor` (since 3.0.0)                           | Optional. Default: `2`. Controls number of upload jobs that can be run in parallel, e.g. if there are 4 available processors and `sinkParallelismFraction = 2` then we run 8 upload job concurrently. Adjusting this value can cause the app to use more or less of the available CPU                                                                                                                                                                                       |
| `initialBufferSize` (since 3.0.0)                                 | Optional. Default: none. Overrides the initial size of the byte buffer that holds the compressed events in-memory before they get written to a file. If not set, the initial size is picked dynamically based on other configuration options. The default is known to work well. Increasing this value is a way to reduce in-memory copying, but comes at the cost of increased memory usage                                                                                |
| `monitoring.sentry.dsn`                                           | Optional. For tracking uncaught run time exceptions                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.hostname`                              | Optional. For sending loading metrics (latency and event counts) to a `statsd` server                                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.port`                                  | Optional. Port of the statsd server                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.tags`                                  | E.g.`{ "key1": "value1", "key2": "value2" }`. Tags are used to annotate the statsd metric with any contextual information                                                                                                                                                                                                                                                                                                                                                   |
| `monitoring.metrics.statsd.prefix`                                | Optional. Default `snoplow.s3loader`. Configures the prefix of statsd metric names                                                                                                                                                                                                                                                                                                                                                                                          |
| `monitoring.healthProbe.port` (since 3.0.0)                       | Optional. Default: `8080`. Port of the HTTP server that returns OK only if the app is healthy                                                                                                                                                                                                                                                                                                                                                                               |
| `monitoring.healthProbe.unhealthyLatency` (since 3.0.0)           | Optional. Default: `2 minutes`. Health probe becomes unhealthy if any received event is still not fully processed before this cutoff time                                                                                                                                                                                                                                                                                                                                   |

---

# S3 Loader
> Archive Snowplow events from Kinesis to S3 in LZO or Gzip format for raw payloads, enriched events, and failed events.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/

Snowplow S3 Loader consumes records from an [Amazon Kinesis](http://aws.amazon.com/kinesis/) stream and writes them to [S3](http://aws.amazon.com/s3/). A typical Snowplow pipeline would use the S3 loader in several places:

- Load enriched events from the "enriched" stream. These serve as input for [the RDB loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) when loading to a warehouse.
- Load failed events from the "bad" stream.

Records that can't be successfully written to S3 are written to a [second Kinesis stream](https://github.com/snowplow/snowplow-s3-loader/blob/master/examples/config.hocon.sample#L75) with the error message.

## Output format : GZIP

The records are treated as byte arrays containing UTF-8 encoded strings (whether CSV, JSON or TSV). New lines are used to separate records written to a file. This format can be used with the Snowplow Kinesis Enriched stream, among other streams.

Gzip encoding is generally used for both enriched data and bad data.

## Running

### Available on Terraform Registry

A Terraform module which deploys the Snowplow S3 Loader on AWS EC2 for use with Kinesis. For installing in other environments, please see the other installation options below.

### Docker image

We publish two different flavours of the docker image:

- `snowplow/snowplow-s3-loader:3.1.0`

- `snowplow/snowplow-s3-loader:3.1.0-distroless` (lightweight alternative)

Here is a standard command to run the loader on a EC2 instance in AWS:

```bash
docker run \
    -d \
    --name snowplow-s3-loader \
    --restart always \
    --log-driver awslogs \
    --log-opt awslogs-group=snowplow-s3-loader \
    --log-opt awslogs-stream='ec2metadata --instance-id' \
    --network host \
    -v $(pwd):/snowplow/config \
    -e 'JAVA_OPTS=-Xms512M -Xmx1024M -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN' \
    snowplow/snowplow-s3-loader:3.1.0 \
    --config /snowplow/config/config.hocon
```

---

# S3 Loader monitoring
> Monitor S3 Loader with StatsD metrics, Sentry error tracking, and Snowplow event tracking for application health and failures.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/monitoring/

The S3 loader has several types of monitoring built in, to help the pipeline operator: Statsd metrics, Sentry alerts, and Snowplow tracking.

## Statsd

[Statsd](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

When processing enriched events, the S3 loader can emit metrics to a statsd daemon describing every S3 file it writes. Here is a string representation of the metrics it sends:

```text
snowplow.s3loader.count:42|c|#tag1:value1
snowplow.s3loader.latency_collector_to_load:123|g|#tag1:value1
snowplow.s3loader.latency_millis:56|g|#tag1:value1
snowplow.s3loader.e2e_latency_millis:123|g|#tag1:value1
```

- `count`: total number of events that got written to S3.
- `latency_collector_to_load`: time difference between reaching the collector and getting loaded to S3 (only for enriched events). Will get deprecated eventually in favor of `e2e_latency_millis`.
- `latency_millis`: delay between the input record getting written to the stream and S3 loader starting to process it.
- `e2e_latency_millis`: same as `latency_collector_to_load`, which will get deprecated eventually and replaced with this metric.

Statsd monitoring is configured by setting the `monitoring.metrics.statsd` section in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/):

```json
"monitoring": {
  "metrics": {
    "hostname": "localhost"
    "port": 8125
    "tags": {
      "tag1": "value1"
      "tag2": "value2"
    }
    "prefix": "snowplow.s3loader"
  }
}
```

## Health probe

Starting with `3.0.0` version S3 loader gets a health probe, configured via the `monitoring.healthProbe` section (see the configuration reference).

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. The S3 loader can send an error report to sentry whenever something unexpected happens. The reasons for the error can then be explored in the sentry server’s UI.

Common reasons might be failure to read or write from Kinesis, or failure to write to S3.

Sentry monitoring is configured by setting the `monitoring.sentry.dsn` key in [the hocon file](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/) with the url of your sentry server:

```json
"monitoring": {
  "dsn": "http://sentry.acme.com"
}
```

---

# S3 Loader 1.0.0 configuration
> Configure S3 Loader 1.0.0 with HOCON settings for Kinesis or NSQ streams, compression, buffering, and monitoring.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/1-0-0-configuration/

The sink is configured using a HOCON file. These are the fields:

- `source`: Choose kinesis or nsq as a source stream
- `sink`: Choose between kinesis or nsq as a sink stream for failed events
- `aws.accessKey` and `aws.secretKey`: Change these to your AWS credentials. You can alternatively leave them as "default", in which case the [DefaultAWSCredentialsProviderChain](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html) will be used.
- `kinesis.initialPosition`: Where to start reading from the stream the first time the app is run. "TRIM\_HORIZON" for as far back as possible, "LATEST" for as recent as possibly, "AT\_TIMESTAMP" for after the specified timestamp.
- `kinesis.initialTimestamp`: Timestamp for "AT\_TIMESTAMP" initial position
- `kinesis.maxRecords`: Maximum number of records to read per GetRecords call
- `kinesis.region`: The Kinesis region name to use.
- `kinesis.appName`: Unique identifier for the app which ensures that if it is stopped and restarted, it will restart at the correct location.
- `kinesis.customEndpoint`: Optional endpoint url configuration to override aws kinesis endpoints. This can be used to specify local endpoints when using localstack.
- `kinesis.disableCloudWatch`: Optional override to disable CloudWatch metrics for KCL
- `nsq.channelName`: Channel name for NSQ source stream. If more than one application reading from the same NSQ topic at the same time, all of them must have unique channel name to be able to get all the data from the same topic.
- `nsq.host`: Hostname for NSQ tools
- `nsq.port`: HTTP port number for nsqd
- `nsq.lookupPort`: HTTP port number for nsqlookupd
- `stream.inStreamName`: The name of the input stream of the tool which you choose as a source. This should be the stream to which your are writing records with the Scala Stream Collector.
- `streams.outStreamName`: The name of the output stream of the tool which you choose as sink. This is stream where records are sent if the compression process fails.
- `streams.buffer.byteLimit`: Whenever the total size of the buffered records exceeds this number, they will all be sent to S3.
- `streams.buffer.recordLimit`: Whenever the total number of buffered records exceeds this number, they will all be sent to S3.
- `streams.buffer.timeLimit`: If this length of time passes without the buffer being flushed, the buffer will be flushed. **Note**: With NSQ streams, only record limit is taken into account. Other two option will be ignored.
- `s3.region`: The AWS region for the S3 bucket
- `s3.bucket`: The name of the S3 bucket in which files are to be stored
- `s3.format`: The format the app should write to S3 in (`lzo` or `gzip`)
- `s3.maxTimeout`: The maximum amount of time the app attempts to PUT to S3 before it will kill itself

### Monitoring

It's possible to include Snowplow monitoring in the application. This is setup through the `monitoring` section at the bottom of the config file:

- `monitoring.snowplow.collectorUri` your snowplow collector URI
- `monitoring.snowplow.appId` the app-id used in decorating the events sent

To disable Snowplow monitoring, just remove the entire `monitoring` section from the config.

---

# S3 Loader 2.0.0 upgrade guide
> Upgrade S3 Loader to 2.0.0 with configuration refactoring, purpose property, and new Sentry and StatsD monitoring features.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/2-0-0-upgrade-guide/

## Caution

If you're upgrading from Snowplow pre-R119 and S3 Loader pre-version 0.7.0 you have to upgrade to version 0.7.0 or 1.0.0 first in order to split bad data produced during transition period.

In [Snowplow R119](https://snowplowanalytics.com/blog/2020/05/12/snowplow-release-r119/) we introduced a new self-describing bad rows format. S3 Loader 0.7.0 was the first version capable of partitioning self-describing data based on its schema. 0.7.0 and 1.0.0 are capable to recognize at runtime whether old or new format is consumed and use `partitionedBucket` output path only if necessary, so both formats can be consumed.

S3 Loader 2.0.0 supports only new self-describing format and will be raising exceptions if legacy bad data is pushed.

## Config file

In 2.0.0 the S3 Loader went through a major configuration refactoring. A [sample config](https://github.com/snowplow/snowplow-s3-loader/blob/2.0.0/config/config.hocon.sample) is available in GitHub repository.

- No more `aws` property allowing to hardcode credentials - [default credentials chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) is used
- NSQ support has been dropped
- Instead of `kinesis` and `s3` the topology now is represented as `input` (Kinesis Stream) and `output` (S3 bucket and a Kinesis Stream for bad data)
- `partitionedBucket` property has been removed (see Caution above)
- New `purpose` property allowing Loader to recognize the data it works with: `ENRICHED` for enriched TSVs enabling latency monitoring, `SELF_DESCRIBING` generally for any self-describing JSON but usually used for bad rows and `RAW`

## New features

- `metrics.sentry.dsn` can be used to track exceptions, including internal KCL exceptions
- `metricsd.statsd` can be used to send observability data to StatsD-compatible server

---

# S3 Loader 2.2.x upgrade guide
> Upgrade S3 Loader to 2.2.x with separate Docker images for GZip, LZO, and distroless variants for security improvements.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/2-2-0-upgrade-guide/

Starting from the 2.2.0 release we started publishing three different flavours of the docker image.

- Pull the `:2.2.9` tag if you only need GZip output format

- Pull the `:2.2.9-lzo` tag if you also need LZO output format

- Pull the `:2.2.9-distroless` tag for an lightweight alternative to `:2.2.0`

```bash
docker pull snowplow/snowplow-s3-loader:2.2.9
docker pull snowplow/snowplow-s3-loader:2.2.9-lzo
docker pull snowplow/snowplow-s3-loader:2.2.9-distroless
```

We removed LZO support from the standard image, because it means we can more easily eliminate security vulnerabilities that are brought in from a dependency on hadoop version 2.

The "distroless" docker image is built from [a more lightweight base image](https://github.com/GoogleContainerTools/distroless). It provides some security advantages, because it carries only the minimal files and executables needed for the loader to run.

---

# 3.0.0 upgrade guide
> Guide for upgrading to S3 Loader 3.0.0, covering buffering changes, LZO deprecation, configuration refactoring, and filename changes.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/3-0-0-upgrade-guide/

S3 loader was using AWS SDK v1 which goes EOL at the end of 2025. Bumping to AWS SDK v2 required a full rewrite of the app.

## Buffering

S3 loader buffers the events into memory before writing them to S3. There are 2 key differences between the previous loader and the new one:

- In the previous loader we had one buffer per Kinesis shard, each buffer getting written to one file. In the new loader, records from all the Kinesis shards go to the same buffer and file. The consequence is that the new loader writes fewer but bigger files.

- In the previous loader, records were compressed after the buffer was full, before getting written to disk. In the new loader, records get compressed before getting added to the buffer. The consequence is that the new loader writes bigger files (very close to `maxBytes` if this limit is reached before `maxDelay`).

```mermaid
flowchart LR
  subgraph "Old S3 loader"
    buffer1["buffer"]
    buffer2["buffer"]
    buffer3["buffer"]
  end
  shard1["Kinesis shard"] --> buffer1 -->|"compression"| file1["file/"]
  shard2["Kinesis shard"] --> buffer2 -->|"compression"| file2["file/"]
  shard3["Kinesis shard"] --> buffer3 -->|"compression"| file3["file/"]
  classDef noborder stroke:none,fill:none
  class shard1,shard2,shard3,file1,file2,file3 noborder
```

```mermaid
flowchart LR
  subgraph "New S3 loader"
    buffer["buffer"]
  end
  shard1["Kinesis shard"] & shard2["Kinesis shard"] & shard3["Kinesis shard"] -->|"compression"| buffer --> file["file/"]
  classDef noborder stroke:none,fill:none
  class shard1,shard2,shard3,file noborder
```

## LZO deprecation

Starting from version `3.0.0`, S3 loader should only be used to load enriched events and bad rows (no more `purpose = "RAW"`). The reason for this is that storing the events emitted by the collector is redundant and it is not compatible with features that we have on the roadmap. LZO compression format is not supported any more (it was used in old batch pipelines). Only the following Docker images with GZIP get published:

- `snowplow/snowplow-s3-loader:3.1.0`

- `snowplow/snowplow-s3-loader:3.1.0-distroless` (lightweight alternative)

## Config file

In `3.0.0` S3 Loader went through a major configuration refactoring. A [sample config](https://github.com/snowplow/snowplow-s3-loader/blob/3.0.0/config/config.aws.reference.hocon) is available in GitHub repository.

These config fields have been removed:

- `region`: it is now retrieved from the region provider chain.
- `buffer.recordLimit`: only `maxDelay` and `maxBytes` are now used for the buffering.
- `monitoring.snowplow`: Snowplow tracking (sending events e.g. `app_initialized` or `app_heartbeat`) got removed.
- `output.s3.maxTimeout`

These sections/fields have been renamed:

- `output.s3` -> `output.good`
- `buffer.byteLimit` -> `batching.maxBytes`
- `buffer.timeLimit` -> `batching.maxDelay`
- `input.maxRecords` -> `input.retrievalMode.maxRecords`

For more details, refer to the [configuration reference](/docs/api-reference/loaders-storage-targets/s3-loader/configuration-reference/).

## Change in the filename

There is a change in the name of the files written to S3. In `2.x` the filename was `yyyy-MM-dd-HHmmss-<first_sequence_number_in_shard>-<last_sequence_number_in_shard>.gz`. In `3.0.0` the new filename is `yyyy-MM-dd-HHmmss-<uuid>.gz`. The reason for this change is that S3 loader `2.x` was writing the events one file per shard whereas the new loader is writing the events many shards to the same file.

---

# Upgrade guides for the S3 Loader
> Step-by-step upgrade guides for S3 Loader with configuration changes, breaking changes, and migration paths for major versions.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/s3-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of the S3 Loader.

---

# Schema translation to warehouse column types
> How Snowplow JSON schemas map to column types and structures in Redshift, BigQuery, Snowflake, Databricks, Iceberg, and Delta Lake.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/

[Self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number). This page explains what happens to this information in the warehouse.

## Location

Where can you find the data carried by a self-describing event or an entity?

**Redshift:**

Each type of self-describing event and each type of entity get their own dedicated tables. The name of such a table is composed of the schema vendor, schema name and its major version (more on versioning [later](#versioning)).

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting table              |
| --------------------- | ------------------------------------------- | ---------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `com_example_user_1`         |

Inside the table, there will be columns corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each column is the name of the schema field converted to snake case.

> **Warning:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {
  "type": "string",
  "maxLength": 100
}
```

It will be translated into a column called `last_name` (notice the underscore), of type `VARCHAR(100)`.

**BigQuery:**

**Version 2.x:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

**Version 1.x:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and full schema version (more on versioning [later](#versioning)).

Examples:

| Kind                  | Schema                                      | Resulting column                                       |
| --------------------- | ------------------------------------------- | ------------------------------------------------------ |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1_0_0` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1_0_0`               |

***

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

For self-describing events, the column will be of a `RECORD` type, while for entities the type will be `REPEATED RECORD` (because an event can have more than one entity attached).

Inside the record, there will be fields corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each record field is the name of the schema field converted to snake case.

> **Warning:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {
  "type": "string",
  "maxLength": 100
}
```

It will be translated into a field called `last_name` (notice the underscore), of type `STRING`.

**Snowflake:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

For self-describing events, the column will be of an `OBJECT` type, while for entities the type will be an `ARRAY` of objects (because an event can have more than one entity attached).

Inside the object, there will be keys corresponding to the fields in the schema. The values for the keys will be of type `VARIANT`.

> **Note:** If an event or entity includes fields not defined in the schema, those fields will be included in the object. However, remember that you need to set `additionalProperties` to `true` in the respective schema for such events and entities to pass schema validation.

For example, suppose you have the following field in the schema:

```json
"lastName": {
  "type": "string",
  "maxLength": 100
}
```

It will be translated into an object with a `lastName` key that points to a value of type `VARIANT`.

**Databricks, Iceberg, Delta:**

Each type of self-describing event and each type of entity get their own dedicated columns in the `events` table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

For self-describing events, the column will be of a `STRUCT` type, while for entities the type will be `ARRAY` of `STRUCT` (because an event can have more than one entity attached).

Inside the `STRUCT`, there will be fields corresponding to the fields in the schema. Their types are determined according to the logic described [below](#types).

> **Note:** The name of each record field is the name of the schema field converted to snake case.

> **Warning:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the warehouse.

For example, suppose you have the following field in the schema:

```json
"lastName": {
  "type": "string",
  "maxLength": 100
}
```

It will be translated into a field called `last_name` (notice the underscore), of type `STRING`.

**Synapse Analytics:**

Each type of self-describing event and each type of entity get their own dedicated columns in the underlying data lake table. The name of such a column is composed of the schema vendor, schema name and major schema version (more on versioning [later](#versioning)).

The column name is prefixed by `unstruct_event_` for self-describing events, and by `contexts_` for entities. _(In case you were wondering, those are the legacy terms for self-describing events and entities, respectively.)_

> **Note:** All characters are converted to lowercase and all symbols (like `.`) are replaced with an underscore.

Examples:

| Kind                  | Schema                                      | Resulting column                                   |
| --------------------- | ------------------------------------------- | -------------------------------------------------- |
| Self-describing event | `com.example/button_press/jsonschema/1-0-0` | `events.unstruct_event_com_example_button_press_1` |
| Entity                | `com.example/user/jsonschema/1-0-0`         | `events.contexts_com_example_user_1`               |

The column will be formatted as JSON — an object for self-describing events and an array of objects for entities (because an event can have more than one entity attached).

Inside the JSON object, there will be fields corresponding to the fields in the schema.

> **Note:** The name of each JSON field is the name of the schema field converted to snake case.

> **Warning:** If an event or entity includes fields not defined in the schema, those fields will not be stored in the data lake, and will not be availble in Synapse.

For example, suppose you have the following field in the schema:

```json
"lastName": {
  "type": "string",
  "maxLength": 100
}
```

It will be translated into a field called `last_name` (notice the underscore) inside the JSON object.

***

## Versioning

What happens when you evolve your schema to a [new version](/docs/event-studio/data-structures/versioning/)?

**Redshift:**

Because the table name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new table:

| Schema                                      | Resulting table              |
| ------------------------------------------- | ---------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-2-0` | `com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing table automatically. For example, if you change the `maxLength` of a `string` field, the limit of the `VARCHAR` column would be updated accordingly.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the table to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new table, with a name like `com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original table. The data for events with that `1-0-2` schema will start going to the original table as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` table and copy it to the original one.
>
> Note that this behavior was introduced in RDB Loader 6.0.0. In older versions, breaking changes will halt the loading process.

> **Info:** Once the loader creates a column for a given schema version as `NULLABLE` or `NOT NULL`, it will never alter the nullability constraint for that column. For example, if a field is nullable in schema version `1-0-0` and not nullable in version `1-0-1`, the column will remain nullable. (In this example, the Enrich application will still validate data according to the schema, accepting `null` values for `1-0-0` and rejecting them for `1-0-1`.)

**BigQuery:**

**Version 2.x:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-2-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing column automatically. For example, if you add a new optional field in the schema, a new optional field will be added to the `RECORD`.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the column to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new column, with a name like `unstruct_event_com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original column. The data for events with that schema will start going to the original column as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` column and copy it to the original one.

**Version 1.x:**

Because the column name for the self-describing event or entity includes the full schema version, each version of a schema gets a new column:

| Schema                                      | Resulting column                                |
| ------------------------------------------- | ----------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1_0_0` |
| `com.example/button_press/jsonschema/1-2-0` | `unstruct_event_com_example_button_press_1_2_0` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2_0_0` |

If you are [modeling your data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/), you can use [this macro](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) to aggregate the data across multiple columns.

> **Info:** While our recommendation is to use major schema versions to indicate breaking changes (e.g. changing a type of a field from a `string` to a `number`), this is not particularly relevant for BigQuery Loader version 1.x. Indeed, each schema version gets its own column, so there is no difference between major and minor versions. That said, we believe sticking to our recommendation is a good idea:
>
> - Breaking changes might affect downstream consumers of the data, even if they don’t affect BigQuery
> - Version 2 of the loader has stricter behavior that matches our loaders for other warehouses and lakes

***

**Snowflake:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-2-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

> **Info:** While our recommendation is to use major schema versions to indicate breaking changes (e.g. changing a type of a field from a `string` to a `number`), this is not particularly relevant for Snowflake. Indeed, the event or entity data is stored in the column as is in the `VARIANT` form, so Snowflake is not “aware” of the schema. That said, we believe sticking to our recommendation is a good idea:
>
> - Breaking changes might affect downstream consumers of the data, even if they don’t affect Snowflake
> - In the future, you might decide to migrate to a different data warehouse where our rules are stricter (e.g. Databricks)
>
> Also, creating a new major version of the schema (and hence a new column) is the only way to indicate a change in semantics, where the data is in the same format but has different meaning (e.g. amounts in dollars vs euros).

**Databricks, Iceberg, Delta:**

Because the column name for the self-describing event or entity includes the major schema version, each major version of a schema gets a new column:

| Schema                                      | Resulting column                            |
| ------------------------------------------- | ------------------------------------------- |
| `com.example/button_press/jsonschema/1-0-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/1-2-0` | `unstruct_event_com_example_button_press_1` |
| `com.example/button_press/jsonschema/2-0-0` | `unstruct_event_com_example_button_press_2` |

When you evolve your schema within the same major version, (non-destructive) changes are applied to the existing column automatically. For example, if you add a new optional field in the schema, a new optional field will be added to the `STRUCT`.

> **Info:** If you make a breaking schema change (e.g. change a type of a field from a `string` to a `number`) without creating a new major schema version, the loader will not be able to modify the column to accommodate the new data.
>
> In this case, _upon receiving the first event with the offending schema_, the loader will instead create a new column, with a name like `unstruct_event_com_example_button_press_1_0_1_recovered_9999999`, where:
>
> - `1-0-1` is the version of the offending schema
> - `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one)
>
> To resolve this situation:
>
> - Create a new schema version (e.g. `1-0-2`) that reverts the offending changes and is again compatible with the original column. The data for events with that schema will start going to the original column as expected.
> - You might also want to manually adapt the data in the `..._recovered_...` column and copy it to the original one.

***

## Types

How do schema types translate to the database types?

### Nullability

**Redshift:**

All non-required schema fields translate to nullable columns.

Required fields translate to `NOT NULL` columns:

```json
{
  "properties": {
    "myRequiredField": {"type": ...}
  },
  "required": [ "myRequiredField" ]
}
```

However, it is possible to define a required field where `null` values are allowed (the Enrich application will still validate that the field is present, even if it’s `null`):

```json
"myRequiredField": {
  "type": ["null", ...]
}
```

OR

```json
"myRequiredField": {
  "enum": ["null", ...]
}
```

In this case, the column will be nullable. It does not matter if `"null"` is in the beginning, middle or end of the list of types or enum values.

> **Info:** See also how [versioning](#versioning) affects this.

**BigQuery:**

All non-required schema fields translate to nullable `RECORD` fields.

Required schema fields translate to required `RECORD` fields:

```json
{
  "properties": {
    "myRequiredField": {"type": ...}
  },
  "required": [ "myRequiredField" ]
}
```

However, it is possible to define a required field where `null` values are allowed (the Enrich application will still validate that the field is present, even if it’s `null`):

```json
"myRequiredField": {
  "type": ["null", ...]
}
```

OR

```json
"myRequiredField": {
  "enum": ["null", ...]
}
```

In this case, the `RECORD` field will be nullable. It does not matter if `"null"` is in the beginning, middle or end of the list of types or enum values.

**Snowflake:**

All fields are nullable (because they are stored inside the `VARIANT` type).

**Databricks, Iceberg, Delta:**

All schema fields, including the required ones, translate to nullable fields inside the `STRUCT`.

***

### Types themselves

**Redshift:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                                                                                                                                                        | Redshift Type                                                                                                                                     |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| ```json
{
  "enum": [E1, E2, ...]
}
```The `enum` can contain more than **one** JavaScript type: `string`, `number\|integer`, `boolean`. For the purposes of this `number` and `integer` are the same.`array`, `object`, `NaN` and other types in the `enum` will be cast as fallback `VARCHAR(4096)`._If content size is longer than 4096 it would be truncated when inserted into the Redshift._ | `VARCHAR(M)``M` is the maximum size of `json.stringify(E*)`                                                                                       |
| ```json
{
  "type": ["boolean", "integer"]
}
```OR```json
{
  "type": ["integer", "boolean"]
}
```                                                                                                                                                                                                                                                                                                 | `VARCHAR(10)`                                                                                                                                     |
| ```json
{
  "type": [T1, T2, ...]
}
```                                                                                                                                                                                                                                                                                                                                                            | `VARCHAR(4096)`_If content size is longer than 4096 it would be truncated when inserted into the Redshift._                                       |
| ```json
{
  "type": "string",
  "format": "date-time"
}
```                                                                                                                                                                                                                                                                                                                                        | `TIMESTAMP`                                                                                                                                       |
| ```json
{
  "type": "string",
  "format": "date"
}
```                                                                                                                                                                                                                                                                                                                                             | `DATE`                                                                                                                                            |
| ```json
{
  "type": "array"
}
```                                                                                                                                                                                                                                                                                                                                                                  | `VARCHAR(65535)`_Content is stringified and quoted.__If content size is longer than 65535 it would be truncated when inserted into the Redshift._ |
| ```json
{
  "type": "integer",
  "maximum": M
}
```* `M` ≤ 32767                                                                                                                                                                                                                                                                                                                                   | `SMALLINT`                                                                                                                                        |
| ```json
{
  "type": "integer",
  "maximum": M
}
```* 32767 < `M` ≤ 2147483647                                                                                                                                                                                                                                                                                                                      | `INT`                                                                                                                                             |
| ```json
{
  "type": "integer",
  "maximum": M
}
```* `M` >2147483647                                                                                                                                                                                                                                                                                                                               | `BIGINT`                                                                                                                                          |
| ```json
{
  "type": "integer",
  "enum": [E1, E2, ...]
}
```* Maximum `E*` ≤ 32767                                                                                                                                                                                                                                                                                                                 | `SMALLINT`                                                                                                                                        |
| ```json
{
  "type": "integer",
  "enum": [E1, E2, ...]
}
```* 32767 < maximum `E*` ≤ 2147483647                                                                                                                                                                                                                                                                                                    | `INT`                                                                                                                                             |
| ```json
{
  "type": "integer",
  "enum": [E1, E2, ...]
}
```* Maximum `E*` > 2147483647                                                                                                                                                                                                                                                                                                            | `BIGINT`                                                                                                                                          |
| ```json
{
  "type": "integer"
}
```                                                                                                                                                                                                                                                                                                                                                                | `BIGINT`                                                                                                                                          |
| ```json
{
  "multipleOf": B
}
```                                                                                                                                                                                                                                                                                                                                                                  | `INT`                                                                                                                                             |
| ```json
{
  "type": "number",
  "multipleOf": B
}
```* Only works for `B`=0.01                                                                                                                                                                                                                                                                                                                     | `DECIMAL(36,2)`                                                                                                                                   |
| ```json
{
  "type": "number"
}
```                                                                                                                                                                                                                                                                                                                                                                 | `DOUBLE`                                                                                                                                          |
| ```json
{
  "type": "boolean"
}
```                                                                                                                                                                                                                                                                                                                                                                | `BOOLEAN`                                                                                                                                         |
| ```json
{
  "type": "string",
  "minLength": M,
  "maxLength": M
}
```* `M` is the same in minLength and maxLength                                                                                                                                                                                                                                                                                 | `CHAR(M)`                                                                                                                                         |
| ```json
{
  "type": "string",
  "format": "uuid"
}
```                                                                                                                                                                                                                                                                                                                                             | `CHAR(36)`                                                                                                                                        |
| ```json
{
  "type": "string",
  "format": "ipv6"
}
```                                                                                                                                                                                                                                                                                                                                             | `VARCHAR(39)`                                                                                                                                     |
| ```json
{
  "type": "string",
  "format": "ipv4"
}
```                                                                                                                                                                                                                                                                                                                                             | `VARCHAR(15)`                                                                                                                                     |
| ```json
{
  "type": "string",
  "format": "email"
}
```                                                                                                                                                                                                                                                                                                                                            | `VARCHAR(255)`                                                                                                                                    |
| ```json
{
  "type": "string",
  "maxLength": M
}
```* `enum` is not defined                                                                                                                                                                                                                                                                                                                        | `VARCHAR(M)`                                                                                                                                      |
| ```json
{
  "enum": ["E1"]
}
```* `E1` is the only element                                                                                                                                                                                                                                                                                                                                         | `CHAR(M)``M` is the size of `json.stringify("E1")`                                                                                                |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                                                                                                                                                     | `VARCHAR(4096)`_Values will be quoted as in JSON.__If content size is longer than 4096 it would be truncated when inserted into the Redshift._    |

**BigQuery:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                       | BigQuery Type                                    |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| ```json
{
  "type": "object",
  "properties": {...}
}
```If the `"properties"` key is missing, the type for the entire object will be `JSON` instead of `RECORD`.Objects can be nullable. Nested fields can also be nullable (same rules as for everything else). | `RECORD`                                         |
| ```json
{
  "type": "array",
  "items": {...}
}
```The type of the repeated value is determined by the `"items"` key of the schema. If the `"items"` key is missing, then the repeated type is `JSON`.                                                            | `REPEATED`                                       |
| ```json
{
  "type": "string",
  "format": "date-time"
}
```                                                                                                                                                                                                       | `TIMESTAMP`                                      |
| ```json
{
  "type": "string",
  "format": "date"
}
```                                                                                                                                                                                                            | `DATE`                                           |
| ```json
{
  "type": "boolean"
}
```                                                                                                                                                                                                                               | `BOOLEAN`                                        |
| ```json
{
  "type": "string"
}
```                                                                                                                                                                                                                                | `STRING`                                         |
| ```json
{
  "type": "integer"
}
```                                                                                                                                                                                                                               | `INT`                                            |
| ```json
{
  "type": "number"
}
```OR```json
{
  "type": [ "integer", "number"]
}
```                                                                                                                                                                              | `FLOAT`                                          |
| ```json
{
  "enum": [I1, I2, ...]
}
```* All `Ix` are integer.                                                                                                                                                                                                    | `INT`                                            |
| ```json
{
  "enum": [I1, N1, ...]
}
```* All `Ix`, `Nx` are integer or number.                                                                                                                                                                                    | `FLOAT`                                          |
| ```json
{
  "enum": [S1, S2, ...]
}
```* All `Sx` are strings                                                                                                                                                                                                     | `STRING`                                         |
| ```json
{
  "enum": [A1, A2, ...]
}
```* `Ax` are a mix of different types                                                                                                                                                                                        | `JSON`_String values will be quoted as in JSON._ |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                    | `JSON`                                           |

**Snowflake:**

All types are `VARIANT`.

**Databricks, Iceberg, Delta:**

> **Note:** The row order in this table is important. Type lookup stops after the first match is found scanning from top to bottom.

| Json Schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Databricks Type                                                |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| ```json
{
  "type": "object",
  "properties": {...}
}
```The `STRUCT` has nested fields, whose types are determined by the `"properties"` key of the schema.If the `"properties"` key is missing, the type for the entire object will be `STRING` instead of `STRUCT`, and data will be JSON-serialized in the string column.Objects can be nullable. Nested fields can also be nullable (same rules as for everything else).                                                                                                                                                                                                                                                                                                                 | `STRUCT`                                                       |
| ```json
{
  "type": "array",
  "items": {...}
}
```The type of values within the `ARRAY` is determined by the `"items"` key of the schema. If the `"items"` key is missing, then the values within the array will have type `STRING`, and array items will be JSON-serialized.Arrays can be nullable. Nested fields can also be nullable (same rules as for everything else).                                                                                                                                                                                                                                                                                                                                                                 | `REPEATED`                                                     |
| ```json
{
  "type": "string",
  "format": "date-time"
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `TIMESTAMP`                                                    |
| ```json
{
  "type": "string",
  "format": "date"
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `DATE`                                                         |
| ```json
{
  "type": "boolean"
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `BOOLEAN`                                                      |
| ```json
{
  "type": "string"
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `STRING`                                                       |
| ```json
{
  "type": "integer",
  "minimum": N,
  "maximum": M
}
```* `M` ≤ 2147483647
* `N` ≥ -2147483648                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `INT`                                                          |
| ```json
{
  "type": "integer",
  "minimum": N,
  "maximum": M
}
```* `M` ≤ 9223372036854775807
* `N` ≥ -9223372036854775808                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `BIGINT`                                                       |
| ```json
{
  "type": "integer",
  "minimum": N,
  "maximum": M
}
```* `M` >1e38-1
* `N` <-1e38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `DECIMAL(38,0)`                                                |
| ```json
{
  "type": "integer",
  "minimum": N,
  "maximum": M
}
```* `M` < 1e38-1
* `N` >-1e38                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{
  "type": "integer"
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `BIGINT`                                                       |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `M` ≤ 2147483647
* `N` ≥ -2147483648
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `INT`                                                          |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `M` ≤ 9223372036854775807
* `N` ≥ -9223372036854775808
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `BIGINT`                                                       |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `M` > 1e38-1
* `N` < -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DECIMAL(38,0)`                                                |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `M` < 1e38-1
* `N` > -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "multipleOf": F
}
```* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `BIGINT`                                                       |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `P` ≤ 38, where `P` is the maximum precision (total number of digits) of `M` and `N`, adjusted for scale (number of digits after the `.`) of `F`.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.**More details**`P` = `MAX`(`M.precision` - `M.scale` + `F.scale`, `N.precision` - `N.scale` + `F.scale`)`S` = `F.scale`For example, `M=10.9999, N=-10, F=0.1` will be `DECIMAL(9,1)`. Calculation as follows:`M` is `DECIMAL(6,4)`, `N` is `DECIMAL(2,0)`, `F` is `DECIMAL(2,1)``P` = `MAX`(6 - 4 + 1, 2 + 1) = 3, rounded up to 9`S` = 1result is `DECIMAL(9,1)` | `DECIMAL(P,S)`_`P` is rounded up to either `9`, `18` or `38`._ |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `P` >38, where is the maximum precision (total number of digits) of `M` and `N`, adjusted for scale (number of digits after the `.`) of `F`.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.**More details**`P` = `MAX`(`M.precision` - `M.scale` + `F.scale`, `N.precision` - `N.scale` + `F.scale`)For example, `M=10.9999, N=-1e50, F=0.1` will be `DOUBLE`. Calculation as follows:`M` is `DECIMAL(6,4)`, `N` is `DECIMAL(2,0)`, `F` is `DECIMAL(2,1)``P` = `MAX`(6 - 4 + 1, 50 + 1) = 51 >38                                                                   | `DOUBLE`                                                       |
| ```json
{
  "type": "number", // OR ["number", "integer"]
  "minimum": N,
  "maximum": M,
  "multipleOf": F
}
```* `M` < 1e38-1
* `N` > -1e38
* `F` is integer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{
  "type": "number" // OR ["number", "integer"]
}
```                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `DOUBLE`                                                       |
| ```json
{
  "enum": [N1, I1, ...]
}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is lesser or equal than 2147483647.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `INT`                                                          |
| ```json
{
  "enum": [N1, I1, ...]
}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is lesser or equal than 9223372036854775807.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `BIGINT`                                                       |
| ```json
{
  "enum": [N1, I1, ...]
}
```* All `Nx` and `Ix` are of types number or integer.
* Maximum scale (number of digits after the `.`) in the enum list is 0.
* Maximum absolute value of the enum list is greater than 9223372036854775807.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `BIGINT`                                                       |
| ```json
{
  "enum": [N1, I1, ...]
}
```* All `Nx` and `Ix` are of types number or integer.
* Absolute maximum value of the enum list and less than 1e38.
* `S` is the maximum scale (number of digits after the `.`) in the enum list and it is greater than 0.
* `P` is precision (total number of digits in `M`). Rounded up to `9`, `18` or `38`.                                                                                                                                                                                                                                                                                                                                                                                          | `DECIMAL(P,S)`_`P` is rounded up to either `9`, `18` or `38`._ |
| ```json
{
  "enum": [S1, S2, ...]
}
```* All `Sx` are string                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `STRING`                                                       |
| ```json
{
  "enum": [A1, A2, ...]
}
```* `Ax` are a mix of different types                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `STRING`_String values will be quoted as in JSON._             |
| If nothing matches above, this is a catch-all.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `STRING`_Values will be quoted as in JSON._                    |

***

---

# Snowflake Streaming Loader configuration reference
> Configure Snowflake Streaming Loader with Snowpipe Streaming, Kinesis, Pub/Sub, and Kafka settings for real-time warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/configuration-reference/

The configuration reference in this page is written for Snowflake Streaming Loader `0.5.1`

### License

The Snowflake Streaming Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, configure the `license.accept` option in the config file:

```json
"license": {
  "accept": true
}
```

### Snowflake configuration

| Parameter                          | Description                                                                                                                                                                                                                                               |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.good.url`                  | Required, e.g. `https://orgname.accountname.snowflakecomputing.com`. URI of the Snowflake account.                                                                                                                                                        |
| `output.good.user`                 | Required. Snowflake user who has necessary privileges                                                                                                                                                                                                     |
| `output.good.privateKey`           | Required. Snowflake private key, used to connect to the account                                                                                                                                                                                           |
| `output.good.privateKeyPassphrase` | Optional. Passphrase for the private key                                                                                                                                                                                                                  |
| `output.good.role`                 | Optional. Snowflake role which the Snowflake user should assume                                                                                                                                                                                           |
| `output.good.database`             | Required. Name of the Snowflake database containing the events table                                                                                                                                                                                      |
| `output.good.schema`               | Required. Name of the Snowflake schema containing the events table                                                                                                                                                                                        |
| `output.good.table`                | Optional. Default value `events`. Name to use for the events table                                                                                                                                                                                        |
| `output.good.channel`              | Optional. Default value `snowplow`. Prefix to use for the snowflake channels. The full name will be suffixed with a number, e.g. `snowplow-1`. If you run multiple loaders in parallel, then each loader must be configured with a unique channel prefix. |

### Streams configuration

**AWS:**

| Parameter                                           | Description                                                                                                                                                                                                                                                                                                                                  |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.streamName`                                  | Required. Name of the Kinesis stream with the enriched events                                                                                                                                                                                                                                                                                |
| `input.appName`                                     | Optional, default `snowplow-snowflake-loader`. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.                                                                                                                                                                                      |
| `input.initialPosition.type`                        | Optional, default `LATEST`. Allowed values are `LATEST`, `TRIM_HORIZON`, `AT_TIMESTAMP`. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table. |
| `input.initialPosition.timestamp`                   | Required if `input.initialPosition` is `AT_TIMESTAMP`. A timestamp in ISO8601 format from where the loader should start consuming events.                                                                                                                                                                                                    |
| `input.retrievalMode`                               | Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.                                                                                                                                                                                                                                                |
| `input.retrievalMode.maxRecords`                    | Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.                                                                                                                                                                                                |
| `input.workerIdentifier`                            | Optional. Defaults to the `HOSTNAME` environment variable. The name of this KCL worker used in the dynamodb lease table.                                                                                                                                                                                                                     |
| `input.leaseDuration`                               | Optional. Default value `10 seconds`. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.                                                                                                                                                                         |
| `input.maxLeasesToStealAtOneTimeFactor`             | Optional. Default value `2.0`. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and `maxLeasesToStealAtOneTimeFactor = 2.0`, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency. |
| `input.checkpointThrottledBackoffPolicy.minBackoff` | Optional. Default value `100 milliseconds`. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                  |
| `input.checkpointThrottledBackoffPolicy.maxBackoff` | Optional. Default value `1 second`. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.                                                                                                                                                                                                          |
| `output.bad.streamName`                             | Required. Name of the Kinesis stream that will receive failed events.                                                                                                                                                                                                                                                                        |
| `output.bad.throttledBackoffPolicy.minBackoff`      | Optional. Default value `100 milliseconds`. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                            |
| `output.bad.throttledBackoffPolicy.maxBackoff`      | Optional. Default value `1 second`. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.                                                                                                                                                                                                    |
| `output.bad.recordLimit`                            | Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                        |
| `output.bad.byteLimit`                              | Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.                                                                                                                                                                                                                      |
| `output.bad.maxRecordSize`                          | Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes                                                                                                                                                                                                                          |

**GCP:**

| Parameter                         | Description                                                                                                                                                                                                                                                                                                                                                           |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`              | Required, e.g. `projects/myproject/subscriptions/snowplow-enriched`. Name of the Pub/Sub subscription with the enriched events                                                                                                                                                                                                                                        |
| `input.parallelPullFactor`        | Optional. Default value 0.5. `parallelPullFactor * cpu count` will determine the number of threads used internally by the Pub/Sub client library for fetching events                                                                                                                                                                                                  |
| `input.durationPerAckExtension`   | Optional. Default value `60 seconds`. Pub/Sub ack deadlines are extended for this duration when needed.                                                                                                                                                                                                                                                               |
| `input.minRemainingAckDeadline`   | Optional. Default value `0.1`. Controls when ack deadlines are re-extended, for a message that is close to exceeding its ack deadline. For example, if `durationPerAckExtension` is `60 seconds` and `minRemainingAckDeadline` is `0.1` then the loader will wait until there is `6 seconds` left of the remining deadline, before re-extending the message deadline. |
| `input.maxMessagesPerPull`        | Optional. Default value 1000. How many Pub/Sub messages to pull from the server in a single request.                                                                                                                                                                                                                                                                  |
| `input.debounceRequests`          | Optional. Default value `100 millis`. Adds an artifical delay between consecutive requests to Pub/Sub for more messages. Under some circumstances, this was found to slightly alleviate a problem in which Pub/Sub might re-deliver the same messages multiple times.                                                                                                 |
| `output.bad.topic`                | Required, e.g. `projects/myproject/topics/snowplow-bad`. Name of the Pub/Sub topic that will receive failed events.                                                                                                                                                                                                                                                   |
| `output.bad.batchSize`            | Optional. Default value 1000. Bad events are sent to Pub/Sub in batches not exceeding this count.                                                                                                                                                                                                                                                                     |
| `output.bad.requestByteThreshold` | Optional. Default value 1000000. Bad events are sent to Pub/Sub in batches with a total size not exceeding this byte threshold                                                                                                                                                                                                                                        |
| `output.bad.maxRecordSize`        | Optional. Default value 9000000. Any single failed event sent to Pub/Sub should not exceed this size in bytes                                                                                                                                                                                                                                                         |

**Azure:**

| Parameter                     | Description                                                                                                                                                                             |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`             | Required. Name of the Kafka topic for the source of enriched events.                                                                                                                    |
| `input.bootstrapServers`      | Required. Hostname and port of Kafka bootstrap servers hosting the source of enriched events.                                                                                           |
| `input.consumerConf.*`        | Optional. A map of key/value pairs for [any standard Kafka consumer configuration option](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html). |
| `output.bad.topicName`        | Required. Name of the Kafka topic that will receive failed events.                                                                                                                      |
| `output.bad.bootstrapServers` | Required. Hostname and port of Kafka bootstrap servers hosting the bad topic                                                                                                            |
| `output.bad.producerConf.*`   | Optional. A map of key/value pairs for [any standard Kafka producer configuration option](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html). |
| `output.bad.maxRecordSize`    | Optional. Default value 1000000. Any single failed event sent to Kafka should not exceed this size in bytes                                                                             |

> **Info:** You can use the `input.consumerConf` and `output.bad.producerConf` options to configure authentication to Azure event hubs using SASL. For example:
>
> ```json
> "input.consumerConf": {
>     "security.protocol": "SASL_SSL"
>     "sasl.mechanism": "PLAIN"
>     "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\$ConnectionString\" password=<PASSWORD>;"
> }
> ```

***

## Other configuration options

| Parameter                             | Description                                                                                                                                                                                                                                                                                                                                       |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batching.maxBytes`                   | Optional. Default value `16000000`. Events are emitted to Snowflake when the batch reaches this size in bytes                                                                                                                                                                                                                                     |
| `batching.maxDelay`                   | Optional. Default value `1 second`. Events are emitted to Snowflake after a maximum of this duration, even if the `maxBytes` size has not been reached                                                                                                                                                                                            |
| `batching.uploadParallelismFactor`    | Optional. Default value 3.5. Controls how many batches can we send simultaneously over the network to Snowflake. E.g. If there are 4 available processors, and `uploadParallelismFactor` is 3.5, then the loader sends up to 14 batches in parallel. Adjusting this value can cause the app to use more or less of the available CPU.             |
| `cpuParallelismFactor`                | Optional. Default value 0.75. Controls how the loaders splits the workload into concurrent batches which can be run in parallel. E.g. If there are 4 available processors, and `cpuParallelismFactor` is 0.75, then the loader processes 3 batches concurrently. Adjusting this value can cause the app to use more or less of the available CPU. |
| `retries.setupErrors.delay`           | Optional. Default value `30 seconds`. Configures exponential backoff on errors related to how Snowflake is set up for this loader. Examples include authentication errors and permissions errors. This class of errors are reported periodically to the monitoring webhook.                                                                       |
| `retries.transientErrors.delay`       | Optional. Default value `1 second`. Configures exponential backoff on errors that are likely to be transient. Examples include server errors and network errors.                                                                                                                                                                                  |
| `retries.transientErrors.attempts`    | Optional. Default value 5. Maximum number of attempts to make before giving up on a transient error.                                                                                                                                                                                                                                              |
| `retries.checkCommittedOffset.delay`  | Optional. Default value `100 millis`. Configures a delay in between flushing events to Snowflake and fetching the latest offset token from Snowflake to check the events are fully ingested.                                                                                                                                                      |
| `skipSchemas`                         | Optional, e.g. `["iglu:com.example/skipped1/jsonschema/1-0-0"]` or with wildcards `["iglu:com.example/skipped2/jsonschema/1--"]`. A list of schemas that won't be loaded to Snowflake. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                  |
| `monitoring.metrics.statsd.hostname`  | Optional. If set, the loader sends statsd metrics over UDP to a server on this host name.                                                                                                                                                                                                                                                         |
| `monitoring.metrics.statsd.port`      | Optional. Default value 8125. If the statsd server is configured, this UDP port is used for sending metrics.                                                                                                                                                                                                                                      |
| `monitoring.metrics.statsd.tags.*`    | Optional. A map of key/value pairs to be sent along with the statsd metric.                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.statsd.period`    | Optional. Default `1 minute`. How often to report metrics to statsd.                                                                                                                                                                                                                                                                              |
| `monitoring.metrics.statsd.prefix`    | Optional. Default `snowplow.snowflake-loader`. Prefix used for the metric name when sending to statsd.                                                                                                                                                                                                                                            |
| `monitoring.webhook.endpoint`         | Optional, e.g. `https://webhook.example.com`. The loader will send to the webhook a payload containing details of any error related to how Snowflake is set up for this loader.                                                                                                                                                                   |
| `monitoring.webhook.tags.*`           | Optional. A map of key/value strings to be included in the payload content sent to the webhook.                                                                                                                                                                                                                                                   |
| `monitoring.webhook.heartbeat.*`      | Optional. Default value `5.minutes`. How often to send a heartbeat event to the webhook when healthy.                                                                                                                                                                                                                                             |
| `monitoring.sentry.dsn`               | Optional. Set to a Sentry URI to report unexpected runtime exceptions.                                                                                                                                                                                                                                                                            |
| `monitoring.sentry.tags.*`            | Optional. A map of key/value strings which are passed as tags when reporting exceptions to Sentry.                                                                                                                                                                                                                                                |
| `telemetry.disable`                   | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                         |
| `telemetry.userProvidedId`            | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                               |
| `output.good.jdbcLoginTimeout`        | Optional. Sets the login timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                   |
| `output.good.jdbcNetworkTimeout`      | Optional. Sets the network timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                 |
| `output.good.jdbcQueryTimeout`        | Optional. Sets the query timeout on the JDBC driver which connects to Snowflake                                                                                                                                                                                                                                                                   |
| `http.client.maxConnectionsPerServer` | Optional. Default value 4. Configures the internal HTTP client used for alerts and telemetry. The maximum number of open HTTP requests to any single server at any one time.                                                                                                                                                                      |

---

# Snowflake Streaming Loader
> Load Snowplow events to Snowflake with sub-minute latency from Kinesis, Pub/Sub, or Kafka using Snowpipe Streaming.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/

The Snowflake Streaming Loader is an application that loads Snowplow events to Snowflake.

> **Tip:** Both [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) and [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) can load data into Snowflake.
>
> Snowflake Streaming Loader is newer and has two advantages:
>
> - Much lower latency — you can get data in Snowflake in seconds, as opposed to minutes with RDB Loader
> - Much lower cost — unlike with RDB Loader, there is no need for EMR and extensive Snowflake compute to load batch files
>
> We recommend the Streaming Loader over the RDB Loader. If you already use RDB Loader, see the [migration guide](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/) for more information.

**AWS:**

On AWS, the Snowflake Streaming Loader continually pulls events from Kinesis and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any AWS VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-kinesis:0.5.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \
snowplow/snowflake-loader-kinesis:0.5.1 \
--config=/myconfig/loader.hocon
```

**GCP:**

On GCP, the Snowflake Streaming Loader continually pulls events from Pub/Sub and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any GCP VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-pubsub:0.5.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \
snowplow/snowflake-loader-pubsub:0.5.1 \
--config=/myconfig/loader.hocon
```

**Azure:**

On Azure, the Snowflake Streaming Loader continually pulls events from Kafka and writes to Snowflake using the [Snowpipe Streaming API](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview).

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Snowflake Streaming Loader</b>"}}
subgraph snowflake [Snowflake]
  table[("<b>Events table</b>")]
end
stream-->loader-->|Snowpipe Streaming API|snowflake
```

The Snowflake Streaming Loader is published as a Docker image which you can run on any Azure VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/snowflake-loader-kafka:0.5.1
```

To run the loader, mount your config file into the docker image, and then provide the file path on the command line. We recommend setting the `SNOWFLAKE_PRIVATE_KEY` environment variable so that you can refer to it in the config file.

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
--env SNOWFLAKE_PRIVATE_KEY="${SNOWFLAKE_PRIVATE_KEY}" \
snowplow/snowflake-loader-kafka:0.5.1 \
--config=/myconfig/loader.hocon
```

***

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.kinesis.minimal.hocon)

**GCP:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.pubsub.minimal.hocon)

**Azure:**

```json
loading...
```

[View on GitHub](https://github.com/snowplow-incubator/snowplow-snowflake-loader/blob/main/config/config.azure.minimal.hocon)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/configuration-reference/) for all possible configuration parameters.

---

# Migrating to Snowflake Streaming Loader from RDB Loader
> Migrate from RDB Loader to Snowflake Streaming Loader for lower latency and cost with same table or fresh table strategies.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/

This guide is aimed at Snowplow users who load events into Snowflake via the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/).

We recommend migrating to use the [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) because it has a much lower latency and is cheaper to run.

There are two migration strategies you might take:

1. [Load into the same table as before](#load-into-the-same-table-as-before). This way you have a single events table, containing old events loaded with RDB Loader and new events loaded by the Streaming Loader.
2. [Load into a fresh new table](#load-into-a-fresh-new-table-with-the-streaming-loader) with the Streaming Loader. This is a more cautious approach, but you will need to point any data models or downstream applications, dashboards, etc, to the new table.

## Load into the same table as before

The Streaming Loader is fully compatible with the table created and managed by the recent versions of RDB Loader. In particular, these aspects are exactly the same as before:

- There are 129 columns for the atomic fields, common to all Snowplow events
- [Self-describing events](/docs/fundamentals/events/#self-describing-events) are loaded into columns named like `unstruct_event_com_example_button_press_1`
- [Entities](/docs/fundamentals/entities/) are loaded into columns named like `contexts_com_example_user_1`
- For both self-describing events and entities, a new column is created for each major version of the Iglu schema

> **Tip:** [This page](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) explains how Snowplow data maps to the warehouse in more detail.

You will notice some subtle differences:

#### No loader-side deduplication

RDB Loader performs [within-batch and cross-batch deduplication](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/) during loading. The Streaming Loader does not deduplicate events, so you may see more duplicates than with the RDB Loader.

Snowplow's [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) handle deduplication automatically. If you write custom queries, see [dealing with duplicates](/docs/destinations/warehouses-lakes/querying-data/#dealing-with-duplicates).

#### New `_schema_version` property in entities

Previously, when loading entities into the table, RDB Loader would drop any information about exactly which version of the schema had been used to validate them.

The Streaming Loader adds an extra property called `_schema_version`, so the versioning information is not lost in the warehouse.

For example, if a tracker sends an entity like this:

```json
{
  "schema": "iglu:com.example/my_schema/jsonschema/1-0-3",
  "data": {
    "a": 1
  }
}
```

Then the value loaded into the `contexts_com_example_my_schema_1` column is this:

```json
{
  "a": 1,
  "_schema_version": "1-0-3"
}
```

#### Null values omitted from entities

For self-describing events and entities, the Streaming Loader omits values that are explicitly set to `null`. This differs from RDB loader, which stores them.

For example, if a tracker sends an entity like this:

```json
{
  "schema": "iglu:com.example/my_schema/jsonschema/1-0-0",
  "data": {
    "a": 1,
    "b": null,
    "c": null
  }
}
```

Then the value loaded into the `contexts_com_example_my_schema_1` column is this (note the `b` and `c` fields are missing):

```json
{
  "a": 1
}
```

We made this change as a performance optimization for querying data. The Snowflake docs [explain](https://docs.snowflake.com/en/user-guide/semistructured-considerations) that JSON `null` values affect how Snowflake extracts nested properties. Snowflake automatically builds indexes on the nested properties of VARIANT columns, but only if those properties do not contain explicit `null` values.

#### The `load_tstamp` field is required before migrating

> **Note:** This only affects users migrating from RDB Loader older than version 4.0.0.

The Snowflake events table must have a column named `load_tstamp` of type `TIMESTAMP`. If you have ever used a version of RDB Loader newer than 4.0.0, then it will have already added this column for you. But if you are migrating from an older version of RDB Loader then you will need to add the column manually:

```sql
ALTER TABLE events ADD COLUMN load_tstamp TIMESTAMP
```

## Load into a fresh new table with the Streaming Loader

The Snowflake Streaming Loader will automatically create the events table when you run it for the first time. If you are familiar with the RDB Loader's table, then all of the points in the previous section are still relevant to you. You will also notice a few other differences:

#### No maximum lengths on VARCHAR columns

The old table created by RDB Loader had maximum lengths on some of the columns, e.g. `app_id VARCHAR(255)`. The new Streaming Loader creates columns without max lengths, e.g. `app_id VARCHAR`.

#### VARCHAR instead of CHAR

When RDB Loader created the events table, it used a mixture of VARCHAR and CHAR column types for the various different string fields. For the sake of simplicity, the Streaming Loader uses VARCHAR column types only. In Snowflake, there is no meaningful difference between the two column types.

---

# Postgres Loader for testing and development
> Load Snowplow enriched events into PostgreSQL database from Kinesis or Pub/Sub for development and testing environments.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/

With Snowplow Postgres Loader you can load enriched data or [failed events](/docs/fundamentals/failed-events/) into PostgreSQL database.

> **Danger:** The Postgres loader is not recommended for production use, especially with large data volumes. We recommend using a fully-fledged data warehouse like Databricks, Snowflake, BigQuery or Redshift, together with a [respective loader](/docs/destinations/warehouses-lakes/).

> **Tip:** For more information on how events are stored in Postgres, check the [mapping between Snowplow schemas and the corresponding Postgres column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=postgres).

## Available on Terraform Registry

A Terraform module which deploys the Snowplow Postgres Loader on AWS EC2 for use with Kinesis. For installing in other environments, please see the other installation options below.

## Getting a Docker image

Snowplow Postgres Loader is [published on DockerHub](https://hub.docker.com/r/snowplow/snowplow-postgres-loader):

```bash
docker pull snowplow/snowplow-postgres-loader:0.3.3
```

It accepts very typical configuration for Snowplow Loader:

```bash
docker run --rm \
-v $PWD/config:/snowplow/config \
snowplow/snowplow-postgres-loader:0.3.3 \
--resolver /snowplow/config/resolver.json \
--config /snowplow/config/config.hocon
```

## Iglu

Where `resolver.json` is a typical [Iglu Client](/docs/api-reference/iglu/iglu-resolver/) configuration.

**Please pay attention that schemas for all self-describing JSONs flowing through Postgres Loader must be hosted on Iglu Server 0.6.0 or above.** Iglu Central is static registry and if you use Snowplow-authored schemas - you need to upload all schemas from there as well.

## Configuration

The configuration file is in HOCON format, and it specifies connection details for the target database and the input stream of events.

```json
{
  "input": {
    "type": "Kinesis"
    "streamName": "enriched-events"
    "region": "eu-central-1"
  }
  "output" : {
    "good": {
      "type": "Postgres"
      "host": "localhost"
      "database": "snowplow"
      "username": "postgres"
      "password": ${POSTGRES_PASSWORD}
      "schema": "atomic"
    }
  }
}
```

The `input` section can alternatively specify a GCP PubSub subscription, instead of a kinesis stream like in the example above.

```json
  "input": {
    "type": "PubSub"
    "projectId": "my-project"
    "subscriptionId": "my-subscription"
  }
```

See [the configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/postgres-loader-configuration-reference/) for a complete description of all parameters.

## Other

Loader creates `events` table on the start and every other table when it first encounters its corresponding schema.

You should ensure that the database and schema specified in the configuration exist before starting the loader.

---

# Postgres Loader configuration reference
> Configure Postgres Loader with Kinesis, Pub/Sub, or local input sources and database connection settings for event loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/postgres-loader-configuration-reference/

This is a complete list of the options that can be configured in the postgres loader's HOCON config file. The [example configs in github](https://github.com/snowplow-incubator/snowplow-postgres-loader/tree/master/config) show how to prepare an input file.

|                                  |                                                                                                                                                                                                                                                                                                                   |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.type`                     | Required. Can be "Kinesis", "PubSub" or "Local". Configures where input events will be read from.                                                                                                                                                                                                                 |
| `input.streamName`               | Required when `input.type` is Kinesis. Name of the Kinesis stream to read from.                                                                                                                                                                                                                                   |
| `input.region`                   | Required when `input.type` is Kinesis. AWS region in which the Kinesis stream resides.                                                                                                                                                                                                                            |
| `input.initialPosition`          | Optional. Used when `input.type` is Kinesis. Use "TRIM\_HORIZON" (the default) to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use "LATEST" to start streaming just after the most recent record in the shard.                                     |
| `input.retrievalMode.type`       | Optional. When `input.type` is Kinesis, this sets the polling mode for retrieving records. Can be "FanOut" (the default) or "Polling".                                                                                                                                                                            |
| `input.retrievalMode.maxRecords` | Optional. Used when `input.retrievalMode.type` is "Polling". Configures how many records are fetched in each poll of the kinesis stream. Default 10000.                                                                                                                                                           |
| `input.projectId`                | Required when `input.type` is PubSub. The name of your GCP project.                                                                                                                                                                                                                                               |
| `input.subscriptionId`           | Required when `input.type` is PubSub. Id of the PubSub subscription to read events from                                                                                                                                                                                                                           |
| `input.path`                     | Required when `input.type` is Local. Path for event source. It can be directory or file. If it is directory, all the files under given directory will be read recursively. Also, given path can be both absolute path or relative path w\.r.t. executable.                                                        |
| `output.good.host`               | Required. Hostname of the postgres database.                                                                                                                                                                                                                                                                      |
| `output.good.port`               | Optional. Port number of the postgres database. Default 5432.                                                                                                                                                                                                                                                     |
| `output.good.database`           | Required. Name of the postgres database.                                                                                                                                                                                                                                                                          |
| `output.good.username`           | Required. Postgres role name to use when connecting to the database                                                                                                                                                                                                                                               |
| `output.good.password`           | Required. Password for the postgres user.                                                                                                                                                                                                                                                                         |
| `output.good.schema`             | Required. The Postgres schema in which to create tables and write events.                                                                                                                                                                                                                                         |
| `output.good.sslMode`            | Optional. Configures how the client and server agree on ssl protection. Default "REQUIRE"                                                                                                                                                                                                                         |
| `output.bad.type`                | Optional. Can be "Kinesis", "PubSub", "Local" or "Noop". Configures where failed events will be sent. Default is "Noop" which means failed events will be discarded                                                                                                                                               |
| `output.bad.streamName`          | Required when `bad.type` is Kinesis. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                      |
| `output.bad.region`              | Required when `bad.type` is Kinesis. AWS region in which the Kinesis stream resides.                                                                                                                                                                                                                              |
| `output.bad.projectId`           | Required when `bad.type` is PubSub. The name of your GCP project.                                                                                                                                                                                                                                                 |
| `output.bad.topicId`             | Required when `bad.type` is PubSub. Id of the PubSub topic to write failed events to                                                                                                                                                                                                                              |
| `output.bad.path`                | Required when `bad.type` is Local. Path of the file to write failed events                                                                                                                                                                                                                                        |
| `purpose`                        | Optional. Set this to "ENRICHED\_EVENTS" (the default) when reading the stream of enriched events in tsv format. Set this to "JSON" when reading a stream of self-describing json, e.g. snowplow [bad rows](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows). |
| `monitoring.metrics.cloudWatch`  | Optional boolean, with default true. For kinesis input, this is used to disable sending metrics to cloudwatch.                                                                                                                                                                                                    |

#### Advanced options

We believe these advanced options are set to sensible defaults, and hopefully you won't need to ever change them.

|                                          |                                                                                                                                                                                                           |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backoffPolicy.minBackoff`               | If producer (PubSub or Kinesis) fails to send item, it will retry to send it again. This field configures backoff time for first retry. Every retry will double the backoff time of previous one.         |
| `backoffPolicy.maxBackoff`               | Maximum backoff time for retry. After this value is reached, backoff time will no more increase.                                                                                                          |
| `input.checkpointSettings.maxBatchSize`  | Used when `input.type` is Kinesis. Determines the max number of records to aggregate before checkpointing the records. Default is 1000.                                                                   |
| `input.checkpointSettings.maxBatchWait`  | Used when `input.type` is Kinesis. Determines the max amount of time to wait before checkpointing the records. Default is 10 seconds.                                                                     |
| `input.checkpointSettings.maxConcurrent` | Used when `input.type` is PubSub. The max number of concurrent evaluation for checkpointer.                                                                                                               |
| `output.good.maxConnections`             | Maximum number of connections database pool is allowed to reach. Default 10                                                                                                                               |
| `output.good.threadPoolSize`             | Size of the thread pool for blocking database operations. Default is value of "maxConnections"                                                                                                            |
| `output.bad.delayThreshold`              | Set the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. Default 200 milliseconds |
| `output.bad.maxBatchSize`                | A batch of messages will be emitted when the number of events in batch reaches the given size. Default 500                                                                                                |
| `output.bad.maxBatchBytes`               | A batch of messages will be emitted when the size of the batch reaches the given size. Default 5 MB                                                                                                       |

---

# RDB Loader for Redshift and Databricks
> Load Snowplow events into Redshift, Databricks, or Snowflake with transformation and deduplication using Spark or stream transformers.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/

We use the name RDB Loader (from "relational database") for a set of applications that can be used to load Snowplow events into a data warehouse. Use these tools if you want to load into **Redshift** (including Redshift serverless), **Databricks**, or **Snowflake** (the latter not recommended). For other destinations, see [here](/docs/api-reference/loaders-storage-targets/).

**Redshift:**

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Redshift, stores them in an S3 bucket and instructs Redshift to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Redshift"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Redshift
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Redshift.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to TSV
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Redshift: Send SQL commands for loading
  Note over Redshift: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Redshift, stores them in an S3 bucket and instructs Redshift to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Redshift"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Redshift
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Redshift.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to TSV
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Redshift: Send SQL commands for loading
  Note over Redshift: Load the data from the S3 bucket using “COPY FROM”
end
```

***

**Databricks:**

> **Note:** The cloud selection below is for your _pipeline_. We don’t have restrictions on where Databricks itself is deployed.

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an S3 bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an S3 bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the S3 bucket using “COPY FROM”
end
```

**GCP:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Databricks, stores them in an GCS bucket and instructs Databricks to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(GCS bucket)")]
subgraph "Databricks"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Databricks
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Databricks.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to Parquet
  
  Note over Transformer: Write data to the GCS bucket
  Transformer->>Loader: Notify the loader (via Pub/Sub)
  Loader->>Databricks: Send SQL commands for loading
  Note over Databricks: Load the data from the GCS bucket using “COPY FROM”
end
```

***

**Snowflake:**

> **Tip:** Both [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) and [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) can load data into Snowflake.
>
> Snowflake Streaming Loader is newer and has two advantages:
>
> - Much lower latency — you can get data in Snowflake in seconds, as opposed to minutes with RDB Loader
> - Much lower cost — unlike with RDB Loader, there is no need for EMR and extensive Snowflake compute to load batch files
>
> We recommend the Streaming Loader over the RDB Loader. If you already use RDB Loader, see the [migration guide](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/migrating/) for more information.

> **Note:** The cloud selection below is for your _pipeline_. We don’t have restrictions on where Snowflake itself is deployed.

**AWS (Batching, recommended):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an S3 bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

  s3loader{{"<b>S3 Loader</b>"}}
  prebucket[("<b>Enriched Events</b>
(S3 bucket)")]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->s3loader-->prebucket-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the S3 bucket using “COPY FROM”
end
```

**AWS (Micro-batching):**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an S3 bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(S3 bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the S3 bucket
  Transformer->>Loader: Notify the loader (via SQS)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the S3 bucket using “COPY FROM”
end
```

**GCP:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an GCS bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(GCS bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the GCS bucket
  Transformer->>Loader: Notify the loader (via Pub/Sub)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the GCS bucket using “COPY FROM”
end
```

**Azure:**

At the high level, RDB Loader reads batches of enriched Snowplow events, converts them to the format supported by Snowflake, stores them in an Azure Blob Storage bucket and instructs Snowflake to load them.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]

loader{{"<b>RDB Loader</b>
(Transformer and Loader apps)"}}
bucket[("<b>Transformed Events</b>
(Azure Blob Storage bucket)")]
subgraph "Snowflake"
  table[("<b>Events table</b>")]
  
end
stream-->loader-->bucket--->Snowflake
```

RDB loader consists of two applications: Transformer and Loader. The following diagram illustrates the interaction between them and Snowflake.

```mermaid
sequenceDiagram
loop
  Note over Transformer: Read a batch of events
  Note over Transformer: Transform events to JSON
  
  Note over Transformer: Write data to the Azure Blob Storage bucket
  Transformer->>Loader: Notify the loader (via Kafka)
  Loader->>Snowflake: Send SQL commands for loading
  Note over Snowflake: Load the data from the Azure Blob Storage bucket using “COPY FROM”
end
```

***

***

> **Tip:** For more information on how events are stored in the warehouse, check the [mapping between Snowplow schemas and the corresponding warehouse column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

To run RDB Loader, you will need to run one instance of the Transformer and one instance of the Loader.

## How to pick a transformer

The transformer app currently comes in two flavours: a Spark job that processes data in batches, and a long-running streaming app.

The process of transforming the data is not dependent on the storage target. Which one is best for your use case depends on three factors:

- cloud provider you want to use (AWS, GCP or Azure)
- your expected data volume
- how much importance you place on deduplicating the data before loading it into the data warehouse.

### Based on cloud provider

If you want to run the transformer on AWS, you can use Spark transformer (`snowplow-transformer-batch`) or Transformer Kinesis (`snowplow-transformer-kinesis`).

If you want to run the transformer on GCP, you can use Transformer Pubsub (`snowplow-transformer-pubsub`).

If you want to run the transformer on Azure, you can use Transformer Kafka (`snowplow-transformer-kafka`).

### Based on expected data volume

The Spark transformer (`snowplow-transformer-batch`) is the best choice for big volumes, as the work can be split across multiple workers. However, the need to run it on EMR creates some overhead that is not justified for low-volume pipelines.

The stream transformer (`snowplow-transformer-kinesis`, `snowplow-transformer-pubsub` and `snowplow-transformer-kafka`) is a much leaner alternative and suggested for use with low volumes that can be comfortably processed on a single node. However, multiple stream transformers can be run parallel therefore it is possible to process big data volume with stream transformer too.

To make the best choice, consider:

- What is the underlying infrastructure? For example, a single-node stream transformer will perform differently based on the resources it is given by the machine it runs on.
- What is the frequency for processing data? For example, even in a low-volume pipeline, if you only run the transform job once a day, the accumulated data might be enough to justify the use of Spark.

### Based on the importance of deduplication

The transformer is also in charge of [deduplicating](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/) the data. Currently, only the Spark transformer can do that.

If duplicates are not a concern, or if you are happy to deal with them after the data has been loaded in the warehouse, then pick a transformer based on your expected volume (see above). Otherwise, use the Spark transformer.

## How to pick a loader based on the destination

There are different loader applications depending on the storage target. Currently, RDB Loader supports Redshift (AWS only), Snowflake and Databricks.

For loading into **Redshift** (including Redshift serverless), use the `snowplow-rdb-loader-redshift` artifact.

For loading into **Snowflake**, use the `snowplow-rdb-loader-snowflake` artifact.

For loading into **Databricks**, use the `snowplow-rdb-loader-databricks` artifact.

## How `transformer` and `loader` interface with other Snowplow components and each other

The applications communicate through messages.

The transformer consumes enriched tsv-formatted Snowplow events from S3 (AWS) or stream (AWS, GCP and Azure). It writes its output to blob storage (S3, GCS or Azure Blob Storage). Once it's finished processing a batch of data, it issues a message with details about the run.

The loader consumes a stream of these messages and uses them to determine what data needs to be loaded. It issues the necessary SQL commands to the storage target.

---

# Load into Databricks using the RDB Loader
> Load wide row Parquet data into Databricks with automatic schema creation and Delta Lake optimization.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/

To set up the Databricks loader, the following resources need to be created:

- [Databricks cluster](https://docs.databricks.com/clusters/create-cluster.html)
- [Databricks access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html)

The `events` table and the database schema will be created automatically by the loader. You can [configure](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/) the name of the database schema with the `storage.schema` config field. The table name (`events`) can’t be changed.

Keep in mind that the Databricks Loader database user needs to have permissions to create schemas on the given database to be able to perform this operation. Check [this page](https://docs.databricks.com/sql/language-manual/security-grant.html) for more information about granting privileges in Databricks. You can also create the schema manually if you prefer.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-databricks:6.3.0`.

## Configuring `rdb-loader-databricks`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Databricks loader can be found [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) version 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Databricks loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-databricks:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Databricks Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Loading transformed data into warehouses
> Load transformed Snowplow data into Redshift, Snowflake, or Databricks with automated table management and SQL COPY operations.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/

_For a high-level overview of the RDB Loader architecture, of which the loader is a part, see [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)._

The loader applications are specialised to a specific storage target. Each one performs 3 key tasks:

- Consume messages from SQS / SNS / Pubsub / Kafka to discover information about transformed data: where it is stored and what it looks like.
- Use the information from the message to determine if any changes to the target table(s) are required, eg to add a column for a new event field. If required, submit the appropriate SQL statement for execution by the storage target.
- Prepare and submit for execution the appropriate SQL `COPY` statement.

For loading into **Redshift**, use the [Redshift loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/). This loads [shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) into multiple Redshift tables.

For loading into **Snowflake**, use the [Snowflake loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/). This loads [wide row JSON format data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) into a single Snowflake table.

For loading into **Databricks**, use the [Databricks loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/). This loads [wide row Parquet format data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) into a single Databricks table.

> **Note:** AWS is fully supported for both Snowflake and Databricks. GCP is supported for Snowflake (since 5.0.0). Azure is supported for Snowflake (since 5.7.0).

---

# Monitoring RDB Loader
> Monitor RDB Loader with folder checks, health checks, StatsD metrics, Sentry alerts, and Snowplow tracking for warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/monitoring/

The loader app has several types of monitoring built in to help the pipeline operator: folder monitoring, warehouse health checks, StatsD metrics, Sentry alerts, and Snowplow tracking.

For all monitoring configuration options, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

## Webhook alerts

The loader can send `POST` requests via HTTP webhook to a configurable URL whenever there is an issue which needs investigation by the pipeline operator. The webhook payload conforms to the [`alert`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.monitoring.batch/alert/jsonschema/1-0-0) schema on Iglu Central.

You can configure where the webhook is sent by setting the `monitoring.webhook` section in the `config.hocon` file.

The webhook monitoring can be used for folder monitoring and warehouse health checks.

### Folder monitoring

A webhook alert is sent whenever the loader identifies inconsistencies between the transformed output in S3 and the data in the warehouse. The algorithm is as follows:

- Check if all folders on S3 have a `shredding_complete.json` file (the legacy name is kept for backwards compatibility, but this applies to wide row format data as well). A missing file suggests the transformer failed to complete writing the transformed data, and so manual intervention is required to remove the folder from S3 and rerun.
- Check if all folders on S3 created within a specific time range are listed in the warehouse manifest table. This table is maintained by the loader and contains information about loads. If a folder is missing from the manifest table, it suggests the loader has previously tried and failed to load it. Manual intervention is required to resend the `shredding_complete.json` message via SQS / SNS to trigger reloading of the folder.

Folder monitoring is configured by setting the `monitoring.folders` section in the `config.hocon` file.

### Warehouse health check

The loader can send an alert if the warehouse does not respond to a periodic `SELECT 1` statement. For each failed health check, a `POST` request is sent via the webhook.

The health check is configured by setting the `monitoring.healthCheck` section in the `config.hocon` file.

## StatsD and stdout

[StatsD](https://github.com/statsd/statsd) is a daemon that aggregates and summarizes application metrics. It receives metrics sent by the application over UDP, and then periodically flushes the aggregated metrics to a [pluggable storage backend](https://github.com/statsd/statsd/blob/master/docs/backend.md).

The loader can emit metrics to a StatsD daemon describing every batch it processes. Here is a string representation of the metrics it sends:

```text
snowplow.rdbloader.count_good:42|c|#tag1:value1
snowplow.rdbloader.count_bad:2|c|#tag1:value1
snowplow.rdbloader.latency_collector_to_load_min:123.4|g|#tag1:value1
snowplow.rdbloader.latency_collector_to_load_max:234.5|g|#tag1:value1
snowplow.rdbloader.latency_transformer_start_to_load:66.6|g|#tag1:value1
snowplow.rdbloader.latency_transformer_end_to_load:44.4|g|#tag1:value1
```

These are the meanings of the individual metrics:

- `count_good`: the total number of good events in the batch that was loaded
- `count_bad`: the total number of bad events in the batch that was loaded (available since version 5.4.0)
- `latency_collector_to_load_min`: for the most recent event in the batch, this is the time difference between reaching the collector and getting loaded to the warehouse
- `latency_collector_to_load_min`: for the oldest event in the batch, this is the time difference between reaching the collector and getting loaded to the warehouse
- `latency_transformer_start_to_load`: time difference between the transformer starting on this batch and the loader completing loading to the warehouse
- `latency_transformer_end_to_load`: time difference between the transformer completing this batch and the loader completing loading it into the warehouse.

StatsD monitoring is configured by setting the `monitoring.metrics.statsd` section in the `config.hocon` file.

You can expose these metrics in `stdout` for easier debugging by setting the `monitoring.metrics.stdout` section in the `config.hocon` file.

## Sentry

[Sentry](https://docs.sentry.io/) is a popular error monitoring service, which helps developers diagnose and fix problems in an application. The loader and transformer can both send an error report to sentry whenever something unexpected happens. The reasons for the error can then be explored in the Sentry server’s UI.

Common reasons might be lost connection to the database, or an HTTP error fetching a schema from an Iglu server.

Sentry monitoring is configured by setting the `monitoring.sentry.dsn` section in the `config.hocon` file.

## Snowplow tracking

The loader can emit a Snowplow event to a collector when the application crashes with an unexpected error. The event conforms to the [`load_failed`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.monitoring.batch/load_failed/jsonschema/1-0-0) schema on Iglu Central.

Snowplow tracking is configured by setting the `monitoring.snowplow` section in the `config.hocon` file.

---

# Load into Redshift using the RDB Loader
> Load shredded Snowplow events into Amazon Redshift with automatic schema creation and table management.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/

The `events` table and the database schema will be created automatically by the loader. You can [configure](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/) the name of the database schema with the `storage.schema` config field. The table name (`events`) can’t be changed.

Keep in mind that the Redshift Loader database user needs to have permissions to create schemas on the given database to be able to perform this operation. Check [this page](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) for more information about granting privileges in Redshift. You can also create the schema manually if you prefer.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-redshift:6.3.0`.

## Configuring `rdb-loader-redshift`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Redshift loader can be found [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Redshift loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-redshift:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Redshift Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Load into Snowflake using the RDB Loader
> Load wide row JSON data into Snowflake on AWS, GCP, or Azure with TempCreds or NoCreds authentication methods.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/

It is possible to run Snowflake Loader on AWS, GCP and Azure.

### Setting up Snowflake

You can use the steps outlined in our [quick start guide](/docs/get-started/self-hosted/quick-start/?warehouse=snowflake#prepare-the-destination) to create most of the necessary Snowflake resources.

There are two different authentication methods with Snowflake Loader:

- With the `TempCreds` method, there are no additional Snowflake resources needed.
- With the `NoCreds` method, the Loader needs a Snowflake stage.

This choice is controlled by the `loadAuthMethod` [configuration setting](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/#snowflake-loader-storage-section).

> **Note:** For GCP pipelines, only the `NoCreds` method is available.

**Using the NoCreds method**

First, create a Snowflake stage. For that, you will need a Snowflake database, Snowflake schema, Snowflake storage integration, Snowflake file format, and the path to the transformed events bucket (in S3, GCS or Azure Blob Storage).

You can follow [this tutorial](https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html) to create the storage integration.

Assuming you created the other required resources for it, you can create the Snowflake stage by following [this document](https://docs.snowflake.com/en/sql-reference/sql/create-stage.html).

Finally, use the `transformedStage` [configuration setting](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/#snowflake-loader-storage-section) to point the loader to your stage.

### Running the loader

There are dedicated terraform modules for deploying Snowflake Loader on [AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-loader-ec2/aws/latest) and [Azure](https://github.com/snowplow-devops/terraform-azurerm-snowflake-loader-vmss). You can see how they are used in our full pipeline deployment examples [here](/docs/get-started/self-hosted/quick-start/).

We don't have a terraform module for deploying Snowflake Loader on GCP yet. Therefore, it needs to be deployed manually at the moment.

### Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/rdb-loader-snowflake:6.3.0`.

### Configuring `rdb-loader-snowflake`

The loader takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

| Minimal Configuration                                                                                                                                  | Extended Configuration                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [aws/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.minimal.hocon)     | [aws/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.reference.hocon)     |
| [gcp/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.minimal.hocon)     | [gcp/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.reference.hocon)     |
| [azure/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.minimal.hocon) | [azure/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.reference.hocon) |

For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by RDB Loader **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

### Running the Snowflake loader

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/rdb-loader-snowflake:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Snowflake Loader (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# RDB Loader configuration reference
> Configure RDB Loader for Redshift, Snowflake, and Databricks with storage, messaging, scheduling, and monitoring settings.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/

The configuration reference in this page is written for RDB Loader 5.0.0 or higher.

The configuration reference pages for previous versions can be found [here](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/rdb-loader-configuration-reference/rdb-loader-previous-versions/).

| Minimal Configuration                                                                                                                                  | Extended Configuration                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [aws/redshift.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.minimal.hocon)       | [aws/redshift.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/redshift.config.reference.hocon)       |
| [aws/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.minimal.hocon)     | [aws/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/snowflake.config.reference.hocon)     |
| [aws/databricks.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.minimal.hocon)   | [aws/databricks.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/aws/databricks.config.reference.hocon)   |
| [gcp/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.minimal.hocon)     | [gcp/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/gcp/snowflake.config.reference.hocon)     |
| [azure/snowflake.config.minimal.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.minimal.hocon) | [azure/snowflake.config.reference.hocon](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/loader/azure/snowflake.config.reference.hocon) |

All applications use a common module for core functionality, so only the `storage` sections are different in their config.

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

## Password rotation

To rotate the password for your RDB Loader, you'll need to contact [Snowplow Support](https://support.snowplow.io/), and share the intended password using the secure Credential sharing form in Console. To avoid disruption to your pipeline, we'll coordinate a time with you to update the credentials.

## Redshift Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`                                        | Optional. The only valid value is the default: `redshift`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `host`                                        | Required. Host name of Redshift cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `port`                                        | Required. Port of Redshift cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `database`                                    | Required. Redshift database which the data will be loaded to.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `roleArn`                                     | Required if 'NoCreds' is chosen as load auth method.. AWS Role ARN allowing Redshift to load data from S3.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `schema`                                      | Required. Redshift schema name, eg “atomic”.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `username`                                    | Required. DB user with permissions to load data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `jdbcAuth.*` (since 6.3.0)                    | Required. JDBC authentication configuration. Supports two modes: `password` and `iam`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `jdbcAuth.type` (since 6.3.0)                 | Required. The authentication type. Possible values: `password` and `iam`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `jdbcAuth.password` (since 6.3.0)             | Required if `jdbcAuth.type` is `password`. Password of the DB user.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `jdbcAuth.roleArn` (since 6.3.0)              | Required if `jdbcAuth.type` is `iam`. IAM role ARN with permissions to call `GetClusterCredentials`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `jdbcAuth.roleExternalId` (since 6.3.0)       | Required if `jdbcAuth.type` is `iam`. External ID for assuming the IAM role, used to restrict role assumption to trusted principals.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `jdbcAuth.redshiftRegion` (since 6.3.0)       | Required if `jdbcAuth.type` is `iam`. AWS region where the Redshift cluster is located.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `jdbcAuth.clusterId` (since 6.3.0)            | Required if `jdbcAuth.type` is `iam`. Redshift cluster identifier.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `jdbcAuth.roleSessionName` (since 6.3.0)      | Optional. Session name for the assumed IAM role.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `jdbcAuth.credentialsTtl` (since 6.3.0)       | Optional. TTL for temporary database credentials when using IAM authentication. Must be between 15 minutes and one hour.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `maxError`                                    | Optional. Configures the [Redshift MAXERROR load option](https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-load.html#copy-maxerror). The default is 10.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `loadAuthMethod.*` (since 5.2.0)              | Optional, default method is `NoCreds`. Specifies the auth method to use with the `COPY` statement.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the authentication method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to the `COPY` statement. Instead, Redshift cluster needs to be configured with an AWS Role ARN that allows it to load data from S3. This Role ARN needs to be passed in the `roleArn` setting above. You can find more information [here](https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html). With 'TempCreds', temporary credentials will be created for every load operation and these temporary credentials will be passed to the `COPY` statement. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to, with the following permissions: `s3:GetObject*`, `s3:ListBucket`, and `s3:GetBucketLocation`.                                                                                                                                                                                                                                                                                                                                                                                                   |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.*`                                      | Optional. Custom JDBC configuration. The default value is `{"ssl": true}`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `jdbc.BlockingRowsMode`                       | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.DisableIsValidQuery`                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.DSILogLevel`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.FilterLevel`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.loginTimeout`                           | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.loglevel`                               | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.socketTimeout`                          | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.ssl`                                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.sslMode`                                | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.sslRootCert`                            | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.tcpKeepAlive`                           | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `jdbc.TCPKeepAliveMinutes`                    | Optional. Refer to the [Redshift JDBC driver reference](https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.54.1082/Amazon+Redshift+JDBC+Connector+Install+Guide.pdf).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

## Snowflake Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type`                                        | Optional. The only valid value is the default: `snowflake`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `snowflakeRegion`                             | Required. AWS Region used by Snowflake to access its endpoint.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `username`                                    | Required. Snowflake user with necessary role granted to load data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `role`                                        | Optional. Snowflake role with permission to load data. If it is not provided, the default role in Snowflake will be used.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `password`                                    | Required. Password of the Snowflake user. Can be plain text, read from the EC2 parameter store or GCP secret manager (see below).                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `password.secretStore.parameterName`          | Alternative way for passing in the user password.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `account`                                     | Required. Target Snowflake account.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `warehouse`                                   | Required. Snowflake warehouse which the SQL statements submitted by Snowflake Loader will run on.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `database`                                    | Required. Snowflake database which the data will be loaded to.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `schema`                                      | Required. Target schema                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `transformedStage.*`                          | Required if `NoCreds` is chosen as load auth method. Snowflake stage for transformed events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `transformedStage.name`                       | Required if `transformedStage` is included. The name of the stage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `transformedStage.location`                   | Required if `transformedStage` is included. The S3 path used as stage location. (Not needed since 5.2.0 because it is auto-configured)                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `folderMonitoringStage.*`                     | Required if `monitoring.folders` section is configured and `NoCreds` is chosen as load auth method. Snowflake stage to load folder monitoring entries into temporary Snowflake table.                                                                                                                                                                                                                                                                                                                                                                                                            |
| `folderMonitoringStage.name`                  | Required if `folderMonitoringStage` is included. The name of the stage.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `folderMonitoringStage.location`              | Required if `folderMonitoringStage` is included. The S3 path used as stage location. (Not needed since 5.2.0 because it is auto-configured)                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `appName`                                     | Optional. Name passed as 'application' property while creating Snowflake connection. The default is `Snowplow_OSS`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `maxError`                                    | Optional. A table copy statement will skip an input file when the number of errors in it exceeds the specified number. This setting is used during initial loading and thus can filter out only invalid JSONs (which is impossible situation if used with Transformer).                                                                                                                                                                                                                                                                                                                          |
| `jdbcHost`                                    | Optional. Host for the JDBC driver that has priority over automatically derived hosts. If it is not given, host will be created automatically according to given `snowflakeRegion`.                                                                                                                                                                                                                                                                                                                                                                                                              |
| `loadAuthMethod.*`                            | Optional, default method is `NoCreds`. Specifies the auth method to use with `COPY INTO` statement. Note that `TempCreds` auth method doesn't work when data is loaded from GCS.                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the auth method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to `COPY INTO` statement. Instead, `transformedStage` and `folderMonitoringStage` specified above will be used. More information can be found [here](https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html). With `TempCreds`, temporary credentials will be created for every load operation and these temporary credentials will be passed to `COPY INTO` statement. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to. You can find the list of necessary permissions needs to be given to role in [here](https://docs.snowflake.com/en/user-guide/data-load-s3-config-aws-iam-user.html).                                                                                                                                                                                                                             |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                  |
| `readyCheck` since (5.4.0)                    | Optional. Either `ResumeWarehouse` (the default) or `Select1`. The command the loader runs to prepare the JDBC connection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

## Databricks Loader `storage` section

| Parameter                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                                        | Optional. The only valid value is the default: `databricks`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `host`                                        | Required. Hostname of Databricks cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `password`                                    | Required if `oauth` section is not defined. [Databricks access token](https://docs.databricks.com/dev-tools/api/latest/authentication.html). Can be plain text, read from the EC2 parameter store or GCP secret manager (see below).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `oauth.clientId`                              | Required if `password` is not defined. Client id used in [Databricks OAuth machine-to-machine](https://docs.databricks.com/en/integrations/jdbc/authentication.html#oauth-machine-to-machine-m2m-authentication) authentication flow.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `oauth.clientSecret`                          | Required if `password` is not defined. Client secret used in [Databricks OAuth machine-to-machine](https://docs.databricks.com/en/integrations/jdbc/authentication.html#oauth-machine-to-machine-m2m-authentication) authentication flow.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `eventsOptimizePeriod`                        | Optional. The default value is `2 days`. Optimize period per table, that will be used as predicate for the `OPTIMIZE` command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `password.secretManager.parameterName`        | Alternative way for passing in the access token.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `schema`                                      | Required. Target schema.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `port`                                        | Required. Port of Databricks cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `httpPath`                                    | Required. Http Path of Databricks cluster. Get it from the JDBC connection details after the cluster has been created.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `catalog`                                     | Optional. The default value is `hive_metastore`. [Databricks unity catalog name](https://docs.databricks.com/data-governance/unity-catalog/index.html).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `userAgent`                                   | Optional. The default value is `snowplow-rdbloader-oss`. User agent name for Databricks connection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `loadAuthMethod.*`                            | Optional, default method is `NoCreds`. Specifies the auth method to use with `COPY INTO` statement                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `loadAuthMethod.type`                         | Required if `loadAuthMethod` section is included. Specifies the type of the auth method. The possible values are `NoCreds` and `TempCreds`. With `NoCreds`, no credentials will be passed to `COPY INTO` statement. Databricks cluster needs to have permission to access transformer output S3 bucket. More information can be found [here](https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html). With `TempCreds`, temporary credentials will be created for every load operation and these temporary credentials will be passed to `COPY INTO` statement. With this way, Databricks cluster doesn't need permission to access to transformer output S3 bucket. This access will be provided by temporary credentials. |
| `loadAuthMethod.roleArn`                      | Required if `loadAuthMethod.type` is `TempCreds`. IAM role that is used while creating temporary credentials. This role should allow access to the S3 bucket the transformer will write data to, with the following permissions: `s3:GetObject*`, `s3:ListBucket`, and `s3:GetBucketLocation`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `loadAuthMethod.credentialsTtl` since (5.4.0) | Optional, default value `1 hour`. If `TempCreds` load auth method is used, this value will be used as a session duration of temporary credentials used for loading data and folder monitoring. In that case, it can't be greater than 1 hour and can't be less than 15 minutes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `logLevel` since (5.3.2)                      | Optional. The default value is 3. Specifies JDBC driver log level. 0 - Disable all logging. 1 - Log severe error events that lead the driver to abort. 2 - Log error events that might allow the driver to continue running. 3 - Log events that might result in an error if action is not taken. (default) 4 - Log general information that describes the progress of the driver. 5 - Log detailed information that is useful for debugging the driver. 6 - Log all driver activity.                                                                                                                                                                                                                                                                                 |

## AWS specific settings

| Parameter                | Description                                                                                                                                                                                                          |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region`                 | Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html). AWS region of the S3 bucket. |
| `messageQueue.type`      | Required. Type of the message queue. It should be `sqs` when application is run on AWS.                                                                                                                              |
| `messageQueue.queueName` | Required. The name of the SQS queue used by the transformer and loader to communicate.                                                                                                                               |
| `jsonpaths`              | Optional. An S3 URI that holds JSONPath files.                                                                                                                                                                       |

## GCP specific settings

Only Snowflake Loader can be run on GCP at the moment.

| Parameter                   | Description                                                                                      |
| --------------------------- | ------------------------------------------------------------------------------------------------ |
| `messageQueue.type`         | Type of the message queue. It should be `pubsub` when application is run on GCP.                 |
| `messageQueue.subscription` | Required. The name of the Pubsub subscription used by the transformer and loader to communicate. |

## Azure specific settings

Only Snowflake Loader can be run on Azure at the moment.

| Parameter                       | Description                                                                                                                     |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `blobStorageEndpoint`           | Endpoint of Azure Blob Storage container that contains transformer's output.                                                    |
| `azureVaultName`                | Name of the Azure Key Vault where application secrets are stored. Required if secret store is used in `storage.password` field. |
| `messageQueue.type`             | Type of the message queue. It should be `kafka` when application is run on Azure.                                               |
| `messageQueue.topicName`        | Name of the Kafka topic used to communicate with Transformer.                                                                   |
| `messageQueue.bootstrapServers` | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster.                                  |
| `messageQueue.consumerConf`     | Optional. Kafka consumer configuration. See <https://kafka.apache.org/documentation/#consumerconfigs> for all properties.       |

## Common loader settings

| Parameter                            | Description                                                                                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schedules.*`                        | Optional. Tasks scheduled to run periodically.                                                                                                                                                                                                                                                                                                                                                            |
| `schedules.noOperation.[*]`          | Optional. Array of objects which specifies no-operation windows during which periodically scheduled tasks (configured in this section) will not run.                                                                                                                                                                                                                                                      |
| `schedules.noOperation.[*].name`     | Human-readable name of the no-op window.                                                                                                                                                                                                                                                                                                                                                                  |
| `schedules.noOperation.[*].when`     | Cron expression with second granularity.                                                                                                                                                                                                                                                                                                                                                                  |
| `schedules.noOperation.[*].duration` | For how long the loader should be paused.                                                                                                                                                                                                                                                                                                                                                                 |
| `schedules.optimizeEvents`           | Optional. The default value is `"0 0 0 ? * *"` (i.e. every day at 00:00, JVM timezone). Cron expression with second granularity that specifies the schedule to run periodically an `OPTIMIZE` statement on event table. (Only for Databricks Loader)                                                                                                                                                      |
| `schedules.optimizeManifest`         | Optional. The default value is `"0 0 5 ? * *"` (i.e. every day at 05:00 AM, JVM timezone). Cron expression with second granularity that specifies the schedule to run periodically an `OPTIMIZE` statement on manifest table. (Only for Databricks Loader)                                                                                                                                                |
| `retryQueue.*`                       | Optional. Additional backlog of recently failed folders that could be automatically retried. Retry queue saves a failed folder and then re-reads the info from `shredding_complete` S3 file. (Despite the legacy name of the message, which is required for backward compatibility, this also works with wide row format data.)                                                                           |
| `retryQueue.period`                  | Required if `retryQueue` section is configured. How often batch of failed folders should be pulled into a discovery queue.                                                                                                                                                                                                                                                                                |
| `retryQueue.size`                    | Required if `retryQueue` section is configured. How many failures should be kept in memory. After the limit is reached new failures are dropped.                                                                                                                                                                                                                                                          |
| `retryQueue.maxAttempts`             | Required if `retryQueue` section is configured. How many attempts to make for each folder. After the limit is reached new failures are dropped.                                                                                                                                                                                                                                                           |
| `retryQueue.interval`                | Required if `retryQueue` section is configured. Artificial pause after each failed folder being added to the queue.                                                                                                                                                                                                                                                                                       |
| `retries.*`                          | Optional. Unlike `retryQueue` these retries happen immediately, without proceeding to another message.                                                                                                                                                                                                                                                                                                    |
| `retries.backoff`                    | Required if `retries` section is configured. Starting backoff period, eg '30 seconds'.                                                                                                                                                                                                                                                                                                                    |
| `retries.strategy`                   | Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                                                             |
| `retries.attempts`                   | Optional. How many attempts to make before sending the message into retry queue. If missing, `cumulativeBound` will be used.                                                                                                                                                                                                                                                                              |
| `retries.cumulativeBound`            | Optional. When backoff reaches this delay, eg '1 hour', the loader will stop retrying. If both this and `attempts` are not set, the loader will retry indefinitely.                                                                                                                                                                                                                                       |
| `timeouts.loading`                   | Optional. How long, eg '1 hour', `COPY` statement execution can take before considering Redshift unhealthy. If no progress (ie, moving to a different subfolder) within this period, the loader will abort the transaction.                                                                                                                                                                               |
| `timeouts.nonLoading`                | Optional. How long, eg '10 mins', non-loading steps such as `ALTER TABLE` can take before considering Redshift unhealthy.                                                                                                                                                                                                                                                                                 |
| `timeouts.sqsVisibility`             | Optional. The time window in which a message must be acknowledged. Otherwise it is considered abandoned. If a message has been pulled, but hasn't been acked, the time before it is again available to consumers is equal to this, eg '5 mins'. Another consequence is that if the loader has failed on processing a message, the next time it will get this (or anything) from the queue has this delay. |
| `readyCheck.*`                       | Optional. Check the target destination to make sure it is ready.                                                                                                                                                                                                                                                                                                                                          |
| `readyCheck.backoff`                 | Optional. The default value is `15 seconds`. Starting backoff period.                                                                                                                                                                                                                                                                                                                                     |
| `readyCheck.strategy`                | Optional. The default value is `CONSTANT`. Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                  |
| `initRetries.*`                      | Optional. Retries configuration for initialization block. It will retry on all exceptions from there.                                                                                                                                                                                                                                                                                                     |
| `initRetries.backoff`                | Required if `initRetries` section is configured. Starting backoff period, eg '30 seconds'.                                                                                                                                                                                                                                                                                                                |
| `initRetries.strategy`               | Backoff strategy used during retry. The possible values are `JITTER`, `CONSTANT`, `EXPONENTIAL`, `FIBONACCI`.                                                                                                                                                                                                                                                                                             |
| `initRetries.attempts`               | Optional. How many attempts to make before sending the message into retry queue. If missing, `cumulativeBound` will be used.                                                                                                                                                                                                                                                                              |
| `initRetries.cumulativeBound`        | Optional. When backoff reaches this delay, eg '1 hour', the loader will stop retrying. If both this and `attempts` are not set, the loader will retry indefinitely.                                                                                                                                                                                                                                       |
| `telemetry.disable`                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                 |
| `telemetry.userProvidedId`           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                       |

## Common monitoring settings

| Parameter                              | Description                                                                                                                                                                                                       |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `monitoring.webhook.endpoint`          | Optional. An HTTP endpoint where monitoring alerts should be sent.                                                                                                                                                |
| `monitoring.webhook.tags`              | Optional. Custom key-value pairs which can be added to the monitoring webhooks. Eg, `{"tag1": "label1"}`.                                                                                                         |
| `monitoring.snowplow.appId`            | Optional. When using Snowplow tracking, set this `appId` in the event.                                                                                                                                            |
| `monitoring.snowplow.collector`        | Optional. Set to a collector URL to turn on Snowplow tracking.                                                                                                                                                    |
| `monitoring.sentry.dsn`                | Optional. For tracking runtime exceptions.                                                                                                                                                                        |
| `monitoring.metrics.*`                 | Send metrics to a StatsD server or stdout.                                                                                                                                                                        |
| `monitoring.metrics.period`            | Optional. The default is 5 minutes. Period for metrics emitted periodically.                                                                                                                                      |
| `monitoring.metrics.statsd.*`          | Optional. For sending loading metrics (latency and event counts) to a StatsD server.                                                                                                                              |
| `monitoring.metrics.statsd.hostname`   | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                |
| `monitoring.metrics.statsd.port`       | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                         |
| `monitoring.metrics.statsd.tags`       | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                      |
| `monitoring.metrics.statsd.prefix`     | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.rdbloader`.                                                                                                                       |
| `monitoring.metrics.stdout.*`          | Optional. For sending metrics to stdout.                                                                                                                                                                          |
| `monitoring.metrics.stdout.prefix`     | Optional. Overrides the default metric prefix.                                                                                                                                                                    |
| `monitoring.folders.*`                 | Optional. Configuration for periodic unloaded / corrupted folders checks.                                                                                                                                         |
| `monitoring.folders.staging`           | Required if `monitoring.folders` section is configured. Path where loader could store auxiliary logs for folder monitoring. Loader should be able to write here, storage target should be able to load from here. |
| `monitoring.folders.period`            | Required if `monitoring.folders` section is configured. How often to check for unloaded / corrupted folders.                                                                                                      |
| `monitoring.folders.since`             | Optional. Specifies from when folder monitoring will start to monitor. Note that this is a duration, eg `7 days`, relative to when the loader is launched.                                                        |
| `monitoring.folders.until`             | Optional. Specifies until when folder monitoring will monitor. Note that this is a duration, eg `7 days`, relative to when the loader is launched.                                                                |
| `monitoring.folders.transformerOutput` | Required if `monitoring.folders` section is configured. Path to transformed archive.                                                                                                                              |
| `monitoring.folders.failBeforeAlarm`   | Required if `monitoring.folders` section is configured. How many times the check can fail before generating an alarm. Within the specified tolerance, failures will log a `WARNING` instead.                      |
| `monitoring.healthCheck.*`             | Optional. Periodic DB health check, raising a warning if DB hasn't responded to `SELECT 1`.                                                                                                                       |
| `monitoring.healthCheck.frequency`     | Required if `monitoring.healthCheck` section is configured. How often to run a periodic DB health check.                                                                                                          |
| `monitoring.healthCheck.timeout`       | Required if `monitoring.healthCheck` section is configured. How long to wait for a health check response.                                                                                                         |

---

# Event deduplication with Spark transformer
> Deduplicate Snowplow events in-batch and cross-batch using natural and synthetic deduplication strategies with DynamoDB storage.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/

**NOTE:** Deduplication is currently only available in the [Spark transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/).

Duplicates are a common problem in event pipelines. At the root of it is the fact that we can't guarantee every event has a unique UUID because:

- We have no exactly-once delivery guarantees
- User-side software can send events more than once
- Robots can send events reusing the same event ID

Depending on your use case, you may choose to ignore duplicates, or deal with them once the events are in the data warehouse.

If you are loading into **Redshift** (using [shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data)), we strongly recommend to deduplicate the data upstream of loading. Once duplicates are loaded into separate tables, table joins would create a Cartesian product.

This is less of a concern with [wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format) loading into **Snowflake**, where it's easier to deduplicate during the data modeling step in the warehouse.

This table shows the available deduplication mechanisms:

| Strategy                            | Batch?      | Same event ID? | Same event fingerprint? | Availability      |
| ----------------------------------- | ----------- | -------------- | ----------------------- | ----------------- |
| In-batch natural deduplication      | In-batch    | Yes            | Yes                     | Spark transformer |
| In-batch synthetic deduplication    | In-batch    | Yes            | No                      | Spark transformer |
| Cross-batch natural deduplication   | Cross-batch | Yes            | Yes                     | Spark transformer |
| Cross-batch synthetic deduplication | Cross-batch | Yes            | No                      | Not supported     |

## In-batch natural deduplication

"Natural duplicates" are events which share the same event ID (`event_id`) and the same event payload (`event_fingerprint`), meaning that they are semantically identical to each other. For a given batch of events being processed, RDB Transformer keeps only the first out of each group of natural duplicates and discards all others.

To enable this functionality, you need to have the [Event Fingerprint Enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) enabled in Enrich. This will correctly populate the `event_fingerprint` property. No changes are required in the transformer's own `config.hocon` file.

If the fingerprint enrichment is not enabled, the transformer will assign a random UUID to each event, effectively marking all events as non-duplicates (in the 'natural' sense).

## In-batch synthetic deduplication

"Synthetic duplicates" are events which share the same event ID (`event_id`), but have different event payload (`event_fingerprint`), meaning that they can be either semantically independent events or the same events with slightly different payloads (caused by third-party software). For a given batch of events being processed, RDB Transformer uses the following strategy:

- Collect all the events with identical `event_id` which are left after natural deduplication
- Generate new random `event_id` for each of them
- Create a [`duplicate`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/duplicate/jsonschema/1-0-0) context with the original `event_id` for each event where the duplicated `event_id` was found.

There is no transformer configuration required for this functionality: deduplication is performed automatically. It is optional but highly recommended to use the [Event Fingerprint Enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) in Enrich in order to correctly populate the `event_fingerprint` property.

## Cross-batch natural deduplication

The strategies described above deal with duplicates within the same batch of data being processed. But what if events are duplicated across batches?

To apply any of these strategies, we need to store information about previously seen duplicates, so that we can compare events in the current batch against them. We don't need to store the whole event: just the `event_id` and the `event_fingerprint` fields.

We need to store these in a database that allows fast random access, so we chose DynamoDB, a fully managed NoSQL database service.

### How to enable cross-batch natural deduplication

To enable cross-batch natural deduplication, you must provide a third configuration option in the `RDB Transformer` step of the Dataflow Runner playbook, using the `--duplicate-storage-config` flag. Like the other options, this needs to be provided as a base64-encoded string. This config file contains information about the DynamoDB table to be used, as well as credentials for accessing it. For more details on the config file structure, refer to the [Snowplow Events Manifest](https://github.com/snowplow-incubator/snowplow-events-manifest) library and its documentation.

An example step definition can look like this:

```json
{
  "type": "CUSTOM_JAR",
  "name": "RDB Transformer",
  "actionOnFailure": "CANCEL_AND_WAIT",
  "jar": "command-runner.jar",
  "arguments": [
    "spark-submit",
    "--class", "com.snowplowanalytics.snowplow.rdbloader.shredder.batch.Main",
    "--master", "yarn",
    "--deploy-mode", "cluster",
    "s3://snowplow-hosted-assets-eu-central-1/4-storage/transformer-batch/snowplow-transformer-batch-4.1.0.jar",
    "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",
    "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}",
    "--duplicate-storage-config", "{{base64File "/home/snowplow/configs/snowplow/duplicate-storage-config.json"}}"
  ]
}
```

If this configuration option is not provided, cross-batch natural deduplication will be disabled. In-batch deduplication will still work however.

### Costs and performance implications

Cross-batch deduplication uses DynamoDB as transient storage and therefore has associated AWS costs. The default write capacity is 100 units, which should roughly cost USD50 per month. Note that at this rate your shred job can get throttled by insufficient capacity, even with a very powerful EMR cluster. You can tweak throughput to match your needs but that will inflate the bill.

### How RDB Transformer uses DynamoDB for deduplication

We store duplicate data in a DynamoDB table with the following attributes:

- `eventId`, a String
- `fingerprint`, a String
- `etlTime`, a Date
- `ttl`, a Date.

We can query this table to see if the event that is currently being processed has been seen before based on `event_id` and `event_fingerprint`.

We store the `etl_timestamp` to prevent issues in case of a failed transformer run. If a run fails and is then rerun, we don't want the rerun to consider rows in the DynamoDB table which were written as part of the failed run. Otherwise all events that were processed by the failed run will be rejected as duplicates.

To update the DynamoDB table, RDB Transformer uses so-called 'conditional updates' to perform a check-and-set operation on a per-event basis. The algorithm is as follows:

- Attempt to write the `(event_id, event_fingerprint, etl_timestamp)` triple to DynamoDB but succeed only if the `(event_id, event_fingerprint)` pair cannot be found in the table with an earlier `etl_timestamp` than the current one.
- If the write fails, we have a natural duplicate. We can safely drop it because we know that we have the 'original' of this event already safely in the data warehouse.
- If the write succeeds, we know we have an event which is not a natural duplicate. (It could still be a synthetic duplicate however.)

The transformer performs this check after grouping the batch by `event_id` and `event_fingerprint`. This ensures that all check-and-set requests for a specific `(event_id, event_fingerprint)` pair will come from a single mapper, avoiding race conditions.

To keep the DynamoDB table size in check, we're using the time-to-live feature which provides automatic cleanup after the specified time. For event manifests this time is the ETL timestamp plus 180 days. This is stored in the table's `ttl` attribute.

### Creating the DynamoDB table and IAM policy

If you provide a `duplicate-storage-config` that specifies a DynamoDB table but RDB Transformer can't find it upon launch, it will create it with the default provisioned throughput. That might not be enough for the amount of data you want to process. Creating the table upfront gives you the opportunity to spec it out according to your needs. This step is optional but recommended.

1. The table name can be anything, but it must be unique.
2. The partition key must be called `eventId` and have type String. The sort key must be called `fingerprint` and have type String. You can refer to the the [DynamoDB table definition](#how-rdb-transformer-uses-dynamodb-for-deduplication) above for the full table schema.
3. Uncheck the "Use default settings" checkbox and set "Write capacity units" to 100 (or your desired value).
4. After the table is created, note down its ARN in the "Overview" tab.
5. Create the IAM policy In the AWS console, navigate to IAM and go to "Policies". Select "Create Your Own Policy" and choose a descriptive name. Here's an example Policy Document that you can paste:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1486765706000",
      "Effect": "Allow",
      "Action": [
        "dynamodb:CreateTable",
        "dynamodb:DeleteTable",
        "dynamodb:DescribeTable",
        "dynamodb:PutItem"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-east-1:{{AWS_ACCOUNT_ID}}:table/snowplow-deduplication"
      ]
    }
  ]
}
```

Replace the element in the `Resources` array with the ARN that you noted down in step 4. If you've already created the table, the policy does not require the `dynamodb:CreateTable` and `dynamodb:DeleteTable` permissions.

---

# How the RDB Loader transforms enriched data for warehouses
> Transform Snowplow enriched events into shredded data for Redshift or wide row format for Snowflake and Databricks using Spark or stream transformers.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/

_For a high-level overview of the RDB Loader architecture, of which the transformer is a part, see [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)._

The `transformer` application can have two types of output:

- 'shredded' data
- wide row format.

Both the Spark transformer and the stream transformer can output both types. Which type to pick, depends on the intended storage target.

For loading into **Redshift**, use [shredded data](#shredded-data).

For loading into **Snowflake**, use [wide row format](#wide-row-format).

For loading into **Databricks**, use [wide row format](#wide-row-format).

## Shredded data

Shredding is the process of splitting a Snowplow enriched event into several smaller chunks, which can be inserted directly into **Redshift** tables.

A Snowplow enriched event is a 131-column tsv line. Each line contains all fields that constitute a specific event, including its id, timestamps, custom and derived contexts, etc.

After shredding, the following entities are split out from the original event:

1. **Atomic event.** A tsv line very similar to the enriched event but not containing JSON fields (`contexts`, `derived_contexts` and `unstruct_event`). The results are stored under a path similar to `shredded/good/run=2016-11-26-21-48-42/atomic-events/part-00000` and are available to load with RDB Loader or directly with Redshift `COPY`.
2. **Contexts.** Two JSON fields -- `contexts` and `derived_contexts` -- are extracted from the enriched event. Their original values are validated self-describing JSONs, consisting of a `schema` and a `data` property. After shredding, a third property is added, called `hierarchy`. This `hierarchy` contains fields you can use to later join your context SQL tables with the `atomic.events` table. One atomic event can be associated with multiple context entities. The results are stored under a path like `shredded/good/run=2016-11-26-21-48-42/shredded-types/vendor=com.acme/name=my_context/format=jsonschema/version=1-0-1/part-00000`, where the `part-*` files are valid ndJSON files which can be loaded with RDB Loader or directly with Redshift `COPY`.
3. **Self-describing (unstructured) events.** Same as the contexts described above but there is a strict one-to-one relation with atomic events. The results are stored under a path with the same structure as for contexts and are ready to be loaded with RDB Loader or directly with Redshift `COPY`.

These files are stored on S3 partitioned by type. When the data is loaded into Redshift, each type goes to a dedicated table.

The following diagram illustrates the process:

![](/assets/images/storage-loader-dataflow-96341b5e426da988ea3bc5c07a4949d7.png)

**NOTE:** Shredded data can currently only be loaded into **Redshift**.

## Wide row format

Unlike shredding, wide row format preserves data as a single line per event, with one column for each different type of contexts and self-describing events.

For contexts (aka entities), the type of the column is `ARRAY` and the name looks like `contexts_com_acme_my_custom_entity_schema_1`. Note the plural that matches the `ARRAY` type: in theory each `contexts_*` column may contain multiple entities.

For self-describing events, the type of the column is `OBJECT` and the name looks like `unstruct_event_com_acme_my_custom_event_schema_1`. Each line in the table contains only 1 event.

The values in these columns have recursive structure with arbitrary depth, which depends on the schema that describes them.

The results are stored under a path like `output=good/part-00000` and can be loaded with RDB Loader.

There are two options as output file format with wide row transformation: JSON and Parquet

JSON file format can be used for loading into **Snowflake** and Parquet file format can be used for loading into **Databricks**

---

# Spark transformer configuration reference
> Configure Spark batch transformer with EMR settings, S3 paths, output formats, and deduplication options for warehouse loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/configuration-reference/

The configuration reference in this page is written for Spark Transformer `6.3.0`

An example of the minimal required config for the Spark transformer can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

| Parameter                                                       | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input`                                                         | Required. S3 URI of the enriched archive. It must be populated separately with `run=YYYY-MM-DD-hh-mm-ss` directories.                                                                                                                                                                                                                                                                                         |
| `runInterval.*`                                                 | Specifies interval to process.                                                                                                                                                                                                                                                                                                                                                                                |
| `runInterval.sinceAge`                                          | Optional. A duration that specifies the maximum age of folders that should get processed. If `sinceAge` and `sinceTimestamp` are both specified, then the latest value of the two determines the earliest folder that will be processed.                                                                                                                                                                      |
| `runInterval.until`                                             | Optional. Process until this timestamp.                                                                                                                                                                                                                                                                                                                                                                       |
| `monitoring.sentry.dsn`                                         | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                    |
| `monitoring.metrics.cloudwatch` (since 5.5.0)                   | Optional. For sending metrics to Cloudwatch. If not set, metrics are not sent.                                                                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.cloudwatch.namespace` (since 5.5.0)         | Namespace that will contain the metrics in Cloudwatch. Example: `snowplow/transformer_batch`                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.cloudwatch.transformDuration` (since 5.5.0) | Name of the metric that contains the number of milliseconds needed to transform a folder. Example: `transform_duration`                                                                                                                                                                                                                                                                                       |
| `monitoring.metrics.cloudwatch.dimensions` (since 5.5.0)        | Any key-value pairs to be added as dimensions in Cloudwatch metrics. Example:```json
{"app_version": "x.y.z", "env": "prod"}
```                                                                                                                                                                                                                                                                              |
| `deduplication.*`                                               | Configure the way in-batch deduplication is performed                                                                                                                                                                                                                                                                                                                                                         |
| `deduplication.synthetic.type`                                  | Optional. The default is `BROADCAST`. Can be `NONE` (disable), `BROADCAST` (default) and `JOIN` (different low-level implementations).                                                                                                                                                                                                                                                                        |
| `deduplication.synthetic.cardinality`                           | Optional. The default is 1. Do not deduplicate pairs with less-or-equal cardinality.                                                                                                                                                                                                                                                                                                                          |
| `deduplication.natural`                                         | Optional. The default is 'true'. Enable or disable natural deduplication. Available since `5.1.0`                                                                                                                                                                                                                                                                                                             |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0)            | Optional, default = false. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                                                                                                                                              |
| `skipSchemas` (since 5.7.1)                                     | Optional, default = none. Supply a list of Iglu URIs and the transformer's output files will omit any columns using that schema. This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table.                                                                                                                                                    |
| `output.path`                                                   | Required. S3 URI of the transformed output.                                                                                                                                                                                                                                                                                                                                                                   |
| `output.compression`                                            | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                                                                                                                                     |
| `output.region`                                                 | AWS region of the S3 bucket. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                          |
| `output.maxRecordsPerFile` (since 5.4.0)                        | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.type` (since 5.4.0)                                 | Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                                                                                                                                   |
| `output.bad.streamName` (since 5.4.0)                           | Required if output type is `kinesis`. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                                                                                                                 |
| `output.bad.region` (since 5.4.0)                               | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `output.bad.recordLimit` (since 5.4.0)                          | Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                                  |
| `output.bad.byteLimit` (since 5.4.0)                            | Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                               |
| `output.bad.backoffPolicy.minBackoff` (since 5.4.0)             | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                     |
| `output.bad.backoffPolicy.maxBackoff` (since 5.4.0)             | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                           |
| `output.bad.backoffPolicy.maxRetries` (since 5.4.0)             | Optional, default = 10. Maximum number of retries for internal Kinesis errors.                                                                                                                                                                                                                                                                                                                                |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0)    | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.                                                                                                                                                                                                                                                                           |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0)    | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.                                                                                                                                                                                                                                                     |
| `queue.type`                                                    | Required. Type of the message queue. Can be either `sqs` or `sns`.                                                                                                                                                                                                                                                                                                                                            |
| `queue.queueName`                                               | Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.                                                                                                                                                                                                                                                                                                                           |
| `queue.topicArn`                                                | Required if queue type is `sns`. ARN of the SNS topic.                                                                                                                                                                                                                                                                                                                                                        |
| `queue.region`                                                  | AWS region of the SQS queue or SNS topic. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                             |
| `formats.*`                                                     | Schema-specific format settings.                                                                                                                                                                                                                                                                                                                                                                              |
| `formats.transformationType`                                    | Required. Type of transformation, either `shred` or `widerow`. See [Shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) and [Wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format).                                                                             |
| `formats.fileFormat`                                            | Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                                                                                                    |
| `formats.default`                                               | Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation. |
| `formats.tsv`                                                   | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.                                                                                                                                                                                                                                    |
| `formats.json`                                                  | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.                                                                                                                                                                                                                                    |
| `formats.skip`                                                  | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.                                                                                                                                                                                                                                                                                |
| `validations.*`                                                 | Optional. Criteria to validate events against                                                                                                                                                                                                                                                                                                                                                                 |
| `validations.minimumTimestamp`                                  | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                                                                                                                                            |
| `featureFlags.*`                                                | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                                                                                                                                                   |
| `featureFlags.legacyMessageFormat`                              | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                                                                                                                                          |
| `featureFlags.truncateAtomicFields` (since 5.4.0)               | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                                                                                                                                                  |

---

# Spark transformer for batch processing
> Transform enriched Snowplow data in batches using Spark on EMR with support for deduplication and wide row or shredded output.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/

> **Info:** For a high-level overview of the Transform process, see [Transforming enriched data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/). For guidance on picking the right `transformer` app, see [How to pick a transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/#how-to-pick-a-transformer).

The Spark-based transformer is a batch job designed to be deployed in an EMR cluster and process a bounded data set stored on S3.

In order to run it, you will need:

- the `snowplow-transformer-batch` jar file (from version 3.0.0 this replaces the `snowplow-rdb-shredder` asset)
- configuration files for the jar file
- an EMR cluster specification
- a way to spin up an EMR cluster and submit a job to it.

You can use any suitable tool to periodically submit the transformer job to an EMR cluster. We recommend you use our purpose-built [Dataflow Runner](https://github.com/snowplow/dataflow-runner) tool. All the examples below assume that Dataflow Runner is being used. Refer to the app's [documentation](/docs/api-reference/dataflow-runner/) for more details.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available in several S3 buckets that are accessible to an EMR cluster:

```text
s3://snowplow-hosted-assets/4-storage/transformer-batch/snowplow-transformer-batch-6.3.0.jar


-- or --


s3://snowplow-hosted-{{ region }}/4-storage/transformer-batch/snowplow-transformer-batch-6.3.0.jar
```

where `region` is one of `us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`, `eu-central-1`, `eu-west-2`, `ca-central-1`, `sa-east-1`, `ap-southeast-1`, `ap-southeast-2`, `ap-northeast-1`, `ap-northeast-2`, or `ap-south-1`. Pick the region of your EMR cluster.

## Configuring the EMR cluster

> **Warning:** Starting from version `5.5.0`, batch transformer requires to use Java 11 on EMR ([default is Java 8](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/configuring-java8.html)). See the `bootstrapActionConfigs` section in the configuration below.

Here's an example of an EMR cluster config file that can be used with Dataflow Runner:

```json
{
  "schema": "iglu:com.snowplowanalytics.dataflowrunner/ClusterConfig/avro/1-1-0",
  "data": {
    "name": "RDB Transformer",
    "region": "eu-central-1",
    "logUri": "s3://com-acme/logs/",
    "credentials": {
      "accessKeyId": "env",
      "secretAccessKey": "env"
    },
    "roles": {
      "jobflow": "EMR_EC2_DefaultRole",
      "service": "EMR_DefaultRole"
    },
    "ec2": {
      "amiVersion": "6.2.0",
      "keyName": "ec2-key-name",
      "location": {
        "vpc": {
          "subnetId": "subnet-id"
        }
      },
      "instances": {
        "master": {
          "type": "m4.large",
          "ebsConfiguration": {
            "ebsOptimized": true,
            "ebsBlockDeviceConfigs": []
          }
        },
        "core": {
          "type": "r4.xlarge",
          "count": 1
        },
        "task": {
          "type": "m4.large",
          "count": 0,
          "bid": "0.015"
        }
      }
    },
    "tags": [],
    "bootstrapActionConfigs": [
      {
        "name": "Use Java 11",
        "scriptBootstrapAction": {
          "path": "s3://<bucket>/<path>/emr-bootstrap-java-11.sh",
          "args": []
        }
      }
    ],
    "configurations": [
      {
        "classification": "core-site",
        "properties": {
          "Io.file.buffer.size": "65536"
        },
        "configurations": []
      },
      {
        "classification": "yarn-site",
        "properties": {
          "yarn.nodemanager.resource.memory-mb": "57344",
          "yarn.scheduler.maximum-allocation-mb": "57344",
          "yarn.nodemanager.vmem-check-enabled": "false"
        },
        "configurations": []
      },
      {
        "classification": "spark",
        "properties": {
          "maximizeResourceAllocation": "false"
        },
        "configurations": []
      },
      {
        "classification": "spark-defaults",
        "properties": {
          "spark.executor.memory": "7G",
          "spark.driver.memory": "7G",
          "spark.driver.cores": "3",
          "spark.yarn.driver.memoryOverhead": "1024",
          "spark.default.parallelism": "24",
          "spark.executor.cores": "1",
          "spark.executor.instances": "6",
          "spark.yarn.executor.memoryOverhead": "1024",
          "spark.dynamicAllocation.enabled": "false"
        },
        "configurations": []
      }
    ],
    "applications": [
      "Hadoop",
      "Spark"
    ]
  }
}
```

You will need to replace `<bucket>` and `<path>` (in the `bootstrapActionConfigs` section) according to where you placed `emr-bootstrap-java-11.sh`. The content of this file should be as follows:

```bash
#!/bin/bash

set -e

sudo update-alternatives --set java /usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/java

exit 0
```

This is a typical cluster configuration for processing \~1.5GB of uncompressed enriched data.

You need to change the following settings with your own values:

- `region`: the AWS region of your EMR cluster
- `logUri`: the location of an S3 bucket where EMR logs will be written
- `ec2.keyName` (optional): The name of an EC2 key pair that you’ll use to shh into the EMR cluster
- `ec2.location.vpc.subnetId`: your VPN subnet ID.

## Configuring `snowplow-transformer-batch`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Spark transformer can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.batch.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Spark transformer

To run the transformer on EMR with Dataflow Runner, you need:

- the EMR cluster config (see [Configuring the EMR cluster](#configuring-the-emr-cluster) above)
- a Dataflow Runner playbook (a DAG with steps to be submitted to the EMR cluster).

### Preparing the Dataflow Runner playbook

A typical playbook can look like:

```json

{
"schema": "iglu:com.snowplowanalytics.dataflowrunner/PlaybookConfig/avro/1-0-1",
"data": {
  "region": "eu-central-1",
  "credentials": {
    "accessKeyId": "env",
    "secretAccessKey": "env"
  },
  "steps": [
    {
      "type": "CUSTOM_JAR",
      "name": "S3DistCp enriched data archiving",
      "actionOnFailure": "CANCEL_AND_WAIT",
      "jar": "/usr/share/aws/emr/s3-dist-cp/lib/s3-dist-cp.jar",
      "arguments": [
        "--src", "s3://com-acme/enriched/sink/",
        "--dest", "s3://com-acme/enriched/archive/run={{nowWithFormat "2006-01-02-15-04-05"}}/",
        "--s3Endpoint", "s3-eu-central-1.amazonaws.com",
        "--srcPattern", ".*",
        "--outputCodec", "gz",
        "--deleteOnSuccess"
      ]
    },
    {
      "type": "CUSTOM_JAR",
      "name": "RDB Transformer",
      "actionOnFailure": "CANCEL_AND_WAIT",
      "jar": "command-runner.jar",
      "arguments": [
        "spark-submit",
        "--class", "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.Main",
        "--master", "yarn",
        "--deploy-mode", "cluster",
        "s3://snowplow-hosted-assets-eu-central-1/4-storage/transformer-batch/snowplow-transformer-batch-6.3.0.jar",
        "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",
        "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}"
      ]
    }
  ],
  "tags": []
}
}
```

This playbook consists of two steps. The first one copies the enriched data to a dedicated directory, from which the transformer will read it. The second step is the transformer Spark job that transforms the data.

You need to change the following settings with your own values:

- `region`: the AWS region of the EMR cluster
- `"--src"`: the bucket in which your enriched data is sunk by Enrich
- `"--dest"`: the bucket in which the data for your enriched data lake lives.

**NOTE:** The `"--src"` and `"--dest"` settings above apply only to the `s3DistCp` step of the playbook. The source and destination buckets for the transformer step are configured via the `config.hocon` file.

### Submitting the job to EMR with Dataflow Runner

Here's an example of putting all of the above together on a transient EMR cluster:

```bash
$ ./dataflow-runner run-transient \
  --emr-config path/to/cluster.conig \
  --emr-playbook path/to/playbook
```

This will spin up the cluster with the above configuration, submit the steps from the playbook, and terminate the cluster once all steps are completed.

For more examples on running EMR jobs with Dataflow Runner, as well as details on cluster configurations and playbooks, see the app's [documentation](/docs/api-reference/dataflow-runner/). It also details how you can submit steps to a persistent EMR cluster.

---

# Stream transformer for real-time processing
> Transform enriched Snowplow data in real-time from Kinesis, Pub/Sub, or Kafka streams without Spark or EMR.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/

> **Info:** For a high-level overview of the Transform process, see [Transforming enriched data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/). For guidance on picking the right `transformer` app, see [How to pick a transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/#how-to-pick-a-transformer).

Unlike the [Spark transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/), the stream transformer reads data directly from the enriched stream and does not use Spark or EMR. It's a plain JVM application, like Stream Enrich or S3 Loader.

Reading directly from stream means that the transformer can bypass the `s3DistCp` staging / archiving step.

Another benefit is that it doesn't process a bounded data set and can emit transformed folders based only on its configured frequency. This means the pipeline loading frequency is limited only by the storage target.

Stream Transformer has three variants: [Transformer Kinesis](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/), [Transformer Pubsub](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/) and [Transformer Kafka](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/). They are different variants for AWS, GCP and Azure.

---

# Transformer Kafka configuration reference
> Configure Transformer Kafka with stream settings, Azure Blob Storage output, windowing, and monitoring for Azure real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/configuration-reference/

The configuration reference in this page is written for Transformer Kafka `6.3.0`

An example of the minimal required config for the Transformer Kafka can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

| Parameter                                            | Description                                                                                                                                                               |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.topicName`                                    | Name of the Kafka topic to read from                                                                                                                                      |
| `input.bootstrapServers`                             | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                             |
| `input.consumerConf`                                 | Optional. Kafka consumer configuration. See <https://kafka.apache.org/documentation/#consumerconfigs> for all properties                                                  |
| `output.path`                                        | Azure Blob Storage path to transformer output                                                                                                                             |
| `output.compression`                                 | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                 |
| `output.bad.type`                                    | Optional. Either `kafka` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`. |
| `output.bad.topicName`                               | Required if output type is `kafka`. Name of the Kafka topic that will receive the bad data.                                                                               |
| `output.bad.bootstrapServers`                        | Required if output type is `kafka`. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                         |
| `output.producerConf`                                | Optional. Kafka producer configuration. See <https://kafka.apache.org/documentation/#producerconfigs> for all properties                                                  |
| `queue.topicName`                                    | Name of the Kafka topic used to communicate with Loader                                                                                                                   |
| `queue.bootstrapServers`                             | A list of host:port pairs to use for establishing the initial connection to the Kafka cluster                                                                             |
| `queue.producerConf`                                 | Optional. Kafka producer configuration. See <https://kafka.apache.org/documentation/#producerconfigs> for all properties                                                  |
| `monitoring.metrics.*`                               | Send metrics to a StatsD server or stdout.                                                                                                                                |
| `monitoring.metrics.statsd.*`                        | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                             |
| `monitoring.metrics.statsd.hostname`                 | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                        |
| `monitoring.metrics.statsd.port`                     | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                 |
| `monitoring.metrics.statsd.tags`                     | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                              |
| `monitoring.metrics.statsd.prefix`                   | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                             |
| `monitoring.metrics.stdout.*`                        | Optional. For sending metrics to stdout.                                                                                                                                  |
| `monitoring.metrics.stdout.prefix`                   | Optional. Overrides the default metric prefix.                                                                                                                            |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                 |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                       |
| `monitoring.sentry.dsn`                              | Optional. For tracking runtime exceptions.                                                                                                                                |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0) | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                           |
| `validations.*`                                      | Optional. Criteria to validate events against                                                                                                                             |
| `validations.minimumTimestamp`                       | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.        |
| `featureFlags.*`                                     | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                               |
| `featureFlags.legacyMessageFormat`                   | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                      |
| `featureFlags.truncateAtomicFields` (since 5.4.0)    | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.              |

---

# Transformer Kafka for Azure streams
> Stream transformer for Azure that reads enriched events from Kafka and writes transformed data to Azure Blob Storage in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/

> **Info:** This application is available since v5.7.0.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-kafka:6.3.0`

## Configuring `snowplow-transformer-kafka`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Kafka can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/azure/transformer.kafka.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kafka/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Kafka

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-kafka:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer Kafka (since version 5.7.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Transformer Kinesis configuration reference
> Configure Transformer Kinesis with stream settings, S3 output, windowing, and monitoring for AWS real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/configuration-reference/

The configuration reference in this page is written for Transformer Kinesis `6.3.0`

An example of the minimal required config for the Transformer Kinesis can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

| Parameter                                                    | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.appName`                                              | Optional. KCL app name. The default is `snowplow-rdb-transformer`                                                                                                                                                                                                                                                                                                                                             |
| `input.streamName`                                           | Required for `kinesis`. Enriched Kinesis stream name.                                                                                                                                                                                                                                                                                                                                                         |
| `input.region`                                               | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `input.position`                                             | Optional. Kinesis position: `LATEST` or `TRIM_HORIZON`. The default is `LATEST`.                                                                                                                                                                                                                                                                                                                              |
| `windowing`                                                  | Optional. Frequency to emit shredding complete message. The default is `10 minutes`. Maximum allowed value is `60 minutes`                                                                                                                                                                                                                                                                                    |
| `output.path`                                                | Required. S3 URI of the transformed output.                                                                                                                                                                                                                                                                                                                                                                   |
| `output.compression`                                         | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                                                                                                                                     |
| `output.region`                                              | AWS region of the S3 bucket. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                          |
| `output.maxRecordsPerFile` (since 5.4.0)                     | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                                                                                                                                        |
| `output.bad.type` (since 5.4.0)                              | Optional. Either `kinesis` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                                                                                                                                   |
| `output.bad.streamName` (since 5.4.0)                        | Required if output type is `kinesis`. Name of the Kinesis stream to write to.                                                                                                                                                                                                                                                                                                                                 |
| `output.bad.region` (since 5.4.0)                            | AWS region of the Kinesis stream. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                                     |
| `output.bad.recordLimit` (since 5.4.0)                       | Optional, default = 500. Limits the number of events in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                                  |
| `output.bad.byteLimit` (since 5.4.0)                         | Optional, default = 5242880. Limits the number of bytes in a single PutRecords Kinesis request.                                                                                                                                                                                                                                                                                                               |
| `output.bad.backoffPolicy.minBackoff` (since 5.4.0)          | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                     |
| `output.bad.backoffPolicy.maxBackoff` (since 5.4.0)          | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails with internal errors.                                                                                                                                                                                                                                                                                           |
| `output.bad.backoffPolicy.maxRetries` (since 5.4.0)          | Optional, default = 10. Maximum number of retries for internal Kinesis errors.                                                                                                                                                                                                                                                                                                                                |
| `output.bad.throttledBackoffPolicy.minBackoff` (since 5.4.0) | Optional, default = 100 milliseconds. Minimum backoff before retrying when writing to Kinesis fails in case of throughput exceeded.                                                                                                                                                                                                                                                                           |
| `output.bad.throttledBackoffPolicy.maxBackoff` (since 5.4.0) | Optional, default = 10 seconds. Maximum backoff before retrying when writing to Kinesis fails in case of throughput exceeded. Writing is retried forever.                                                                                                                                                                                                                                                     |
| `queue.type`                                                 | Required. Type of the message queue. Can be either `sqs` or `sns`.                                                                                                                                                                                                                                                                                                                                            |
| `queue.queueName`                                            | Required if queue type is `sqs`. Name of the SQS queue. SQS queue needs to be FIFO.                                                                                                                                                                                                                                                                                                                           |
| `queue.topicArn`                                             | Required if queue type is `sns`. ARN of the SNS topic.                                                                                                                                                                                                                                                                                                                                                        |
| `queue.region`                                               | AWS region of the SQS queue or SNS topic. Optional if it can be resolved with [AWS region provider chain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).                                                                                                                                                                             |
| `formats.*`                                                  | Schema-specific format settings.                                                                                                                                                                                                                                                                                                                                                                              |
| `formats.transformationType`                                 | Required. Type of transformation, either `shred` or `widerow`. See [Shredded data](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data) and [Wide row format](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#wide-row-format).                                                                             |
| `formats.fileFormat`                                         | Optional. The default is `JSON`. Output file format produced when transformation is `widerow`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                                                                                                    |
| `formats.default`                                            | Optional. The default is `TSV`. Data format produced by default when transformation is `shred`. Either `TSV` or `JSON`. `TSV` is recommended as it enables table autocreation, but requires an Iglu Server to be available with known schemas (including Snowplow schemas). `JSON` does not require an Iglu Server, but requires Redshift JSONPaths to be configured and does not support table autocreation. |
| `formats.tsv`                                                | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `JSON` this list of schemas will still be shredded into `TSV`.                                                                                                                                                                                                                                    |
| `formats.json`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. If `default` is set to `TSV` this list of schemas will still be shredded into `JSON`.                                                                                                                                                                                                                                    |
| `formats.skip`                                               | Optional. List of Iglu URIs, but can be set to empty list `[]` which is the default. Schemas for which loading can be skipped.                                                                                                                                                                                                                                                                                |
| `monitoring.metrics.*`                                       | Send metrics to a StatsD server or stdout.                                                                                                                                                                                                                                                                                                                                                                    |
| `monitoring.metrics.statsd.*`                                | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.hostname`                         | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                                                                                                                                                                                                            |
| `monitoring.metrics.statsd.port`                             | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                                                                                                                                                                                                                     |
| `monitoring.metrics.statsd.tags`                             | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                                                                                                                                                                                                                  |
| `monitoring.metrics.statsd.prefix`                           | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                                                                                                                                                                                                                                                                 |
| `monitoring.metrics.stdout.*`                                | Optional. For sending metrics to stdout.                                                                                                                                                                                                                                                                                                                                                                      |
| `monitoring.metrics.stdout.prefix`                           | Optional. Overrides the default metric prefix.                                                                                                                                                                                                                                                                                                                                                                |
| `telemetry.disable`                                          | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                                                                     |
| `telemetry.userProvidedId`                                   | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                                                                           |
| `monitoring.sentry.dsn`                                      | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                                                                                                                                    |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0)         | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                                                                                                                                               |
| `validations.*`                                              | Optional. Criteria to validate events against                                                                                                                                                                                                                                                                                                                                                                 |
| `validations.minimumTimestamp`                               | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                                                                                                                                            |
| `featureFlags.*`                                             | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                                                                                                                                                   |
| `featureFlags.legacyMessageFormat`                           | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                                                                                                                                          |
| `featureFlags.truncateAtomicFields` (since 5.4.0)            | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                                                                                                                                                  |

---

# Transformer Kinesis for AWS streams
> Stream transformer for AWS that reads enriched events from Kinesis and writes transformed data to S3 in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-kinesis:6.3.0`

## Configuring `snowplow-transformer-kinesis`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Kinesis can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/aws/transformer.kinesis.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Kinesis

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-kinesis:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer Kinesis (since version 4.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Transformer Pub/Sub configuration reference
> Configure Transformer Pub/Sub with stream settings, GCS output, windowing, and monitoring for GCP real-time transformation.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/configuration-reference/

The configuration reference in this page is written for Transformer Pubsub `6.3.0`

An example of the minimal required config for the Transformer Pubsub can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.reference.hocon).

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

| Parameter                                            | Description                                                                                                                                                                                                                                                                               |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input.subscription`                                 | Name of the Pubsub subscription with the enriched events                                                                                                                                                                                                                                  |
| `input.parallelPullCount`                            | Optional. Default value 1. Number of threads used internally by permutive library to handle incoming messages. These threads do very little "work" apart from writing the message to a concurrent Queue.                                                                                  |
| `input.bufferSize`                                   | Optional. Default value 500. The max size of the buffer queue used between fs2-pubsub and java-pubsub libraries.                                                                                                                                                                          |
| `input.maxAckExtensionPeriod`                        | Optional. Default value '1 hour'. The maximum period a message ack deadline will be extended.                                                                                                                                                                                             |
| `output.path`                                        | Required. GCS URI of the transformed output. It needs to have `gs://` URI scheme                                                                                                                                                                                                          |
| `output.compression`                                 | Optional. One of `NONE` or `GZIP`. The default is `GZIP`.                                                                                                                                                                                                                                 |
| `output.bufferSize`                                  | Optional. Default value 4096. During the window period, processed items are stored in a buffer. This value determines the size of this buffer. When its limit is reached, buffer content is flushed to blob storage.                                                                      |
| `output.maxRecordsPerFile` (since 5.4.0)             | Optional. Default = 10000. Max number of events per parquet partition.                                                                                                                                                                                                                    |
| `output.bad.type` (since 5.4.0)                      | Optional. Either `pubsub` or `file`, default value `file`. Type of bad output sink. When `file`, failed events are written as files under URI configured in `output.path`.                                                                                                                |
| `output.bad.topic` (since 5.4.0)                     | Required if output type is `pubsub`. Name of the PubSub topic that will receive the bad data.                                                                                                                                                                                             |
| `output.bad.batchSize` (since 5.4.0)                 | Optional. Default = 1000, max = 1000. Maximum number of messages sent to PubSub within a batch. When the buffer reaches this number of messages they are sent.                                                                                                                            |
| `output.bad.requestByteThreshold` (since 5.4.0)      | Optional. Default = 8000000, max = 10MB. Maximum number of bytes sent to PubSub within a batch. When the buffer reaches this size messages are sent.                                                                                                                                      |
| `output.bad.delayThreshold` (since 5.4.0)            | Optional. Default = 200 milliseconds. Delay threshold to use for PubSub batching. After this amount of time has elapsed, before `batchSize` and `requestByteThreshold` have been reached, messages from the buffer will be sent.                                                          |
| `queue.topic`                                        | Name of the Pubsub topic used to communicate with Loader                                                                                                                                                                                                                                  |
| `formats.fileFormat`                                 | Optional. The default option at the moment is `JSON`. Either `JSON` or `PARQUET`.                                                                                                                                                                                                         |
| `windowing`                                          | Optional. Frequency to emit shredding complete message. The default is `5 minutes`. Note that there is a problem with acking messages when window period is greater than 10 minute in transformer-pubsub. Therefore, it is advisable to make window period equal or less than 10 minutes. |
| `monitoring.metrics.*`                               | Send metrics to a StatsD server or stdout.                                                                                                                                                                                                                                                |
| `monitoring.metrics.statsd.*`                        | Optional. For sending metrics (good and bad event counts) to a StatsD server.                                                                                                                                                                                                             |
| `monitoring.metrics.statsd.hostname`                 | Required if `monitoring.metrics.statsd` section is configured. The host name of the StatsD server.                                                                                                                                                                                        |
| `monitoring.metrics.statsd.port`                     | Required if `monitoring.metrics.statsd` section is configured. Port of the StatsD server.                                                                                                                                                                                                 |
| `monitoring.metrics.statsd.tags`                     | Optional. Tags which are used to annotate the StatsD metric with any contextual information.                                                                                                                                                                                              |
| `monitoring.metrics.statsd.prefix`                   | Optional. Configures the prefix of StatsD metric names. The default is `snoplow.transformer`.                                                                                                                                                                                             |
| `monitoring.metrics.stdout.*`                        | Optional. For sending metrics to stdout.                                                                                                                                                                                                                                                  |
| `monitoring.metrics.stdout.prefix`                   | Optional. Overrides the default metric prefix.                                                                                                                                                                                                                                            |
| `telemetry.disable`                                  | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                 |
| `telemetry.userProvidedId`                           | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                       |
| `monitoring.sentry.dsn`                              | Optional. For tracking runtime exceptions.                                                                                                                                                                                                                                                |
| `featureFlags.enableMaxRecordsPerFile` (since 5.4.0) | Optional, default = true. When enabled, `output.maxRecordsPerFile` configuration parameter is going to be used.                                                                                                                                                                           |
| `validations.*`                                      | Optional. Criteria to validate events against                                                                                                                                                                                                                                             |
| `validations.minimumTimestamp`                       | This is currently the only validation criterion. It checks that all timestamps in the event are older than a specific point in time, eg `2021-11-18T11:00:00.00Z`.                                                                                                                        |
| `featureFlags.*`                                     | Optional. Enable features that are still in beta, or which aim to enable smoother upgrades.                                                                                                                                                                                               |
| `featureFlags.legacyMessageFormat`                   | This currently the only feature flag. Setting this to `true` allows you to use a new version of the transformer with an older version of the loader.                                                                                                                                      |
| `featureFlags.truncateAtomicFields` (since 5.4.0)    | Optional, default `false`. When enabled, event's atomic fields are truncated (based on the length limits from the atomic JSON schema) before transformation.                                                                                                                              |

---

# Transformer Pub/Sub for GCP streams
> Stream transformer for GCP that reads enriched events from Pub/Sub and writes transformed data to GCS in real-time.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/

> **Info:** This application is available since v5.0.0.

## Downloading the artifact

The asset is published as a jar file attached to the [Github release notes](https://github.com/snowplow/snowplow-rdb-loader/releases) for each version.

It's also available as a Docker image on Docker Hub under `snowplow/transformer-pubsub:6.3.0`

## Configuring `snowplow-transformer-pubsub`

The transformer takes two configuration files:

- a `config.hocon` file with application settings
- an `iglu_resolver.json` file with the resolver configuration for your [Iglu](https://github.com/snowplow/iglu) schema registry.

An example of the minimal required config for the Transformer Pubsub can be found [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.minimal.hocon) and a more detailed one [here](https://github.com/snowplow/snowplow-rdb-loader/tree/master/config/transformer/gcp/transformer.pubsub.config.reference.hocon). For details about each setting, see the [configuration reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/configuration-reference/).

See [here](/docs/api-reference/iglu/iglu-resolver/) for details on how to prepare the Iglu resolver file.

> **Tip:** All self-describing schemas for events processed by the transformer **must** be hosted on [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) 0.6.0 or above. [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) is a registry containing Snowplow-authored schemas. If you want to use them alongside your own, you will need to add it to your resolver file. Keep it mind that it could override your own private schemas if you give it higher priority.

## Running the Transformer Pubsub

The two config files need to be passed in as base64-encoded strings:

```bash
$ docker run snowplow/transformer-pubsub:6.3.0 \
--iglu-config $RESOLVER_BASE64 \
--config $CONFIG_BASE64
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Transformer PubSub (since version 5.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# RDB Loader 1.0.0 upgrade guide
> Upgrade RDB Loader to 1.0.0 with Stream Shredder, unified output partitioning, and new configuration schema for batch processing.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/1-0-0-upgrade-guide/

This is a release adding a new experimental Stream Shredder asset and improving independent Loader architecture, introduced in R35.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/1.0.0).

This is the first release in 1.x branch and no breaking changes will be introduced until 2.x release. If you're upgrading from R34 or earlier it's strictly recommended to follow [R35 Upgrade Guide](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r35-upgrade-guide/) first.

## Assets

RDB Loader, RDB Shredder and Stream RDB Shredder all have 1.o.0 version, despite last one being an experimental asset. RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-1.0.1.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:1.0.1`
- `snowplow/snowplow-rdb-stream-shredder:1.0.1`

## Configuration changes

All configuration changes are scoped to `shredder` property.

Since we added another type of a shredder, one has to specify the type explicitly:

```json

"shredder": {
  "type" : "batch",                                  # Was not necessary in R35
  "input": "s3://snowplow-enriched-archive/path/",   # Remains the same
  "output": ...                                      # Explained below
}
```

\_

The major API change in 1.0.0 is the new partitioning scheme unifying `good` and `bad` output. Whereas previously it was necessary to specify `output` and `outputBad`, now there's only `path` in `shredder.output` object:

\_

_`"output": { # Was a string in R35 "path": "s3://snowplow-shredded-archive/", # Path to shredded output "compression": "GZIP" # Output compression, GZIP or NONE`_ `}`

In Dataflow Runner playbook you have to specify new Main classpath for RDB Shredder:

```text
"--class", "com.snowplowanalytics.snowplow.rdbloader.shredder.batch.Main"
```

## Manifest

The new manifest table has the same name as previous one - `manifest`. In order to avoid a clash, RDB Loader 1.0.0 checks existence of the table every time it starts and if table exists checks if it's new or old one. If table exists and it's legacy - it will be renamed into `manifest_legacy` and can be removed manually later, new table will be created automatically. If table doesn't exist - it will be created.

No user actions necessary here.

## Stream Shredder

You only need to choose one Shredder: batch or stream. **For production environment we recommend using Batch Shredder.**

Stream Shredder is configured within same configuration file as RDB Loader and RDB Batch Shredder, but using following properties:

```json
  "shredder": {
    # A batch loader would fail, if stream type encountered
    "type" : "stream",
    # Input stream information
    "input": {
      # file is another option, but used for debugging only
      "type": "kinesis",
      # KCL app name - a DynamoDB table will be created with the same name
      "appName": "acme-rdb-shredder",
      # Kinesis Stream name, must exist
      "streamName": "enriched-events",
      # Kinesis region
      "region": "us-east-1",
      # Kinesis position: LATEST or TRIM_HORIZON
      "position": "LATEST"
    },

    # A frequency to emit loading finished message - 5,10,15,20,30,60 etc minutes, this is what controls how often your data will be loaded
    "windowing": "10 minutes",

    # Path to shredded archive, same as for batch
    "output": {
      # Path to shredded output
      "path": "s3://bucket/good/",
      # Shredder output compression, GZIP or NONE
      "compression": "GZIP"
    }
}
```

## Directory structure

There is a major change in shredder output directory structure, on top of what has changed in R35.

If you're using a 3rd-party query engine such as Amazon Athena to query shredded data, the new partitioning can break the schema. And thus it's recommended to create a new root for shredded data.

Structure of the typical shredded folder now looks like following:

```text
run=2021-03-29-15-40-30/
    shredding_complete.json
    output=good/
            vendor=com.snowplowanalytics.snowplow/
                name=atomic/
                    format=tsv/
                        model=1/
            vendor=com.acme/
                name=link_click/
                    format=json/
                        model=1/
    output=bad/
            vendor=com.snowplowanalytics.snowplow/
                name=loader_parsing_error/
                    format=json/
                        model=1/
```

---

# RDB Loader 1.2.0 upgrade guide
> Upgrade RDB Loader to 1.2.0 with webhook monitoring, folder monitoring, and enhanced observability for Redshift loading.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/1-2-0-upgrade-guide/

RDB Loader 1.2.0 brings many improvements for monitoring subsystem. If you're not interested in new features - you can just bump versions. If you need webhook monitoring - read below instructions on how to enable it.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/1.2.0).

## Assets

RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-1.2.0.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:1.2.0`
- `snowplow/snowplow-rdb-stream-shredder:1.2.0`

## Enabling Webhook monitoring

All configuration changes are scoped to `monitoring` property.

```json
"monitoring": {
  "webhook": {
    "endpoint": "https://webhooks.acme.com/rdb-loader",
    "tags": {            # Custom set of tags
      "host": $HOST,     # Environment variables are supported
      "pipeline": "production"
    }
  }
}
```

It's up to you to setup a preferable webhook backend. It can be a Snowplow Iglu webhook or custom monitoring system.

## Enabling folder monitoring

All configuration changes are scoped to `monitoring` property.

```json
"monitoring": {
  "folders": {
    "staging": "s3://snowplow-acme-com/logging/",    # This path will contain temporary files
                                                     # Redshift role must have an access for this folder
    "period": "2 hours"                              # How often the check should be performed
  }
}
```

It's up to you to setup a preferable webhook backend. It can be a Snowplow Iglu webhook or custom monitoring system.

---

# RDB Loader 2.0.0 upgrade guide
> Upgrade RDB Loader to 2.0.0 with separate configs for Loader and Shredder, SNS messaging, and split configuration architecture.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/2-0-0-upgrade-guide/

RDB Loader 2.0.0 brings ability to send shredding complete messages from Shredder to SNS topic and splits configs of RDB Loader and RDB Shredder. From now on, Loader and Shredder will not use same config.

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/2.0.0).

## Assets

RDB Shredder is published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-2.0.0.jar`

RDB Loader and RDB Stream Shredder distributed as Docker images, published on DockerHub:

- `snowplow/snowplow-rdb-loader:`2.0.0
- `snowplow/snowplow-rdb-stream-shredder:`2.0.0

## Sending shredding complete messages from Shredder to SNS

Shredding complete message can be sent to SNS topic with following queue configuration:

```json
"queue": {
  "type": "sns",
  "topicArn": "arn:aws:sns:eu-central-1:123456789:test-sns-topic",
  "region": "eu-central-1"
}
```

## New separate configs

RDB Loader and RDB Shredder were using the same config HOCON until version 2.0.0. Starting from 2.0.0, they will use separate configs. Reference docs for new configs can be found on the following pages:

[RDB Loader configuration](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/previous-versions/snowplow-rdb-loader/configuration-reference/)

[RDB Shredder configuration](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/previous-versions/snowplow-rdb-loader/rdb-shredder-configuration-reference/)

---

# RDB Loader 6.0.0 upgrade guide
> Upgrade RDB Loader to 6.0.0 with recovery tables, schema merging, and improved schema evolution for Redshift and other warehouses.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/6-0-0-upgrade-guide/

## License

Since version 6.0.0, RDB Loader is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run RDB Loader, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

## \[Redshift-only] New migration mechanism & recovery tables

### What is schema evolution?

One of Snowplow’s key features is the ability to [define custom schemas and validate events against them](/docs/fundamentals/schemas/). Over time, users often evolve the schemas, e.g. by adding new fields or changing existing fields. To accommodate these changes, RDB loader automatically adjusts the database tables in the warehouse accordingly.

There are two main types of schema changes:

**Breaking**: The schema version has to be changed in a model way (`1-2-3` → `2-0-0`). In Redshift, each model schema version has its own table (`..._1`, `..._2`, etc, for example: `com_snowplowanalytics_snowplow_ad_click_1`).

**Non-breaking**: The schema version can be changed in a addition way (`1-2-3` → `1-3-0` or `1-2-3` → `1-2-4`). Data is stored in the same database table.

### How it used to work

In the past, the transformer would fetch all schemas for an entity (which conforms to the `vendor/name/model-*-*` criterion of the entity) from Iglu Server, merge them, extract properties from JSON, stringify using tab char as delimeter and put null char in case the property is missing. Then the loader would adjust the database table and load the file.

This logic relied on two assumptions:

1. **Old events compatible with new schemas.** Events with older schema versions, e.g. `1-0-0` and `1-0-1`, had to be valid against the newer ones, e.g. `1-0-2`. Those that were not valid would result in failed events.

2. **Old columns compatible with new schemas.** The corresponding Redshift table had to be migrated correctly from one version to another. Changes, such as altering the type of a field from `integer` to `string`, would fail. Loading would break with SQL errors and the whole batch would be stuck and hard to recover.

These assumptions were not always clear to the users, making the transformer and loader error-prone.

### What happens now?

Transformer and loader are now more robust, and the data is easy to recover if the schema was not evolved correctly.

First, we support schema evolution that’s not strictly backwards compatible (although we still recommend against it since it can confuse downstream consumers of the data). This is done by _merging_ multiple schemas so that both old and new events can coexist. For example, suppose we have these two schemas:

```json
{
   // 1-0-0
   "properties": {
      "a": {"type": "integer"}
   }
}
```

```json
{
   // 1-0-1
   "properties": {
      "b": {"type": "integer"}
   }
}
```

These would be merged into the following:

```json
{
   // merged
   "properties": {
      "a": {"type": "integer"},
      "b": {"type": "integer"}
   }
}
```

Second, the loader does not fail when it can’t modify the database table to store both old and new events. (As a reminder, an example would be changing the type of a field from `integer` to `string`.) Instead, it creates a table for the new data as an exception. The users can then run SQL statements to resolve this situation as they see fit. For instance, consider these two schemas:

```json
{
   // 1-0-0
   "properties": {
      "a": {"type": "integer"}
   }
}
```

```json
{
   // 1-0-1
   "properties": {
      "a": {"type": "string"}
   }
}
```

Because `1-0-1` events cannot be loaded into the same table with `1-0-0`, the data would be put in a separate table, e.g. `com_snowplowanalytics_ad_click_1_0_1_recovered_9999999`, where:

- `1_0_1` is the version of the offending schema;
- `9999999` is a hash code unique to the schema (i.e. it will change if the schema is overwritten with a different one).

If you create a new schema `1-0-2` that reverts the offending changes and is again compatible with `1-0-0`, the data for events with that schema will be written to the original table as expected.

### Identifying schemas that need patching

After upgrading RDB Loader, you might find out that events or entities with some of the old schemas land in recovery tables.

To avoid this, the offending older Iglu schemas must be patched to align with the latest.

You can use the latest version of `igluctl` to do this.

To illustrate, let's say we have schema versions `1-0-0` and `1-0-1` that differ in one field only:

- version `1-0-0`: `{ "type": "integer", "maximum": 100 }` - translates to `SMALLINT`. All is good.
- version `1-0-1`: `{ "type": "integer", "maximum": 1000000 }` - breaking change as it translates to `INT` and the loader can't migrate the column.

Since version `1-0-1` had a breaking change, loading must have broken with an older version of RDB Loader. To fix that, the user must have updated the warehouse manually (changing the column type to `INT`). After this manual intervention, events with the `1-0-1` schema would have been loaded successfully with older versions of RDB Loader. However, after an upgrade to RDB Loader 6.0.0, events with the `1-0-1` schema will start to land in the recovery table.

Let's see how we can use `igluctl` to solve this problem.

1. Run the `igluctl static generate` command. If a recovery table is to be created, it will show up as a warning message. Example:

```bash
mkdir <schemas_folder> <sql_folder>
igluctl static pull <schemas_folder> <iglu_url> <iglu_key>
igluctl static generate <schemas_folder> <sql_folder>
# ...
# iglu:com.test/test/jsonschema/1-0-1 has a breaking change Incompatible types in column example_field old RedshiftSmallInt new RedshiftInteger
# ...
```

2. Run the `igluctl table-check` command to check if table structure is in line with the latest schema version that doesn't contain any breaking changes. With the example schemas above, this would be `1-0-0` because `1-0-1` contains a breaking change. Example:

```bash
igluctl table-check \
        --server <iglu_url> \
        --apikey <iglu_key> \
        --host <redshift_host> \
        --port <redshift_port> \
        --username <username> \
        --password <password> \
        --dbname  <database> \
        --dbschema <schema>
# ...
# * Column doesn't match, expected: 'example_field SMALLINT', actual: 'example_field INT'
# ...
```

We got the `Column doesn't match` output with the above example because the table column had been migrated manually from `SMALLINT` to `INT`. Since the latest schema version that doesn't contain any breaking change is `1-0-0`, `table-check` command expects to see `SMALLINT` in the table therefore it gives the `Column doesn't match` output.

In order to solve this problem, we should patch `1-0-0` with `{ "type": "integer", "maximum": 1000000 }`. In this case, there won't be any breaking change between versions `1-0-0` and `1-0-1`. RDB Loader 6.0.0+ will then successfully load events into the intended table as expected.

After identifying all the offending schemas, you should patch them to reflect the changes in the warehouse.

Schema casting rules could be found [here](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=redshift#types).

#### `$.featureFlags.disableRecovery` configuration

If you have older schemas with breaking changes and don’t want the loader to apply the new logic to them, you can use `$.featureFlags.disableRecovery` configuration. For the provided schema criterions only, RDB Loader will neither migrate the corresponding shredded table nor create recovery tables for breaking schema versions. Loader will attempt to load to the corresponding shredded table without migrating.

You can set it as follows:

```json
{
  ...
  "featureFlags": {
    "disableRecovery": [
      "iglu:com.example/myschema1/jsonschema/1-*-*",
      "iglu:com.example/myschema2/jsonschema/1-*-*"
    ]
  }
}
```

---

# Upgrade guides for the Snowplow RDB Loader
> Step-by-step upgrade guides for RDB Loader with breaking changes, schema migrations, and compatibility notes for Redshift, Snowflake, and Databricks.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/

This section contains information to help you upgrade to newer versions of Snowplow RDB Loader.

---

# RDB Loader R32 upgrade guide
> Upgrade RDB Loader to R32 with EMR 5.19.0, automigrations, and updated shredder and loader versions for Redshift.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r32-upgrade-guide/

We recommend to go through the upgrade routine in several independent steps. After every step you should have a working pipeline. If something is not working or Shredder produces unexpected failed events - please let us know.

## Updating assets

1. Upgrade EmrEtlRunner to R116 or higher

2. In your `redshift_config.json`

   1. Update SchemaVer to `4-0-0`
   2. Add `"blacklistTabular": null` field into `data` payload

3. Update your `config.yml` file

```yaml
aws:
  emr:
    ami_version: 5.19.0     # was 5.9.0; Required by RDB Shredder
storage:
  versions:
    rdb_loader: 0.17.0      # was 0.16.0
    rdb_shredder: 0.16.0    # was 0.15.0
```

At this point, your pipeline should be running with new assets as it was before, without automigrations and generated TSV. We recommend to test this setup and monitor shredded failed events for one or two runs before proceeding to enabling automigrations.

## Iglu Server

Automigrations work only with Iglu Server 0.6.0. This component provides information about how columns should be ordered across different ADDITIONs and REVISIOINs. If you still don't have Iglu Server 0.6.0 we recommend to [set it up](https://github.com/snowplow/iglu/wiki/Setting-up-an-Iglu-Server).

You still can use static registries as a backup, they will continue to work for validatioin purposes, but won't work for TSV shredding. Snowplow does not provide a public Iglu Server hosting Iglu Central schemas, so we recommend you to mirror Iglu Central with your own Iglu Server:

```bash
$ git clone https://github.com/snowplow/iglu-central.git
$ igluctl static push iglu-central/schemas $YOUR_SERVER_URL $YOU_API_KEY
$ igluctl static push com.acme-iglu-registry/schemas $YOUR_SERVER_URL $YOU_API_KEY
```

After setting up the Iglu Server, don't forget to add it to your resolver config.

## Tabular blacklisting

New RDB Shredder is still able to produce legacy JSON files. But automigrations can be applied only to tables where data is prepared as TSV. If you setup a new pipeline, you can generate only TSVs abandoning legacy DDLs (except `atomic.events` and `atomic.manifest`) and JSONPaths altogether. However, if you already have tables deployed which DDLs were generated manually or via old igluctl you will likely need to apply so called _tabular blacklisting_ to these tables. It means that Shredder will keep producing data with these schemas as JSONs and Loader won't be able to apply migrations to it. This is necessary because manually generated DDLs are not guaranteed to have predictable column order and the only way to map JSON values to respective columns is JSONPaths files.

[igluctl version 0.7.0](https://github.com/snowplow/igluctl/releases/tag/0.7.0) provides `rdbms table-check` subcommand that get schemas from Iglu Server, figures out what DDL the Loader would generate, then connects to Redshift and compares those DDLs with actual state of the table. Every table that have an incompatible order will have to be "blacklisted" in Redshift storage target config (`redshift_config.json`).

Here's an example of a black list containing several schemas from Iglu Central:

```json
  "blacklistTabular": [
    "iglu:org.w3/PerformanceTiming/jsonschema/1-*-*",

    "iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",
    "iglu:com.snowplowanalytics.snowplow/application_error/jsonschema/1-*-*",

    "iglu:com.mandrill/recipient_unsubscribed/jsonschema/1-*-*",
    "iglu:com.mandrill/message_soft_bounced/jsonschema/1-*-*",
    "iglu:com.mandrill/message_sent/jsonschema/1-*-*",
    "iglu:com.mandrill/message_opened/jsonschema/1-*-*",
    "iglu:com.mandrill/message_marked_as_spam/jsonschema/1-*-*",
    "iglu:com.mandrill/message_delayed/jsonschema/1-*-*",
    "iglu:com.mandrill/message_clicked/jsonschema/1-*-*",
    "iglu:com.mandrill/message_bounced/jsonschema/1-*-*"
  ]
```

As you can see, schemas specified in schema criterion format (with wildcards everywhere except MODEL).

## Conclusion

At this point if you track an event with new schema and this schema resides on an Iglu Server - RDB Shredder will produce TSV data for it and RDB Loader will automatically create a new table. Same with ADDITION and REVISION migrations - they're handled by RDB Loader automatically.

---

# RDB Loader R33/R34 upgrade guide
> Upgrade RDB Loader to R33/R34 with Spark 3, EMR 6.1.0, and bugfixes for long text properties in Redshift.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r33-upgrade-guide/

R34 is a release with bugfixes and performance improvements. R33 was almost identical reelase with major bug preventing some long text properties from loading.

## Updating assets

1. Upgrade EmrEtlRunner to 1.0.4 or higher
2. Your `redshift_config.json` should have 4-0-0 version
3. Update your `config.yml` file

```yaml
aws:
  emr:
    ami_version: 6.1.0     # was 5.19.0; Required by Spark 3
storage:
  versions:
    rdb_loader: 0.18.2      # was 0.17.0
    rdb_shredder: 0.18.2    # was 0.16.0
```

---

# RDB Loader R35 upgrade guide
> Upgrade RDB Loader to R35 with independent Loader architecture, SQS messaging, and removal of EmrEtlRunner dependency.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/r35-upgrade-guide/

R35 is a release with major changes in pipeline architecture:

- No dependency on EmrEtlRunner (neither Shredder nor Loader can be launched using EmrEtlRunner, marking deprecation of EmrEtlRunner)
- Loader is not an EMR step anymore
- Major changes in directory structure
- New dependency on SQS

[Release notes](https://github.com/snowplow/snowplow-rdb-loader/releases/tag/r35).

This is the last release in 0.x branch and breaking changes still might be introduced in 1.0.0 release.

## Assets

Both RDB Shredder and Loader have 0.19.0 version. Both published on S3:

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-0.19.0.jar`

- `s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-loader/snowplow-rdb-loader-0.19.0.jar`

For RDB Loader however it is recommended to use the docker image, published on DockerHub: `snowplow/snowplow-rdb-loader:0.19.0`

## New architecture

Previous workflow was orchestrated by EmrEtlRunner, along with multiple S3DistCp steps, recovery scenarios and dedicated RDB Loader step. RDB Loader was finding out what data needs to be loaded by scanning S3.

In the new architecture there are two EMR steps:

1. S3DistCp, copying enriched data sunk by [S3 Loader](/docs/api-reference/loaders-storage-targets/s3-loader/), from S3 sink bucket (same as "enriched stream bucket") into _enriched data lake_ (aka shredder input, similar as previously known "enriched archive")
2. RDB Shredder, picking up all _unprocessed_ folders in enriched data lake, shredding data there and writing it into _shredded data lake_ (previously known as "shredded archive")

RDB Loader is a stand-alone long-running app, launched either on EC2 box or Fargate cluster. The loading gets triggered by an SQS message, sent by Shredder after it finished processing a new batch.

RDB Shredder decides that folder is unprocessed by:

1. Comparing folder names in _enriched data lake_ and in _shredded data lake_. Every folder that is **in** enriched, but **not in** shredded will be considered unprocessed
2. ...except folders that don't have `shredding_complete.json` file in their root. This file is written at the end of the job and indicates that job has completed successfully. Absence of this file means that shred job has been aborted.

If you're upgrading from R34 or earlier it is strictly recommended to pick new paths for enriched and shredded archives in order to avoid double-loading OR make sure that there's a strict 1:1 correspondence between content of enriched and shredded archive.

We recommend to use either [Dataflow Runner](/docs/api-reference/dataflow-runner/) or boto3 script to launch scheduled S3DistCp and Shredder jobs. Here's an example of a Dataflow Runner playbook:

```json
{
  "schema": "iglu:com.snowplowanalytics.dataflowrunner/PlaybookConfig/avro/1-0-1",
  "data": {
    "region": "eu-central-1",
    "credentials": {
      "accessKeyId": "env",
      "secretAccessKey": "env"
    },
    "steps": [
      {
        "type": "CUSTOM_JAR",
        "name": "S3DistCp enriched data archiving",
        "actionOnFailure": "CANCEL_AND_WAIT",
        "jar": "/usr/share/aws/emr/s3-dist-cp/lib/s3-dist-cp.jar",
        "arguments": [
            "--src", "s3://com-acme/enriched/sink/",
            "--dest", "s3://com-acme/enriched/archive/run={{nowWithFormat "2006-01-02-15-04-05"}}/",
            "--s3Endpoint", "s3-eu-central-1.amazonaws.com",
            "--srcPattern", ".*",
            "--outputCodec", "gz",
            "--deleteOnSuccess"
        ]
      },

      {
        "type": "CUSTOM_JAR",
        "name": "RDB Shredder",
        "actionOnFailure": "CANCEL_AND_WAIT",
        "jar": "command-runner.jar",
        "arguments": [
            "spark-submit",
            "--class", "com.snowplowanalytics.snowplow.shredder.Main",
            "--master", "yarn",
            "--deploy-mode", "cluster",
            "s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-0.19.0.jar",
            "--iglu-config", "{{base64File "/home/snowplow/configs/snowplow/iglu_resolver.json"}}",
            "--config", "{{base64File "/home/snowplow/configs/snowplow/config.hocon"}}"
        ]
      }
    ],
    "tags": [ ]
  }
}
```

We recommend to launch RDB Loader as long-running docker image.

## New configuration file

Common configuration file, previously known as `config.yml` and target JSON configuration file, previously known as `redshift.json` have been replaced by a [single HOCON file](https://github.com/snowplow/snowplow-rdb-loader/blob/master/config/config.hocon.sample).

Here's an example:

```json
{
  # Human-readable identificator, can be random
  "name": "Acme Redshift",
  # Machine-readable unique identificator, must be UUID
  "id": "123e4567-e89b-12d3-a456-426655440000",

  # Data Lake (S3) region
  "region": "us-east-1",
  # SQS topic name used by Shredder and Loader to communicate
  "messageQueue": "messages.fifo",

  # Shredder-specific configs
  "shredder": {
    # Path to enriched archive (must be populated separately with run=YYYY-MM-DD-hh-mm-ss directories)
    "input": "s3://com-acme/ernched/archive/",
    # Path to shredded output
    "output": "s3://com-acme/shredded/good/",
    # Path to data failed being processed
    "outputBad": "s3://com-acje/shredded/bad/",
    # Shredder output compression, GZIP or NONE
    "compression": "GZIP"
  },

  # Optional. S3 path that holds JSONPaths
  "jsonpaths": "s3://bucket/jsonpaths/",

  # Schema-specific format settings (recommended to leave all three groups empty and use TSV as default)
  # To make it compatible with R34, leave default = TSV and populate json array with things from blacklistTabular
  "formats": {
    # Format used by default (TSV or JSON)
    "default": "TSV",
    # Schemas to be shredded as JSONs, corresponding JSONPath files must be present. Automigrations will be disabled
    "json": [ ],
    # Schemas to be shredded as TSVs, presence of the schema on Iglu Server is necessary. Automigartions enabled
    "tsv": [ ],
    # Schemas that won't be loaded
    "skip": [ ]
  },

  # Warehouse connection details, identical to storage target config
  "storage": {
    # Database, redshift is the only acceptable option
    "type": "redshift",
    # Redshift hostname
    "host": "redshift.amazon.com",
    # Database name
    "database": "snowplow",
    # Database port
    "port": 5439,
    # AWS Role ARN allowing Redshift to load data from S3
    "roleArn": "arn:aws:iam::123456789012:role/RedshiftLoadRole",
    # DB schema name
    "schema": "atomic",
    # DB user with permissions to load data
    "username": "storage-loader",
    # DB password
    "password": "secret",
    # Custom JDBC configuration
    "jdbc": {"ssl": true},
    # MAXERROR, amount of acceptable loading errors
    "maxError": 10,
    "compRows": 100000
  },

  # Additional steps. analyze, vacuum and transit_load are valid values
  "steps": ["analyze"],

  # Observability and logging opitons
  "monitoring": {
    # Snowplow tracking (optional)
    "snowplow": null,
    # Sentry (optional)
    "sentry": null
  }
}
```

If you need to use cross-batch deduplication - the file format remains the same for DynamoDB config.

CLI arguments also have changed. Both applications now accept only `--iglu-config` with base64-encoded string representing [Iglu Resolver JSON](/docs/api-reference/iglu/iglu-resolver/) and `--config` with base64-encoded above HOCON. Loader also accepts `--dry-run` flag.

## SQS

SQS serves as message bus between Shredder and Loader. Loader expects to find there self-describing messages with instructions on what to load. The queue must be FIFO.

## Directory structure

There are several major changes in shredder output directory structure:

1. Elements of the paths have changed from Iglu-compatible to shredder-specific, e.g. `format` now can be either `json` or `tsv` (and not `jsonschema` as before) and instead of `version` (that could have been either `1-0-0` or just `1`) it is always just `model`
2. There's no dedicated `atomic-events` folder. It is replaced with unified `vendor=com.snowplowanalytics.snowplow/name=atomic/format=tsv/model=1`
3. There are no `shredded-types` or `shredded-tsv` either, all types are in the root of the folder.

Structure of the typical shredded folder now looks like following:

```text
run=2021-01-27-18-35-00/
   vendor=com.snowplowanalytics.snowplow/
     name=atomic/
       format=tsv/
         model=1/
   vendor=com.snowplowanalytics.snowplow/
     name=ad_click/
       format=json/
         model=1/
   vendor=nj.basjes/
     name=yauaa_context/
       format=tsv/
         model=1/
   shredding_complete.json
   _SUCCESS
```

## Caution

We consider this version a public beta. Although it has been carefully tested in sandbox environments showing significantly decreased AWS costs on associated infrastructure, it still haven't been used in production.

One known issue in this version is absence of protection against double-loading. If Loader receives the same SQS message multiple time (i.e. sent manually) - the same batch will be loaded multiple times.

We also reserve right to make other breaking API changes in next versions.

---

# Snowbridge upgrade guide
> Upgrade Snowbridge to version 3.X.X with configuration changes for transformations, new features, and breaking changes.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/X-X-upgrade-guide/

## Version 4.0.0 Breaking Changes

### HTTP target: ordered response rule evaluation

**Breaking change**: response rules are now evaluated in the order they are defined in the configuration, rather than being organized in separate `invalid` and `setup` blocks.

**Migration required**: you must update your HTTP target configuration to specify a `type` attribute for each rule:

**Before:**

```hcl
response_rules {
  invalid {
    http_codes = [400]
    body = "Invalid value for 'purchase' field"
  }
  setup {
    http_codes = [401, 403]
  }
}
```

**After (4.0.0):**

```hcl
response_rules {
  rule {
    type = "invalid"
    http_codes = [400]
    body = "Invalid value for 'purchase' field"
  }
  rule {
    type = "setup"
    http_codes = [401, 403]
  }
}
```

**Important**: rules are now evaluated in the order they appear in your configuration. The first matching rule determines the error type.

## Version 3.0.0 Breaking Changes

The below breaking changes were made in version 3.0.0. All other functionality is backwards compatible.

### Lua support removed

Support for Lua transformations has been removed. If you are running a Lua transformation, you can port the logic to [Javascript](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) or [JQ](/docs/api-reference/snowbridge/configuration/transformations/builtin/jq/).

### HTTP target: non-JSON data no longer supported

We never intended to support non-JSON data, but prior to version 3.0.0, the request body was simply populated with whatever bytes were found in the message data, regardless of whether it is valid JSON.

From version 3.0.0 onwards, only valid JSON will work, otherwise the message will be considered invalid and sent to the failure target.

### HTTP target: request batching

Many HTTP APIs allow sending several events in a single request by putting them into a JSON array. Since version 3.0.0, if the Snowbridge source provides data in batches, the HTTP target will batch events in this way.

As a consequence, even when the source provides events in a single event batch, it will now be placed into an array of one element. For example, prior to version 3.0.0, a request body might look like this:

```text
{"foo": "bar"}
```

But it will now look like this:

```text
[{"foo": "bar"}]
```

As of version 3.0.0, the SQS source provides events in batches of up to ten, and the Kinesis, Kafka, and Pubsub and Stdin sources provide events in single-event batches. This behavior will likely change in a future version.

You can preserve the previous behavior and ensure that requests are always single-event non-array objects, even with a batching source. To do so, set `request_max_messages` to 1, and provide this template (as long as your data is valid JSON):

```go
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-unwrap-example.file)

---

# Snowbridge batching model
> Understand how Snowbridge handles message batching across sources and targets for optimal throughput and performance.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/batching-model/

Messages are processed in batches according to how the source provides data. The Kinesis and Pubsub sources provide data in message-by-message, data is handled in batches of 1 message. The SQS source is batched according to how the SQS queue returns messages.

Transformations always handle individual messages at a time.

If the source provides the data in batch, the Kinesis, SQS, EventHub and Kafka targets can chunk the data into smaller batches before sending the requests. The EventHub target can further batch the data according to partitionKey, if set - which is a feature of the EventHub client specifically. The Pubsub and HTTP targets handle messages individually at present.

---

# Snowbridge failure model
> Learn how Snowbridge handles target failures, oversized data, invalid data, transformation failures, and fatal errors.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/failure-model/

## Failure targets

When Snowbridge hits an unrecoverable error — for example [oversized](#oversized-data) or [invalid](#invalid-data) data — it will emit a [failed event](/docs/fundamentals/failed-events/) to the configured failure target. A failure target is the same as a target, the only difference is that the configured destination will receive failed events.

You can find more detail on setting up a failure target, in the [configuration section](/docs/api-reference/snowbridge/configuration/targets/).

There are several different failures that Snowbridge may hit.

## Target failure

This is where a request to the destination technology fails or is rejected - for example a HTTP 400 response is received.

Retry behavior for target failures is determined by the retry configuration. You can find details of this in the [configuration section](/docs/api-reference/snowbridge/configuration/retries/).

As of Snowbridge 2.4.2, the Kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.

Before version 3.0.0, Snowbridge treated every kind of target failure the same - it would retry 5 times. If all 5 attempts failed, it would proceed without acking the failed Messages. As long as the source's acking model allows for it, these would be re-processed through Snowbridge again.

Each target failure attempt will be reported as a 'MsgFailed' for monitoring purposes.

## Oversized data

Targets have limits to the size of a single message. Where the destination technology has a hard limit, targets are hardcoded to that limit. Otherwise, this is a configurable option in the target configuration. When a message's data is above this limit, Snowbridge will produce a [size violation failed event](/docs/api-reference/failed-events/#size-violation), and emit it to the failure target.

Writes of oversized messages to the failure target will be recorded with 'OversizedMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).

## Invalid data

In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/api-reference/failed-events/#generic-error), emit it to the failure target, and ack the original message.

As of version 3.0.0, the HTTP target may produce 'invalid' type failures. This occurs when: the a POST request body cannot be formed; the templating feature's attempts to template data result in an error; or the response conforms to a response rules configuration which specifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/api-reference/snowbridge/configuration/targets/http/).

Transformation failures are also treated as invalid, as described below.

Writes of invalid messages to the failure target will be recorded with 'InvalidMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).

## Transformation failure

Where a transformation hits an exception, Snowbridge will consider it invalid, assuming that the configured transformation cannot process the data. It will create a [generic error failed event](/docs/api-reference/failed-events/#generic-error), emit it to the failure target, and ack the original message.

As long as the built-in transformations are configured correctly, this should be unlikely. For scripting transformations, Snowbridge assumes that an exception means the data cannot be processed - make sure to construct and test your scripts accordingly.

Writes of invalid messages to the failure target will be recorded with 'InvalidMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).

## Fatal failure

Snowbridge is built to be averse to crashes, but there are two scenarios where it would be expected to crash.

Firstly, if it hits an error in retrieving data from the source stream, it will log an error and crash. If this occurs it is normally a case of misconfiguration of the source. If that is not the case, it will be safe to redeploy the app — it will attempt to begin from the first unacked message. This may cause duplicates.

Secondly, as described above, where there are failures it will attempt to reprocess the data if it can, and where failures aren't recoverable it will attempt to handle that via a failure target. Normally, even reaching this point is rare.

In the very unlikely event that Snowbridge reaches this point and cannot write to a failure target, the app will crash. Should this happen, and the app is re-deployed, it will begin processing data from the last acked message. Note that the likely impact of this is duplicated sends to the target, but not data loss.

Of course, if you experience crashes or other issues that are not explained by the above, please log an issue detailing the behavior.

---

# Snowbridge core concepts
> Understand Snowbridge architecture including sources, transformations, targets, batching, failure handling, and scaling strategies.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/

Snowbridge’s architecture is fairly simple: it receives data from one streaming technology (via [Sources](/docs/api-reference/snowbridge/concepts/sources/)), optionally runs filtering and transformation logic on them (message-by-message, via [Transformations](/docs/api-reference/snowbridge/concepts/transformations/)), and sends the data to another streaming technology or destination (via [Targets](/docs/api-reference/snowbridge/concepts/targets/)). If it is not possible to process or retry the data [as per the failure model](/docs/api-reference/snowbridge/concepts/failure-model/), it outputs a message to another destination (via [Failure Targets](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets)).

Where the source supports acking, Snowbridge only acks messages once the data is successfully sent to either the target or the failure target (in the case of unrecoverable failure). In the case of a recoverable failure — for example when the target is temporarily unavailable — Snowbridge will not ack the messages and will retry them once the source technology’s ack deadline has passed.

![architecture](/assets/images/snowbridge-architecture-12bf56119494c1ca221081e869411e65.jpg)

## Operational details

Data is processed on an at-least-once basis, and there is no guarantee of order of messages. The application is designed to minimise duplicates as much as possible, but there isn’t a guarantee of avoiding them — for example if there’s a failure, it is possible for messages to be delivered without a successful response, and duplicates can occur.

---

# Scaling Snowbridge horizontally
> Scale Snowbridge horizontally across multiple instances with concurrency controls and target provisioning for optimal throughput.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/scaling/

Snowbridge is built to suit a **horizontal scaling** model, and you can safely deploy multiple instances of Snowbridge to consume the same input out-of-the-box. No addditional configuration or setup is required for the app to smoothly run across multiple instances/environments, compared to a single instance/environment.

> **Note:** If you are using the Kinesis source, you will need to manually create a few DynamoDB tables as described in [the Kinesis source configuration section](/docs/api-reference/snowbridge/configuration/sources/kinesis/). Snowbridge uses these tables to coordinate multiple instances consuming from the same stream.

How to configure scaling behavior will depend on the infrastructure you’re using, and the use case you have implemented. For example, if you choose to scale based on CPU usage, note that this metric will be affected by the size and shape of the data, by the transformations and filters used, and for script transformations, by the content of the scripts.

> **Tip:** Occasionally, new releases of Snowbridge will improve its efficiency. In the past, this has had a large impact on metrics typically used for scaling. To ensure that scaling behaves as expected, we recommend monitoring your metrics after you upgrade Snowbridge or change the transformation configuration.

In addition to configuring the number of Snowbridge instances, you can manage concurrency via the `concurrent_writes` setting (explained in the [next section](#concurrency)). This setting provides a degree of control over throughput and resource usage. Snowbridge should consume as much data as possible, as fast as possible — a backlog of data or spike in traffic should cause the app’s CPU usage to increase significantly. If spikes/backlogs do not induce this behavior, and there are no target retries or failures (see below), then you can increase the `concurrent_writes`.

## Concurrency

Snowbridge is a Go application, which makes use of [goroutines](https://golangdocs.com/goroutines-in-golang). You can think of goroutines as lightweight threads. The source’s `concurrent_writes` setting controls how many goroutines may be processing data at once, in a given instance of the app (others may exist separately, under the hood for non-data processing purposes).

You can determine the total maximum concurrency for the entire application by multiplying `concurrent_writes` by the number of horizontal instances of the app. For example, if Snowbridge is deployed via kubernetes pods, and there are 4 active pods with `concurrent_writes` set to 150, then at any given time there will be up to 600 concurrent goroutines that can process and send data.

## Target scaling

Snowbridge will attempt to send data to the target as fast as resources will allow, so we recommend that you set up the target to scale sufficiently with the expected volume and throughput. Note that in case of failure, Snowbridge will retry sending the messages with an exponential backoff, staring with a 1s delay between retries, and doubling that delay for 5 retries.

If a backlog of data builds up due to some failure — for example target downtime — then we advise to overprovision the target until the backlog is processed. That’s only required until latency falls back to normal rates.

---

# Introduction to Snowbridge sources
> Configure Snowbridge sources to retrieve data from streams and forward them for processing with acking and concurrency controls.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/sources/

Sources deal with retrieving data from the input stream, and forwarding it for processing — once messages are either filtered or successfully sent, they are then acked (if the source technology supports acking). Otherwise, messages will be retrieved again by the source. Sources also have a setting which controls concurrency for the instance — `concurrent_writes`.

You can find more detail on setting up a source, in the [configuration section](/docs/api-reference/snowbridge/configuration/sources/).

---

# Introduction to Snowbridge targets
> Configure Snowbridge targets to validate, batch, and send data to destination streams with size and validity checks.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/targets/

Targets check for [validity and size restrictions](/docs/api-reference/snowbridge/concepts/failure-model/), [batch data](/docs/api-reference/snowbridge/concepts/batching-model/) where appropriate and send data to the destination stream.

You can find more detail on setting up a target, in the [configuration section](/docs/api-reference/snowbridge/configuration/targets/).

---

# Introduction to Snowbridge transformations and filters
> Transform and filter messages with built-in transformations for Snowplow data or custom JavaScript scripts for flexible data processing.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/concepts/transformations/

Transformations allow you to modify messages' data on the fly before they're sent to the destination. There are a set of built-in transformations, specifically for use with Snowplow data (for example transforming Snowplow enriched events to JSON), You can also configure a script to transform your data however you require - for example if you need to rename fields or change a field's format.

It's also possible to exclude messages (ie. not send them to the target) based on a condition, by configuring a special type of transformation called a filter. (Technically then, filters are transformations, but we sometimes refer to them as a separate concept for clarity). Again there are built-in filters to apply to Snowplow data, or you can provide a script to filter the data.

Transformations operate on a per-message basis, are chained together in the order configured, and the same type of transformation may be configured more than once. We recommend to place filters first for performance reasons. When transformations are chained together, the output of the first is the input of the second, however transformations may not depend on each other in any other way.

As an example of how transformations relate to each other - if you have a built-in filter with condition A, and a filter with condition B, you can arrange them one after another, so that the data must satisfy A AND B. But you can't arrange them to satisfy A OR B - because the outcome of each must be determined on their own.

The latter use case, and further nuanced use cases can, however, be achieved using scripting transformation (in the case of the latter example, a single script can perform both checks with an OR condition).

## Custom scripting transformations

Custom scripting transformations allow you to provide a script to transform the data, set the destination's partition key, or filter the data according to your own logic. For scripting, you can use Javascript. Snowbridge uses a runtime engine to run the script against the data. Scipts interface with the rest of the app via the `EngineProtocol` interface, which provides a means to pass data into the scripting layer, and return data from the scripting layer back to the app.

You can find more detail on setting up custom scripts [in the configuration section](/docs/api-reference/snowbridge/configuration/transformations/).

---

# Snowbridge configuration overview
> Configure Snowbridge using HCL format to define sources, transformations, targets, and monitoring for stream replication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/

Snowbridge is configured using [HCL](https://github.com/hashicorp/hcl). To configure Snowbridge, create your configuration in a file with `.hcl` extension, and set the `SNOWBRIDGE_CONFIG_FILE` environment variable to the path to your file. By default, the Snowbridge docker image uses `/tmp/config.hcl` as the config path - when using the docker images you can either mount your config file to `/tmp/config.hcl`, or mount it to a different path, and set the `SNOWBRIDGE_CONFIG_FILE` environment variable in your docker container to that path.

Inside the configuration, you can reference environment variables using the `env` object. For example, to refer to an environment variable named `MY_ENV_VAR` in your configuration, you can use `env.MY_ENV_VAR`. We recommend employing environment variables for any sensitive value, such as a password, as opposed to adding the value to the configuration verbatim.

For most options, Snowbridge uses blocks for configuration. The `use` keyword specifies what you'd like to configure - for example a kinesis source is configured using `source { use "kinesis" {...}}`.

For all configuration blocks except for transformations, you must provide only one block (or none, to use the defaults).

For transformations, you may provide 0 or more `transform` configuration blocks. They will be applied to the data, one after another, in the order they appear in the configuration. The exception to this is when a filter is applied and the filter condition is met - in this case the message will be acked and subsequent transformations will not be applied (neither will the data be sent to the destination).

Some application-level options are not contained in a block, instead they're top-level options in the configuration. For example, to set the log level of the application, we just set the top-level variable `log_level`.

If you do not provide a configuration, or provide an empty one, the application will use the defaults:

- `stdin` source;
- no transformations;
- `stdout` target;
- `stdout` failure target. There'll be no external statistics reporting or sentry error reporting.

## License

Since version 2.4.0, Snowbridge is released under the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run Snowbridge, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `license.accept` option, like this:

```hcl
license {
  accept = true
}
```

## Example configuration

The below example is a complete configuration, which specifies a kinesis source, a builtin Snowplow filter (which may only be used if the input is Snowplow enriched data), a custom javascript transformation, and a pubsub target, as well as the statsD stats receiver, and sentry for error reporting.

In layman's terms, this configuration will read data from a kinesis stream, filter out any data whose `event_name` field is not `page_view`, run a custom Javascript script upon the data to change the `app_id` to `"1"`, and send the transformed page view data to pubsub. It will also send statistics about what it's doing to a statsD endpoint, and will send information about errors to a sentry endpoint.

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/overview-full-example.hcl)

---

# Snowbridge monitoring configuration
> Monitor Snowbridge with configurable logging, pprof profiling, StatsD metrics, and Sentry error reporting for observability.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/monitoring/

Snowbridge comes with configurable logging, [pprof](https://github.com/google/pprof) profiling, [statsD](https://www.datadoghq.com/statsd-monitoring) statistics and [Sentry](https://sentry.io/welcome/) integrations to ensure that you know what's going on.

## Logging

Use the `log_level` parameter to specify the log level.

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/monitoring/log-level-example.hcl)

## Sentry Configuration

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/monitoring/sentry-example.hcl)

## StatsD stats receiver configuration

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/monitoring/statsd-example.hcl)

## End-to-end latency configuration

Snowplow Enriched data only:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/metrics/e2e-latency-example.hcl)

## Metric definitions

Snowbridge sends the following metrics to statsd:

| Metric                   | Definitions                                                                                                                                              |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `target_success`         | Events successfully sent to the target.                                                                                                                  |
| `target_failed`          | Events which failed to reach the target, and will be handled by the retry config. Retries which fail are also counted.                                   |
| `message_filtered`       | Events filtered out via transformation.                                                                                                                  |
| `failure_target_success` | Events we could not send to the target, which are not retryable, successfully sent to the failure target.                                                |
| `failure_target_failed`  | Events we could not send to the target, which are not retryable, which we failed to send to the failure target. In this scenario, Snowbridge will crash. |
| `min_processing_latency` | Min time between entering Snowbridge and write to target.                                                                                                |
| `max_processing_latency` | Max time between entering Snowbridge and write to target.                                                                                                |
| `min_message_latency`    | Min time between entering the source stream and write to target.                                                                                         |
| `max_message_latency`    | Max time between entering the source stream and write to target.                                                                                         |
| `min_transform_latency`  | Min time between start and completion of transformation.                                                                                                 |
| `max_transform_latency`  | Max time between start and completion of transformation.                                                                                                 |
| `min_filter_latency`     | Min time between entering Snowbridge and being filtered out.                                                                                             |
| `max_filter_latency`     | Max time between entering Snowbridge and being filtered out.                                                                                             |
| `min_request_latency`    | Min time between starting request to target and finishing request to target.                                                                             |
| `max_request_latency`    | Max time between starting request to target and finishing request to target.                                                                             |
| `sum_request_latency`    | Sum of request times, use with `target_request_count` to calculate average request latency.                                                              |
| `target_request_count`   | Number of requests sent to target, use with `sum_request_latency` to calculate average request latency.                                                  |
| `min_e2e_latency`        | Min time between Snowplow collector tstamp and finishing request to target. Enabled via configuration - Snowplow enriched data only.                     |
| `max_e2e_latency`        | Max time between Snowplow collector tstamp and finishing request to target. Enabled via configuration - Snowplow enriched data only.                     |

---

# Snowbridge retry behavior configuration (beta)
> Configure retry behavior for transient and setup failures with exponential backoff and configurable max attempts.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/retries/

> **Note:** This feature was added in version 3.0.0
>
> This feature is in beta status because we may make breaking changes in future versions.

This feature allows you to configure the retry behavior when the target encounters a failure in sending the data. There are three types of failure you can define:

A **transient failure** is a failure which we expect to succeed again on retry. For example, some temporary network error. Typically, you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everything is operating as normal. The failed data is retried after a backoff.

A **setup failure** is one we don't expect to be immediately resolved, for example an incorrect address, or an invalid API key. Typically, you would configure a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (e.g. permissions need to be granted). Setup errors will be retried up to the configured `max_attempts` before the app crashes.

A **throttle failure** (added in version 4.0.0) is a special type of failure that indicates the target is rate limiting requests. This is handled separately from transient errors to allow different retry behavior - typically with longer delays to respect rate limits.

As of version 3.0.0, only the http target can be configured to return setup and throttle errors, via response rules - see [the http target configuration section](/docs/api-reference/snowbridge/configuration/targets/http/). For all other targets, all errors returned will be considered transient, and behavior can be configured using the `transient` block of the retry configuration.

Retries will be attempted with an exponential backoff. In other words, on each subsequent failure, the backoff time will double. You can configure transient failures to be retried indefinitely by setting `max_attempts` to 0.

As of version 4.0.0, you can configure transient failures to be sent to the failure target after reaching `max_attempts` by setting `invalid_after_max` to `true`.

## Configuration options

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/retry-example.hcl)

---

# Configure HTTP as a Snowbridge source
> Configure HTTP source for Snowplow Snowbridge to receive data over HTTP endpoints for experimental stream ingestion.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/http/

> **Note:** This source was added in version 3.6.2
>
> This source is experimental and not recommended for production use.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/http-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/http-full-example.hcl)

---

# Snowbridge source configuration
> Configure Snowbridge sources including stdin, Kafka, Kinesis, PubSub, SQS, and HTTP for stream ingestion.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/

**Stdin source** is the default. We also support Kafka, Kinesis, PubSub, SQS and experimental HTTP sources.

Stdin source simply treats stdin as the input. It has one optional configuration to set the concurrency.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/stdin-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/stdin-full-example.hcl)

---

# Configure Kafka as a Snowbridge source
> Configure Kafka source for Snowplow Snowbridge to read data from Kafka topics with authentication and consumer group settings.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/kafka/

Authentication is done by providing valid credentials in the configuration.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/kafka-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/kafka-full-example.hcl)

---

# Configure Kinesis as a Snowbridge source
> Configure Kinesis source for Snowplow Snowbridge to read from AWS Kinesis streams with DynamoDB checkpointing and authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/kinesis/

> **Note:** To use this source, you need the AWS-specific version of Snowbridge that can only be run on AWS. See [the page on Snowbridge distributions](/docs/api-reference/snowbridge/getting-started/) for more information.

## Authentication

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Setup

The AWS kinesis source requires the additional setup of a set of dynamoDB tables for checkpointing purposes. To set up a kinesis source, you will need to:

1. Configure the above required variables in the HCL file.
2. Create three DynamoDB tables which will be used for checkpointing the progress of the replicator on the stream (_Note_: details below)

Under the hood we are using a fork of the [Kinsumer](https://github.com/snowplow-devops/kinsumer) library which has defined this DynamoDB table structure - these tables need to be created by hand before the application can launch.

| TableName                                | DistKey        |
| ---------------------------------------- | -------------- |
| `${SOURCE_KINESIS_APP_NAME}_clients`     | ID (String)    |
| `${SOURCE_KINESIS_APP_NAME}_checkpoints` | Shard (String) |
| `${SOURCE_KINESIS_APP_NAME}_metadata`    | Key (String)   |

Assuming your AWS credentials have sufficient permission for Kinesis and DynamoDB, your consumer should now be able to run when you launch the executable.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/kinesis-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/kinesis-full-example.hcl)

---

# Configure Pub/Sub as a Snowbridge source
> Configure PubSub source for Snowplow Snowbridge to read from GCP Pub/Sub subscriptions with service account authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/pubsub/

Authentication is done using a [GCP Service Account](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa). Create a service account credentials file, and provide the path to it via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.

Snowbridge connects to PubSub using [Google's Go Pubsub sdk](https://cloud.google.com/go/pubsub), which establishes a grpc connection with TLS encryption.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/pubsub-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/pubsub-full-example.hcl)

---

# Configure SQS as a Snowbridge source
> Configure SQS source for Snowplow Snowbridge to read from AWS SQS queues with IAM authentication and message handling.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/sources/sqs/

Read data from an SQS queue.

## Authentication

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/sqs-minimal-example.hcl)

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/sources/sqs-full-example.hcl)

---

# Configure EventHub as a Snowbridge target
> Configure EventHub target for Snowplow Snowbridge to write data to Azure Event Hubs with namespace and environment authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/eventhub/

Authentication for the EventHub target is done by configuring any valid combination of the environment variables [listed in the Azure Event Hubs Client documentation](https://pkg.go.dev/github.com/Azure/azure-event-hubs-go#NewHubWithNamespaceNameAndEnvironment).

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/eventhub-minimal-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/eventhub-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure HTTP as a Snowbridge target
> Configure HTTP target for Snowplow Snowbridge to send data over HTTP with authentication, OAuth2, request templating, and response rules.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/http/

> **Note:** Version 3.0.0 makes breaking changes to the HTTP target. Details on migrating can be found [in the migration guide](/docs/api-reference/snowbridge/X-X-upgrade-guide/)

## Basic authentication

Where basicauth is used, it may be configured using the `basic_auth_username` and `basic_auth_password` options. Where an authorization header is used, it may be set via the `headers` option.

We recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).

TLS may be configured by providing the `key_file`, `cert_file` and `ca_file` options with paths to the relevant TLS files.

## OAuth2

Snowbridge supports sending authorized requests to OAuth2-compliant HTTP targets. This can be enabled by setting `oauth2_client_id`, `oauth2_client_secret`, `oauth2_refresh_token` (these three are long-lived credentials used to generate short-lived bearer access tokens), and `oauth2_token_url` (which is the URL of the authorization server providing access tokens).

Like in the case of basic authentication, we recommend using environment variables for sensitive values.

## Dynamic headers

> **Note:** This feature was added in version 2.3.0

When enabled, this feature attaches a header to the data according to what your transformation provides in the `HTTPHeaders` field of `engineProtocol`. Data is batched independently per each dynamic header value before requests are sent.

## Request templating

> **Note:** This feature was added in version 3.0.0

This feature allows you to provide a [Golang text template](https://pkg.go.dev/text/template) to construct a request body from a batch of data. This feature should be useful in constructing requests to send to an API, for example.

Input data must be valid JSON, any message that fails to be marshaled to JSON will be treated as invalid and sent to the failure target. Equally, if an attempt to template a batch of data results in an error, then all messages in the batch will be considered invalid and sent to the failure target.

Where the dynamic headers feature is enabled, data is split into batches according to the provided header value, and the templater will operate on each batch separately.

### Helper functions

In addition to all base functions available in the Go text/template package, the following custom functions are available for convenience:

`prettyPrint` - Because the input to the templater is a Go data structure, simply providing a reference to an object field won't output it as JSON. `prettyPrint` converts the data to prettified JSON. Use it wherever you expect a JSON object in the output. This is compatible with any data type, but it shouldn't be necessary if the data is not an object.

`env` - Allows you to set and refer to an env var in your template. Use it when your request body must contain sensitive data, for example an API key.

### Template example

The following example provides an API key via environment variable, and iterates the batch to provide JSON-formatted data one by one into a new key, inserting a comma before all but the first event.

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/http-template-full-example.file)

## Response rules (beta)

> **Note:** This feature was added in version 3.0.0
>
> This feature is in beta status because we may make breaking changes in future versions.
>
> **Breaking change in version 4.0.0**: Response rules are now evaluated in the order they are defined in the configuration. Rules must specify a `type` attribute to distinguish between invalid, setup, and throttle errors, rather than organizing them in separate `invalid` and `setup` blocks.

Response rules allow you to configure how the app deals with failures in sending the data. You can configure a response code and an optional string match on the response body to determine how a failure response is handled. Response codes between 200 and 299 are considered successful, and are not handled by this feature.

**Response rules are evaluated in the order they are defined in your configuration.** The first matching rule determines how the error is categorized.

There are four categories of failure:

`invalid` means that the data is considered incompatible with the target for some reason. For example, you may have defined a mapping for a given API, but the event happens to have null data for a required field. In this instance, retrying the data won't fix the issue, so you would configure an invalid response rule, identifying which responses indicate this scenario.

Data that matches an invalid response rule is sent to the failure target.

`setup` means that this error is not retryable, but is something which can only be resolved by a change in configuration or a change to the target. An example of this is an authentication failure - retrying won't fix the issue; the resolution is to grant the appropriate permissions, or provide the correct API key.

Data that matches a setup response rule is handled by a retry as determined in the `setup` configuration block of [retry configuration](/docs/api-reference/snowbridge/configuration/retries/).

`throttle` (added in version 4.0.0) is a special type of error that indicates the target is rate limiting requests. This is handled separately from transient errors to allow different retry behavior - typically with longer delays to respect rate limits.

Data that matches a throttle response rule is handled by a retry as determined in the `throttle` configuration block of [retry configuration](/docs/api-reference/snowbridge/configuration/retries/).

`transient` errors are everything else - we assume that the issue is temporary and retrying will resolve the problem. There is no explicit configuration for transient - rather, anything that is not configured as one of the other types is considered transient.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/http-minimal-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/http-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

## Example for Google Tag Manager Server Side

You can use the HTTP target to send events to Google Tag Manager Server Side, where the [Snowplow Client tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/) is installed.

To do this, you will need to include a [transformation](/docs/api-reference/snowbridge/concepts/transformations/) that converts your events to JSON — [`spEnrichedToJson`](/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/).

Here’s an example configuration. Replace `<your-gtm-host>` with the hostname of your Google Tag Manager instance, and — optionally — `<preview-token>` with your preview mode token.

```hcl
target {
  use "http" {
    url                        = "https://<your-gtm-host>/com.snowplowanalytics.snowplow/enriched"
    request_timeout_in_seconds = 5
    content_type               = "application/json"

    # this line is optional, in case you want to send events to GTM Preview Mode
    headers                    = "{\"x-gtm-server-preview\": \"<preview-token>\"}"
  }
}

transform {
  use "spEnrichedToJson" {}
}
```

---

# Snowbridge target configuration
> Configure Snowbridge targets including stdout, EventHub, HTTP, Kafka, Kinesis, PubSub, and SQS for stream output.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/

**Stdout target** is the default. We also support EventHub, HTTP, Kafka, Kinese, PubSub, and SQS targets.

Stdout target doesn't have any configurable options - when configured it simply outputs the messages to stdout.

## Configuration options

Here is an example of the configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/stdout-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use `failure_target` instead of `target`.

---

# Configure Kafka as a Snowbridge target
> Configure Kafka target for Snowplow Snowbridge to write data to Kafka topics with SASL authentication and TLS encryption.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/kafka/

Where SASL is used, it may be enabled via the `enable_sasl`, `sasl_username`, and `sasl_password` and `sasl_algorithm` options.

we recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).

TLS may be configured by providing the `key_file`, `cert_file` and `ca_file` options with paths to the relevant TLS files.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/kafka-minimal-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/kafka-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure Kinesis as a Snowbridge target
> Configure Kinesis target for Snowplow Snowbridge to write data to AWS Kinesis streams with throttle retry handling and IAM authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/kinesis/

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the stream.

## Throttle retries

As of 2.4.2, the kinesis target handles kinesis write throughput exceptions separately from all other errors and failures. It will back off and retry only the throttled records on an initial back off of 50ms, increasing by 50ms each time, until there are no more throttle errors.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/kinesis-minimal-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target. Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/kinesis-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure Pub/Sub as a Snowbridge target
> Configure PubSub target for Snowplow Snowbridge to write data to GCP Pub/Sub topics with service account authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/pubsub/

Authentication is done using a [GCP Service Account](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa). Create a service account credentials file, and provide the path to it via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.

Snowbridge connects to PubSub using [Google's Go Pubsub sdk](https://cloud.google.com/go/pubsub), which establishes a grpc connection with TLS encryption.

## Configuration options

The PubSub Target has only two required options, and no optional ones.

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/pubsub-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Configure SQS as a Snowbridge target
> Configure SQS target for Snowplow Snowbridge to write data to AWS SQS queues with IAM authentication and message attributes.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/targets/sqs/

Authentication is done via the [AWS authentication environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). Optionally, you can use the `role_arn` option to specify an ARN to use on the queue.

## Configuration options

Here is an example of the minimum required configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/sqs-minimal-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

Here is an example of every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/targets/sqs-full-example.hcl)

If you want to use this as a [failure target](/docs/api-reference/snowbridge/concepts/failure-model/#failure-targets), then use failure\_target instead of target.

---

# Snowbridge telemetry configuration
> Enable or disable telemetry for Snowbridge with user-provided identifiers and privacy controls.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/telemetry/

You can read about our telemetry principles [here](/docs/get-started/self-hosted/telemetry/).

To enable telemetry:

```hcl
# Optional. Set to true to disable telemetry.
disable_telemetry = false

# Optional. An identifier to associate with telemetry data.
user_provided_id = "elmer.fudd@acme.com"
```

To disable telemetry:

```hcl
# Optional. Set to true to disable telemetry.
disable_telemetry = false
```

---

# Snowbridge base64Decode transformation
> Base64 decode message data from base64 byte array to decoded byte array representation for Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/base64Decode/

Introduced in version 2.1.0

`base64Decode`: Base64 decodes the message's data.

This transformation base64 decodes the message's data from a base64 byte array, to a byte array representation of the decoded data.

`base64Decode` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/base64Decode-minimal-example.hcl)

---

# Snowbridge base64Encode transformation
> Base64 encode message data to base64 byte array representation for Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/base64Encode/

Introduced in version 2.1.0

`base64Encode`: Base64 encodes the message's data.

This transformation base64 encodes the message's data to a base 64 byte array.

`base64Encode` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/base64Encode-minimal-example.hcl)

---

# Built-in Snowbridge transformations
> Use built-in Snowbridge transformations for Snowplow data including enriched filtering, JSON conversion, and base64 encoding.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/

Snowbridge includes several configurable built-in transformations.

| Transformation                  | Functionality                                                                          | Snowplow data only |
| ------------------------------- | -------------------------------------------------------------------------------------- | ------------------ |
| `base64Decode`                  | Base64-decodes the message's data.                                                     |                    |
| `base64Encode`                  | Base64 encodes the message's data.                                                     |                    |
| `jq`                            | Runs a `jq` command on the message data, and outputs the result of the command.        |                    |
| `jqFilter`                      | Filters messages based on the output of a `jq` command.                                |                    |
| `spEnrichedFilter`              | Filters messages based on a regex match against an atomic field.                       | ✅                  |
| `spEnrichedFilterContext`       | Filters messages based on a regex match against a field in an entity.                  | ✅                  |
| `spEnrichedFilterUnstructEvent` | Filters messages based on a regex match against a field in a custom event.             | ✅                  |
| `spEnrichedSetPk`               | Sets the message's destination partition key.                                          | ✅                  |
| `spEnrichedToJson`              | Transforms a message's data from Snowplow enriched tsv string format to a JSON object. | ✅                  |
| `spGtmssPreview`                | Attaches a GTM SS preview mode header                                                  | ✅                  |

---

# Snowbridge jq transformation
> Run jq commands on message data to transform JSON structures with custom queries and helper functions, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/jq/

> **Note:** This transformation was added in version 3.0.0.

[jq](https://github.com/jqlang/jq) is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the [gojq](https://github.com/itchyny/gojq) package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the [tutorial](https://jqlang.github.io/jq/tutorial/), and [full reference manual](https://jqlang.github.io/jq/manual/). [This open-source jq playground tool](https://jqplay.org/) may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are [some small differences](https://github.com/itchyny/gojq?tab=readme-ov-file#difference-to-jq) between jq and gojq.

`jq` runs a jq command on the message data, and outputs the result of the command. While jq supports multi-element results, commands must output only a single element - this single element can be an array data type.

If the provided jq command results in an error, the message will be considered invalid, and will be sent to the failure target.

The minimal example here returns the input data as a single element array, and the full example maps the data to a new data structure.

The jq transformation will remove any keys with null values from the data.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/jq-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/jq-full-example.hcl)

## Helper functions

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

- `epoch` converts a Go `time.Time` timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as `time.Time`. This function can be chained with jq native functions to get past this limitation. For example:

```text
{ foo: .collector_tstamp | epoch | todateiso8601 }
```

- `epochMillis` converts a Go `time.Time` timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the `epoch` function truncates to seconds, and the `epochMillis` function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.

```text
{ foo: .collector_tstamp | epochMillis }
```

- `hash(algorithm, salt)` hashes the input value. To use unsalted hash, pass an empty string for salt. Salt may be provided as an environment variable using hcl syntax.

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```text
{ foo: .user_id |  hash("sha1"; "${env.SHA1_SALT}") }
```

---

# Snowbridge jqFilter transformation
> Filter messages using jq commands that return boolean results to keep or discard messages, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/

> **Note:** This transformation was added in version 3.0.0.

[jq](https://github.com/jqlang/jq) is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the [gojq](https://github.com/itchyny/gojq) package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures.

jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the [tutorial](https://jqlang.github.io/jq/tutorial/), and [full reference manual](https://jqlang.github.io/jq/manual/). [This open-source jq playground tool](https://jqplay.org/) may also be helpful.

For most use cases, you are unlikely to encounter them, but note that there are [some small differences](https://github.com/itchyny/gojq?tab=readme-ov-file#difference-to-jq) between jq and gojq.

`jqFilter` filters messages based on the output of a jq command which is run against the data. The provided command must return a boolean result. `false` filters the message out, `true` keeps it.

If the provided jq command returns a non-boolean value error, or results in an error, then the message will be considered invalid, and will be sent to the failure target.

## Configuration options

This example filters out all data that doesn't have an `app_id` key.

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/jqFilter-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/builtin/jqFilter-full-example.hcl)

## Filtering examples

The following examples demonstrate common filtering patterns for Snowplow enriched events. You can filter Snowplow enriched events based on any field in the data.

Match an atomic field to any value from a list:

```hcl
transform {
  use "jqFilter" {

    # Keep only web and mobile data
    jq_command = <<JQEOT
    .platform | IN("web", "mobile")
JQEOT

    snowplow_mode = true
  }
}
```

Regex match against a singular entity:

```hcl
transform {
  use "jqFilter" {

    # Keep only "environment" matching a regex in custom event
    # `// ""` is needed as `null` is not regex compatible
    jq_command = <<JQEOT
    .contexts_com_acme_env_context_1.environment // "" | test("^prod")
JQEOT

    snowplow_mode = true
  }
}
```

Filter where any entity entry matches the condition:

```hcl
transform {
  use "jqFilter" {

    # Keep if any entry's environment matches one of two values:
    jq_command = <<JQEOT
    .contexts_com_acme_env_context_1 | any(.[]; .environment == "prod" or .environment == "staging")
JQEOT

    snowplow_mode = true
  }
}
```

Filter for an exact match on a self-describing event field:

```hcl
transform {
  use "jqFilter" {
    # Keep only "sku" of "test-data" in custom event
    jq_command = <<JQEOT
    .unstruct_event_com_acme_my_custom_event_1.sku == "test-data"
JQEOT

    snowplow_mode = true
  }
}
```

## Helper Functions

In addition to the native functions available in the jq language, the following helper functions are available for use in a jq query:

- `epoch` converts a Go `time.Time` timestamp to an epoch timestamp in seconds, as integer type. jq's native timestamp-based functions expect integer input, but the Snowplow Analytics SDK provides base level timestamps as `time.Time`. This function can be chained with jq native functions to get past this limitation. For example:

```text
{ foo: .collector_tstamp | epoch | todateiso8601 }
```

- `epochMillis` converts a Go `time.Time` timestamp to an epoch timestamp in milliseconds, as unsigned integer type. Because of how integers are handled in Go, unsigned integers aren't compatible with jq's native timestamp functions, so the `epoch` function truncates to seconds, and the `epochMillis` function exists in case milliseconds are needed. This function cannot be chained with native jq functions, but where milliseconds matter for a value, use this function.

```text
{ foo: .collector_tstamp | epochMillis }
```

- `hash(algorithm, salt)` hashes the input value. To use unsalted hash, pass an empty string for salt. Salt may be provided as an environment variable using hcl syntax.

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```text
{ foo: .user_id |  hash("sha1"; "${env.SHA1_SALT}") }
```

---

# Snowbridge spEnrichedFilter transformation
> Filter Snowplow enriched events based on regex matches against atomic fields with keep or drop actions, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilter/

> **Warning:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilter`: Specific to Snowplow data. Filters messages based on a regex match against an atomic field.

This transformation is for use on base-level atomic fields, rather than fields from contexts, or custom events — which can be achieved with `spEnrichedFilterContext` and `spEnrichedFilterUnstructEvent`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

This example filters out all data whose `platform` value does not match either `web` or `mobile`.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilter-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilter-full-example.hcl)

---

# Snowbridge spEnrichedFilterContext transformation
> Filter Snowplow enriched events based on regex matches against entity fields using jsonpath notation, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilterContext/

> **Warning:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilterContext`: Specific to Snowplow data. Filters messages based on a regex match against a field in an entity.

This transformation is for use on fields from entities (contexts).

Note that if the same context is present in the data more than once, one instance of a match is enough for the regex condition to be considered a match — and the message to be kept.

The full parsed context name must be provided, in camel case, in the format returned by the Snowplow analytics SDK: `contexts_{vendor}_{name}_{major version}` — for example `contexts_nl_basjes_yauaa_context_1`.

The path to the field to be matched must then be provided as a jsonpath (dot notation and square braces only) — for example `test1.test2[0].test3`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

The below example keeps messages which contain `prod` in the `environment` field of the `contexts_com_acme_env_context_1` context. Note that the `contexts_com_acme_env_context_1` context is attached more than once, if _any_ of the values at `dev` don't match `environment`, the message will be kept.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterContext-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterContext-full-example.hcl)

---

# Snowbridge spEnrichedFilterUnstructEvent transformation
> Filter Snowplow enriched events based on regex matches against self-describing event fields using jsonpath notation, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedFilterUnstructEvent/

> **Warning:** This transformation is deprecated and can result in unexpected behavior when matching integers. Use the [`jqFilter`](/docs/api-reference/snowbridge/configuration/transformations/builtin/jqFilter/) transformation instead, which provides more robust and flexible filtering capabilities.

`spEnrichedFilterUnstructEvent`: Specific to Snowplow data. Filters messages based on a regex match against a field in a custom event.

This transformation is for use on fields from custom events.

The event name must be provided as it appears in the `event_name` field of the event (eg. `add_to_cart`).

Optionally, a regex can be provided to match against the stringified version of the event (eg. `1-*-*`)

The path to the field to match against must be provided as a jsonpath (dot notation and square braces only) — for example `test1.test2[0].test3`.

Filters can be used in one of two ways, which is determined by the `filter_action` option. `filter_action` determines the behavior of the app when the regex provided evaluates to `true`. If it's set to `"keep"`, the app will complete the remaining transformations and send the message to the destination (unless a subsequent filter determines otherwise). If it's set to `"drop"`, the message will be acked and discarded, without continuing to the next transformation or target.

## Configuration options

This example keeps all events whose `add_to_cart` event data at the `sku` field matches `test-data`.

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterUnstructEvent-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedFilterUnstructEvent-full-example.hcl)

---

# Snowbridge spEnrichedSetPk transformation
> Set destination partition key for Snowplow enriched events using atomic field values, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedSetPk/

`spEnrichedSetPk`: Specific to Snowplow data. Sets the message's destination partition key to an atomic field from a Snowplow Enriched tsv string. The input data must be a valid Snowplow enriched TSV.

## Configuration options

`SpEnrichedSetPk` only takes one option — the field to use for the partition key.

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedSetPk-minimal-example.hcl)

Note: currently, setting partition key to fields in custom events and contexts is unsupported.

---

# Snowbridge spEnrichedToJson transformation
> Transform Snowplow enriched TSV data to JSON format using the Go Analytics SDK, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/

`spEnrichedToJson`: Specific to Snowplow data. Transforms a message's data from Snowplow Enriched tsv string format to a JSON object. The input data must be a valid Snowplow enriched TSV.

`spEnrichedToJson` has no options.

## Configuration options

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spEnrichedToJson-minimal-example.hcl)

The transformation to JSON is done via the [analytics SDK](/docs/api-reference/analytics-sdk/) logic, specifically in this case the [Golang analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/).

In brief, the relevant logic here is that:

- If a field is not populated in the original event, it won't have a key in the resulting JSON

- In the TSV, there's a separate field for `contexts` (sent via tracker), and `derived_contexts` (attached during enrichment). In the analytics SDK, there is one key per context, regardless of which type. (Technically, it's one key per major version of a context. So if you had a 1.0.0 and a 2.0.0 of the same one, you'd have two keys).

---

# Snowbridge spGtmssPreview transformation
> Extract GTM Server Side preview mode header from Snowplow context for debugging with GTM preview mode, with Snowbridge.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/builtin/spGtmssPreview/

> **Note:** This transformation was added in version 2.3.0

`spGtmssPreview`: Specific to Snowplow data. Extracts a value from the `x-gtm-server-preview` field of a [preview mode context](https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0), and attaches it as the GTM SS preview mode header, to enable easier debugging using GTM SS preview mode.

Only one preview mode context should be sent at a time.

> **Note:** As of version 3.0.0:
>
> Invalid preview headers sent to GTM SS can result in requests failing, which may be problematic. There is insufficient information available about the values to allow us to confidently validate them, but we do two things to avoid this problem.
>
> First, we validate to ensure that the value is a valid base64 string. Second, we compare the age of the event (based on `collector_tstamp`) to ensure it is under a configurable timeout age. If either of these conditions fail, we treat the message as invalid, and output to the failure target.

## Configuration options

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spGtmssPreview-minimal-example.hcl)

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/snowplow-builtin/spGtmssPreview-full-example.hcl)

---

# Snowbridge Script transformation examples
> View example Snowbridge JavaScript transformations for Snowplow and non-Snowplow data including filtering, field modification, and partition keys.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/

Examples showing the transformation of Snowplow or non-Snowplow data.

## Non-Snowplow data

For this example, the input data is a json string which looks like this:

```json
{
  "name": "Bruce",
  "id": "b47m4n",
  "batmobileCount": 1
}
```

The script filters out any data with a `batmobileCount` less than 1, otherwise it updates the Data's `name` field to "Bruce Wayne", and sets the PartitionKey to the value of `id`:

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/examples/js-non-snowplow-script-example.js)

The configuration for this script is:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/examples/js-non-snowplow-config-example.hcl)

## Snowplow data

For this example, the input data is a valid Snowplow TSV event - so we can enable `snowplow_mode`, which will convert the data to a JSON before passing it to the script as a JSON object.

The script below filters out non-web data, based on the `platform` value, otherwise it checks for a `user_id` value, setting a new `uid` field to that value if it's found, or `domain_userid` if not.

It also sets the partitionKey to `app_id`.

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/examples/js-snowplow-script-example.js)

The configuration for this script is:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/examples/js-snowplow-config-example.hcl)

---

# Custom Snowbridge script transformations
> Create custom JavaScript Snowbridge transformations to modify data, filter messages, set partition keys, and add HTTP headers.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/

Custom transformation scripts may be defined in Javascript and provided to Snowbridge.

## The scripting interface

The script must define a main function with a single argument. Snowbridge will pass the `engineProtocol` data structure as the argument:

```go
type engineProtocol struct {
    FilterOut    bool
    PartitionKey string
    Data         interface{}
    HTTPHeaders  map[string]string
}
```

This structure is represented as an object in the script engine, and serves as both the input and output of the script.

Scripts must define a `main` function with a single input argument (JSDoc for type information is optional):

```js
/**
 * @typedef {object} EngineProtocol
 * @property {boolean} FilterOut
 * @property {string} PartitionKey
 * @property {(string | Object.<string, ?>)} Data
 * @property {Object.<string, string>} HTTPHeaders
 */

/**
 * @param {EngineProtocol} input
 * @return {Partial<EngineProtocol>}
 */
function main(input) {
    return input
}
```

## Accessing data

Scripts can access the message Data at `input.Data`, and can return modified data by returning it in the `Data` field of the output. Likewise for the partition key to be used for the destination - `input.PartitionKey` and the `PartitionKey` field of the output.

By default, the input's `Data` field will be a string in [enriched TSV format](/docs/pipeline/enriched-tsv-format/). This can be changed with the [SpEnrichedToJson](/docs/api-reference/snowbridge/configuration/transformations/builtin/spEnrichedToJson/) transform, or the Javascript transformation itself has a `snowplow_mode` option, which transforms the data to an object first.

The output of the script must be an object which maps to engineProtocol.

### `snowplow_mode`

`snowplow_mode` uses the [Go Analytics SDK](/docs/api-reference/analytics-sdk/analytics-sdk-go/) to parse the TSV fields into an object suitable for working with in Go. The result of the [`ParsedEvent.ToMap()`](https://pkg.go.dev/github.com/snowplow/snowplow-golang-analytics-sdk/analytics#ParsedEvent.ToMap) method is the input to your transform function.

The [keys of the resulting map](https://github.com/snowplow/snowplow-golang-analytics-sdk/blob/a3430fbe576483d615b713120cfb5e443897d572/analytics/mappings.go#L153) are defined by the Analytics SDK. Values are primitives (string, number, boolean) or, for timestamps (e.g. `derived_tstamp`, `collector_tstamp`), [`time.Time`](https://pkg.go.dev/time#Time) objects.

> **Tip:** To work with native Javascript [Date](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date) objects for timestamps, construct one via `new Date(ts.UnixMilli())` where `ts` is a `time.Time` instance.
>
> When returned as part of `Data`, `time.Time` instances will [serialize in JSON](https://pkg.go.dev/time#Time.MarshalJSON) as [RFC3339](https://www.rfc-editor.org/rfc/rfc3339.html) strings.

Structured data ([Self Describing Event](/docs/fundamentals/events/#self-describing-events) payloads and [Entities](/docs/fundamentals/entities/)) will have keys with a prefix of `unstruct_event_` or `contexts_`, the vendor name converted to snake case, the event/entity name in snake case, and the [schema model version](/docs/api-reference/iglu/common-architecture/schemaver/). For example:

| **Schema URI**                                                  | **Type** | **Key**                                              |
| --------------------------------------------------------------- | -------- | ---------------------------------------------------- |
| `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0` | Entity   | `contexts_com_snowplowanalytics_snowplow_web_page_1` |
| `iglu:org.w3/PerformanceNavigationTiming/jsonschema/1-0-0`      | Entity   | `contexts_org_w3_performance_navigation_timing_1`    |
| `iglu:com.urbanairship.connect/OPEN/jsonschema/1-0-0`           | Event    | `unstruct_event_com_urbanairship_connect_open_1`     |

Timestamps in structured data values will always have their primitive representation (string/number) and not be `time.Time` instances.

## Transforming Data

For all the below examples, the input is a string representation of the below JSON object. For Snowplow data, using `snowplow_mode` will produce a JSON object input - see [the snowplow example](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

```json
{
  "name": "Bruce",
  "id": "b47m4n",
  "batmobileCount": 1
}
```

To modify the message data, return an object which conforms to EngineProtocol, with the `Data` field set to the modified data. The `Data` field may be returned as either a string, or an object.

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-modify-example.js)

## Filtering

If the `FilterOut` field of the output is returned as `true`, the message will be acknowledged immediately and won't be sent to the target. This will be the behavior regardless of what is returned to the other fields in the protocol.

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-filter-example.js)

## Setting the Partition Key

To set the Partition Key in the message, you can simply set the input's PartitionKey field, and return it:

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-setpk-example.js)

Or, if modifying the data as well, return the modified data and PartitionKey field:

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-setpk-modify-example.js)

## Setting an HTTP header

For the `http` target only, you can specify a set of HTTP headers, which will be appended to the configured headers for the `http` target. Do so by providing an object in the `HTTPHeaders` field:

```js
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/create-a-script-header-example.js)

The headers will only be included if the target has the [`dynamic_headers = true` setting](/docs/api-reference/snowbridge/configuration/targets/http/#configuration-options) configured.

## Helper functions

- `hash(input, algorithm)` hashes the input value. Salt is configured using the `hash_salt_secret` parameter in the hcl configuration. If no value is provided, this function will perform an unsalted hash

The following hash algorithms are supported:

- `sha1` - SHA-1 hash (160 bits)
- `sha256` - SHA-256 hash (256 bits)
- `md5` - MD5 hash (128 bits)

```text
hash(input.Data["app_id"], "sha1")
```

## Configuration

Once your script is ready, you can configure it in the app by following the [Javascript](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) configuration page.

You can also find some complete example use cases in [the examples section](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

---

# Snowbridge JavaScript transformation configuration
> Configure Snowbridge JavaScript transformations using the goja embedded JavaScript engine with script paths and timeout settings.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/

This section details how to configure the transformation, once a script is written.

You can also find some complete example use cases in [the examples section](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/examples/).

The custom JavaScript script transformation uses the [goja](https://pkg.go.dev/github.com/dop251/goja) embedded Javascript engine to run scripts upon the data.

If a script errors or times out, a [transformation failure](/docs/api-reference/snowbridge/concepts/failure-model/#transformation-failure) occurs.

Scripts must be available to the runtime of the application at the path provided in the `script_path` configuration option. For docker, this means mounting the script to the container and providing that path.

## Configuration options

Minimal configuration:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/js-configuration-minimal-example.hcl)

Every configuration option:

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/custom-scripts/js-configuration-full-example.hcl)

---

# Snowbridge transformation configuration
> Configure transformations and filters to modify or exclude messages with built-in transformations or custom scripts.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/configuration/transformations/

You can configure any number of transformations to run on the data one after another - transformations will run in the order provided. (You can repeatedly specify the same transformation more than once, if needed.) All transformations operate on a single message basis.

If you're filtering the data, it's best to provide the filter first, for efficiency.

If you're working with Snowplow enriched messages, you can configure scripting transformations, or any of the built-in transformations, which are specific to Snowplow data.

If you're working with any other type of data, you can create transformations via scripting transformations.

## Transformations and filters

Transformations modify messages in-flight. They might rename fields, perform computations, set partition keys, or modify data. For example if you wanted to change a `snake_case` field name to `camelCase`, you would use a transformation to do this.

Filters are a type of transformation which prevent Snowbridge from further processing data based on a condition. When data is filtered, Snowbridge will ack the message without sending it to the target. For example if you only wanted to send page views to the destination, you would set up a filter with a condition where `event_name` matches the string `page_view`.

## Configuration

To configure transformations, supply one or more `transform {}` block. Choose the transformation using `use "${transformation_name}"`.

Example:

The below first filters out any `event_name` which does not match the regex `^page_view$`, then runs a custom javascript script to change the app\_id value to `"1"`

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowbridge/blob/v4.1.0/assets/docs/configuration/transformations/transformations-overview-example.hcl)

---

# Getting started with Snowbridge
> Install and configure Snowbridge using Docker or binaries to start replicating event streams between sources and targets.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/getting-started/

The fastest way to get started and experiment with Snowbridge is to run it via the command line:

1. Download the pre-compiled ZIP from the [releases](https://github.com/snowplow/snowbridge/releases/)
2. Unzip and run the binary with eg. `echo "hello world" | ./snowbridge`

The defaults for the app are stdin source, no transformations, and stdout target - so this should print the message 'hello world' along with some logging data to the console.

Next, the app can be configured using HCL - simply create a configuration file, and provide the path to it using the `SNOWBRIDGE_CONFIG_FILE` environment variable.

You can find a guide to configuration in the [configuration section](/docs/api-reference/snowbridge/configuration/).

**Telemetry notice**

By default, Snowplow collects telemetry data for Snowbridge (since version 1.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `user_provided_id` configuration setting.

If you wish to disable telemetry, you can do so by setting `disable_telemetry` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

## Distribution options

There are two distributions of Snowbridge.

**Default:**

The default distribution contains everything except for the [Kinesis source](/docs/api-reference/snowbridge/configuration/sources/kinesis/), i.e. the ability to read from AWS Kinesis. This distribution is all licensed under the [Snowplow Limited Use License Agreement](/limited-use-license-1.0/). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](/docs/licensing/limited-use-license-faq/).)_

It’s available on Docker:

```bash
docker pull snowplow/snowbridge:4.1.0
docker run snowplow/snowbridge:4.1.0
```

**AWS-specific (includes Kinesis source):**

The AWS-specific distribution contains everything, including the [Kinesis source](/docs/api-reference/snowbridge/configuration/sources/kinesis/), i.e. the ability to read from AWS Kinesis. Like the default distribution, it’s licensed under the [Snowplow Limited Use License Agreement](/limited-use-license-1.0/) ([frequently asked questions](/docs/licensing/limited-use-license-faq/)). However, this distribution has a dependency on [twitchscience/kinsumer](https://github.com/twitchscience/kinsumer), which is licensed by Twitch under the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE).

To comply with the [Amazon Software License](https://github.com/twitchscience/kinsumer/blob/master/LICENSE), you may only use this distribution of Snowbridge _“with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services, Inc.”_

It’s available on Docker:

```bash
docker pull snowplow/snowbridge:4.1.0-aws-only
docker run snowplow/snowbridge:4.1.0-aws-only
```

***

## Deployment

The app can be deployed via services like EC2, ECS or Kubernetes using docker.

Configuration and authentication can be done by mounting the relevant files, and/or setting the relevant environment variables as per the standard authentication methods for cloud services.

---

# Replicate event streams in real time with Snowbridge
> Replicate Snowplow event streams to multiple destinations with Snowbridge, a configurable tool supporting Kinesis, PubSub, Kafka, HTTP, and more.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/

Snowbridge is a flexible, low latency tool which can replicate streams of data of any type to external destinations, optionally filtering or transforming the data along the way. It can be used to consume, transform and relay data to any third party platform which supports HTTP or is listed as a target below — in real-time.

## Features

- [Kinesis](https://aws.amazon.com/kinesis), [SQS](https://aws.amazon.com/sqs/), [PubSub](https://cloud.google.com/pubsub), [Kafka](https://kafka.apache.org/), and stdin sources

- [Kinesis](https://aws.amazon.com/kinesis), [SQS](https://aws.amazon.com/sqs/), [PubSub](https://cloud.google.com/pubsub), [Kafka](https://kafka.apache.org/), [Event Hubs](https://azure.microsoft.com/en-us/services/event-hubs/), HTTP (e.g. for an [integration with Google Tag Manager Server Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)), and stdout targets

- Custom in-flight JS transformations

- Low-latency Snowplow-specific data transformations

- Statsd and Sentry reporting and monitoring interfaces

Snowbridge is a generic tool, built to work on any type of data, developed by the Snowplow team. It began life as a closed-source tool developed to deliver various requirements related to Snowplow data, and so some of the features are specific to that data.

> **Note:** Version 4.0.0 includes breaking changes to response rule evaluation. See the [upgrade guide](/docs/api-reference/snowbridge/X-X-upgrade-guide/) for migration information.

---

# Testing Snowbridge locally
> Test Snowbridge configurations locally using stdin and stdout sources before deploying to production stream infrastructure.
> Source: https://docs.snowplow.io/docs/api-reference/snowbridge/testing/

The easiest way to test Snowbridge configuration (e.g. transformations) is to run it locally. Ideally, you should also use a sample of data that is as close to your real world data as possible. The sample file should contain the events/messages you’d like to test with, one per line.

## Snowplow data

You can get started working with Snowplow data by [downloading this file](/assets/files/input-5ef65c98e57e93e452e9f6bd5a413fed.txt/) which contains a sample of web and mobile Snowplow events in TSV format. However, if you need events that match your actual events, generate your own events sample.

In order to generate your own sample of Snowplow data, you can follow the [guide to use Snowplow Micro](/docs/testing/snowplow-micro/local/) to generate test data, using the `--output-tsv` to get the data into a file, as per the [exporting to tsv section](/docs/testing/snowplow-micro/local/#exporting-events).

For example, here we’re using a file named `data.tsv`:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv > data.tsv
```

Point some test environment tracking to `localhost:9090`, and your events should land in `data.tsv`.

## Testing Snowbridge locally

You can run Snowbridge locally via Docker:

```bash
docker run --env ACCEPT_LIMITED_USE_LICENSE=yes snowplow/snowbridge:4.1.0
```

The default configuration for Snowbridge uses the `stdin` source and the `stdout` target. So, to test sending data through with no transformations, we can run the following command (where `data.tsv` is a file with Snowplow events in TSV format):

```bash
cat data.tsv | docker run --env ACCEPT_LIMITED_USE_LICENSE=yes -i snowplow/snowbridge:4.1.0
```

This will print the data to the terminal, along with logs.

> **Note:** The metrics reported in the logs may state that no data has been processed. This is because the app reached the end of output and exited before the default reporting period of 1 second. You can safely ignore this.

You can output the results to a file to make it easier to examine them:

```bash
cat data.tsv | docker run -i snowplow/snowbridge:4.1.0 > output.txt
```

> **Tip:** The output (in `output.txt`) will contain more than the data itself. There will be additional fields called `PartitionKey`, `TimeCreated` `TimePulled` and `TimeTransformed`. The data that reaches the target in a production setup is under `Data`.

### Adding configuration

To add a specific configuration to test, create a configuration file (`config.hcl`) and pass it to the Docker container. You will need to [mount the file](https://docs.docker.com/storage/bind-mounts/) to the default config path of `/tmp/config.hcl`:

```bash
cat data.tsv | docker run -i \
  --env ACCEPT_LIMITED_USE_LICENSE=yes \
  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \
  snowplow/snowbridge:4.1.0 > output.txt
```

Note that docker expects absolute paths for mounted files - here we use `$(pwd)` but you can specify the aboslute path manually too.

To test transformations, you only need to add the `transform` block(s) to your configuration file. Don’t specify the `source` and `target` blocks to leave them on default (`stdin` and `stdout`).

To test specific sources or targets, add the respective `source` or `target` blocks. For example, see the [configuration](/docs/api-reference/snowbridge/configuration/targets/http/#example-for-google-tag-manager-server-side) for an HTTP target sending data to Google Tag Manager Server Side.

### Adding a custom transformation script

You can add custom scripts by mounting a file, similarly to the above. Assuming the script is in `script.js`, that looks like this:

```bash
cat data.tsv | docker run -i \
  --env ACCEPT_LIMITED_USE_LICENSE=yes \
  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \
  --mount type=bind,source=$(pwd)/script.js,target=/tmp/script.js \
  snowplow/snowbridge:4.1.0 > output.txt
```

The transformation config should point to the path of the script _inside_ the container (`/tmp/script.js` above). For example, the transformation block in the configuration might look like this:

```hcl
transform {
  use "js" {
    script_path = "/tmp/script.js"
  }
}
```

### Adding an HTTP request template

For the HTTP target, you can add a custom HTTP request template by mounting a file, similarly to above. Assuming the template is in `template.tpl`, that looks like this:

```bash
cat data.tsv | docker run -i \
  --env ACCEPT_LIMITED_USE_LICENSE=yes \
  --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \
  --mount type=bind,source=$(pwd)/template.tpl,target=/tmp/template.tpl \
  snowplow/snowbridge:4.1.0
```

The HTTP target config should point to the path of the script _inside_ the container (`/tmp/template.tpl` above). For example, the HTTP target block in the configuration might look like this:

```hcl
target {
  use "http" {
    url             = "http://myApi.com/events"
    template_file   = "/tmp/template.tpl"
  }
}
```

## Using Docker Compose

Instead of creating an intermediate output file with Snowplow Micro and then processing that file in Snowbridge, you can also use Docker Compose to run Micro and Snowbridge together (since Micro 3.0.0 and Snowbridge 3.6.2).

This way, any events sent to Micro will immediately make it to Snowbridge.

Here is an example setup. Copy the code into `docker-compose.yml` and run with `docker-compose up`:

```yaml
services:
  micro:
    image: snowplow/snowplow-micro:4.1.1
    ports:
      - "9090:9090"
    command: --output-tsv --destination http://snowbridge:8080
  snowbridge:
    image: snowplow/snowbridge:4.1.0
    environment:
      - SNOWBRIDGE_CONFIG_FILE=/tmp/config.hcl
    configs:
      - source: snowbridge_config
        target: /tmp/config.hcl

configs:
  snowbridge_config:
    content: |
      license {
        accept = true
      }

      source {
        use "http" {
          url = "0.0.0.0:8080"
        }
      }

      transform {
        use "spEnrichedToJson" {
        }
      }

      # any other Snowbridge configuration
```

## Further testing

You can use either method above (TSV file or Docker Compose) to test all aspects of the app from a local environment too, including sources, targets, failure targets, metrics endpoints etc. In some cases, you'll need to ensure that the local environment has access to any required resources and can authenticate, such as connecting from a laptop to a cloud account/local mock of cloud resources, or setting up a local metrics server for testing. Once that’s done, provide Snowbridge with an hcl file configuring it to connect to those resources, and run it the same way as in the examples above.

---

# Snowplow Micro REST API
> Snowplow Micro REST API endpoints for querying good events, bad events, and resetting cache.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-micro/api/

This page documents the REST API of [Snowplow Micro](/docs/testing/snowplow-micro/).

## /micro/all

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/).

This endpoint responds with a summary JSON object of the number of total, good and bad events currently in the cache.

### HTTP method

`GET`, `POST`, `OPTIONS`

### Response format

Example:

```json
{
  "total": 7,
  "good": 5,
  "bad": 2
}
```

## /micro/good

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/).

This endpoint queries the good events, which are the events that have been successfully validated.

### HTTP method

- `GET`: get _all_ the good events from the cache.
- `POST`: get the good events with the possibility to filter.

### Response format

JSON array of [GoodEvent](https://github.com/snowplow/snowplow-micro/blob/master/src/main/scala/com.snowplowanalytics.snowplow.micro/model.scala#L19)s. A `GoodEvent` contains 4 fields:

- `rawEvent`: contains the [RawEvent](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/adapters/RawEvent.scala#L28). It corresponds to the format of a validated event just before being enriched.
- `event`: contains the [canonical snowplow Event](https://github.com/snowplow/snowplow-scala-analytics-sdk/blob/master/src/main/scala/com.snowplowanalytics.snowplow.analytics.scalasdk/Event.scala#L42). It is in the format of an event after enrichment, even if all the enrichments are deactivated.
- `eventType`: type of the event.
- `schema`: schema of the event in case of an unstructured event.
- `contexts`: contexts of the event.

An example of a response with one event can be found below:

```json
[
  {
    "rawEvent":{
      "api":{
        "vendor":"com.snowplowanalytics.snowplow",
        "version":"tp2"
      },
      "parameters":{
        "e":"pv",
        "duid":"36746fd2-8441-4ea2-8ad0-237d6f4c77cf",
        "vid":"1",
        "co":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"5cea9899-10df-4ccf-bf66-8c36f4a4bba2\"}}]}",
        "eid":"bee0a6d7-fc17-4392-b2bc-2208e8e944f3",
        "url":"http://localhost:8000/",
        "refr":"http://localhost:8000/__/",
        "aid":"shop",
        "tna":"sp1",
        "cs":"UTF-8",
        "cd":"24",
        "stm":"1630238465752",
        "tz":"Europe/London",
        "tv":"js-3.1.3",
        "vp":"1000x660",
        "ds":"988x670",
        "res":"1920x1080",
        "cookie":"1",
        "p":"web",
        "dtm":"1630238465748",
        "uid":"tester",
        "lang":"en-US",
        "sid":"6d15a4fb-9623-4ba1-b876-5240e72e6970"
      },
      "contentType":"application/json",
      "source":{
        "name":"ssc-2.3.1-stdout$",
        "encoding":"UTF-8",
        "hostname":"0.0.0.0"
      },
      "context":{
        "timestamp":"2021-08-29T12:01:05.787Z",
        "ipAddress":"172.17.0.1",
        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "refererUri":"http://localhost:8000/",
        "headers":[
          "Timeout-Access: <function1>",
          "Connection: keep-alive",
          "Host: 0.0.0.0:9090",
          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
          "Accept: */*",
          "Accept-Language: en-US, en;q=0.5",
          "Accept-Encoding: gzip",
          "Referer: http://localhost:8000/",
          "Origin: http://localhost:8000",
          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",
          "application/json"
        ],
        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"
      }
    },
    "eventType":"page_view",
    "schema":"iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0",
    "contexts":[
      "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"
    ],
    "event":{
      "app_id":"shop",
      "platform":"web",
      "etl_tstamp":"2021-08-29T12:01:05.792Z",
      "collector_tstamp":"2021-08-29T12:01:05.787Z",
      "dvce_created_tstamp":"2021-08-29T12:01:05.748Z",
      "event":"page_view",
      "event_id":"bee0a6d7-fc17-4392-b2bc-2208e8e944f3",
      "txn_id":null,
      "name_tracker":"sp1",
      "v_tracker":"js-3.1.3",
      "v_collector":"ssc-2.3.1-stdout$",
      "v_etl":"snowplow-micro-1.3.1-common-2.0.2",
      "user_id":"tester",
      "user_ipaddress":"172.17.0.1",
      "user_fingerprint":null,
      "domain_userid":"36746fd2-8441-4ea2-8ad0-237d6f4c77cf",
      "domain_sessionidx":1,
      "network_userid":"3734601f-5c3d-47c5-b367-0883e1ed74e6",
      "geo_country":null,
      "geo_region":null,
      "geo_city":null,
      "geo_zipcode":null,
      "geo_latitude":null,
      "geo_longitude":null,
      "geo_region_name":null,
      "ip_isp":null,
      "ip_organization":null,
      "ip_domain":null,
      "ip_netspeed":null,
      "page_url":"http://localhost:8000/",
      "page_title":null,
      "page_referrer":"http://localhost:8000/__/",
      "page_urlscheme":"http",
      "page_urlhost":"localhost",
      "page_urlport":8000,
      "page_urlpath":"/",
      "page_urlquery":null,
      "page_urlfragment":null,
      "refr_urlscheme":"http",
      "refr_urlhost":"localhost",
      "refr_urlport":8000,
      "refr_urlpath":"/__/",
      "refr_urlquery":null,
      "refr_urlfragment":null,
      "refr_medium":null,
      "refr_source":null,
      "refr_term":null,
      "mkt_medium":null,
      "mkt_source":null,
      "mkt_term":null,
      "mkt_content":null,
      "mkt_campaign":null,
      "contexts":{
        "schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
        "data":[
          {
            "schema":"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
            "data":{
              "id":"5cea9899-10df-4ccf-bf66-8c36f4a4bba2"
            }
          }
        ]
      },
      "se_category":null,
      "se_action":null,
      "se_label":null,
      "se_property":null,
      "se_value":null,
      "unstruct_event":null,
      "tr_orderid":null,
      "tr_affiliation":null,
      "tr_total":null,
      "tr_tax":null,
      "tr_shipping":null,
      "tr_city":null,
      "tr_state":null,
      "tr_country":null,
      "ti_orderid":null,
      "ti_sku":null,
      "ti_name":null,
      "ti_category":null,
      "ti_price":null,
      "ti_quantity":null,
      "pp_xoffset_min":null,
      "pp_xoffset_max":null,
      "pp_yoffset_min":null,
      "pp_yoffset_max":null,
      "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
      "br_name":null,
      "br_family":null,
      "br_version":null,
      "br_type":null,
      "br_renderengine":null,
      "br_lang":"en-US",
      "br_features_pdf":null,
      "br_features_flash":null,
      "br_features_java":null,
      "br_features_director":null,
      "br_features_quicktime":null,
      "br_features_realplayer":null,
      "br_features_windowsmedia":null,
      "br_features_gears":null,
      "br_features_silverlight":null,
      "br_cookies":true,
      "br_colordepth":"24",
      "br_viewwidth":1000,
      "br_viewheight":660,
      "os_name":null,
      "os_family":null,
      "os_manufacturer":null,
      "os_timezone":"Europe/London",
      "dvce_type":null,
      "dvce_ismobile":null,
      "dvce_screenwidth":1920,
      "dvce_screenheight":1080,
      "doc_charset":"UTF-8",
      "doc_width":988,
      "doc_height":670,
      "tr_currency":null,
      "tr_total_base":null,
      "tr_tax_base":null,
      "tr_shipping_base":null,
      "ti_currency":null,
      "ti_price_base":null,
      "base_currency":null,
      "geo_timezone":null,
      "mkt_clickid":null,
      "mkt_network":null,
      "etl_tags":null,
      "dvce_sent_tstamp":"2021-08-29T12:01:05.752Z",
      "refr_domain_userid":null,
      "refr_dvce_tstamp":null,
      "derived_contexts":{},
      "domain_sessionid":"6d15a4fb-9623-4ba1-b876-5240e72e6970",
      "derived_tstamp":"2021-08-29T12:01:05.783Z",
      "event_vendor":"com.snowplowanalytics.snowplow",
      "event_name":"page_view",
      "event_format":"jsonschema",
      "event_version":"1-0-0",
      "event_fingerprint":null,
      "true_tstamp":null
    }
  }
]
```

### Filters

When querying `/micro/good` with `POST` (`Content-Type: application/json` needs to be set in the headers of the request), it's possible to specify filters, thanks to a JSON in the data of the HTTP request.

Example of command to query the good events: 

```bash
curl -X POST -H 'Content-Type: application/json' <IP:PORT>/micro/good -d '<JSON>'
```

An example of JSON with filters could be:

```json
{
  "schema": "iglu:com.acme/example/jsonschema/1-0-0",
  "contexts": [
    "com.snowplowanalytics.mobile/application/jsonschema/1-0-0",
    "com.snowplowanalytics.mobile/screen/jsonschema/1-0-0"
  ],
  "limit": 10
}
```

List of possible fields for the filters:

- `event_type`: type of the event (in `e` param);
- `schema`: corresponds to the schema of a [self-describing event](/docs/fundamentals/events/#self-describing-events) (schema of the self-describing JSON contained in `ue_pr` or `ue_px`). It automatically implies `event_type` = `ue`.
- `contexts`: list of the schemas contained in the contexts of an event (parameters `co` or `cx`). An event must contain **all** the contexts of the list to be returned. It can also contain more contexts than the ones specified in the request.
- `limit`: limit the number of events in the response (most recent events are returned).

It's not necessary to specify all the fields in a request, only the ones that need to be used for filtering.

## /micro/bad

> **Note:** This endpoint is not available when using Micro [through Snowplow Console](/docs/testing/snowplow-micro/console/).

This endpoint queries the bad events, which are the events that failed validation.

### HTTP method

- `GET`: get _all_ the bad events from the cache.
- `POST`: get the bad events with the possibility to filter.

### Response format

JSON array of [BadEvent](https://github.com/snowplow/snowplow-micro/blob/master/src/main/scala/com.snowplowanalytics.snowplow.micro/model.scala#L28)s. A `BadEvent` contains 3 fields:

- `collectorPayload`: contains the [CollectorPayload](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/loaders/CollectorPayload.scala#L107) with all the raw information of the tracking event. This field can be empty if an error occured before trying to validate a payload.
- `rawEvent`: contains the [RawEvent](https://github.com/snowplow/enrich/blob/master/modules/common/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/adapters/RawEvent.scala#L28). It corresponds to the format of a validated event just before being enriched.
- `errors`: list of errors that occured during the validation of the tracking event.

An example of a response with one bad event can be found below:

```json
[
  {
    "collectorPayload":{
      "api":{
        "vendor":"com.snowplowanalytics.snowplow",
        "version":"tp2"
      },
      "querystring":[],
      "contentType":"application/json",
      "body":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4\",\"data\":[{\"e\":\"ue\",\"eid\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\",\"tv\":\"js-3.1.3\",\"tna\":\"spmicro\",\"aid\":\"sh0pspr33\",\"p\":\"web\",\"cookie\":\"1\",\"cs\":\"UTF-8\",\"lang\":\"en-US\",\"res\":\"1920x1080\",\"cd\":\"24\",\"tz\":\"Europe/London\",\"dtm\":\"1630234190717\",\"vp\":\"1000x660\",\"ds\":\"1003x2242\",\"vid\":\"1\",\"sid\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\",\"duid\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\",\"uid\":\"tester\",\"refr\":\"http://localhost:8000/\",\"url\":\"http://localhost:8000/shop/\",\"ue_pr\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\",\"co\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\",\"stm\":\"1630234190719\"}]}",
      "source":{
        "name":"ssc-2.3.1-stdout$",
        "encoding":"UTF-8",
        "hostname":"0.0.0.0"
      },
      "context":{
        "timestamp":"2021-08-29T10:49:50.727Z",
        "ipAddress":"172.17.0.1",
        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "refererUri":"http://localhost:8000/",
        "headers":[
          "Timeout-Access: <function1>",
          "Connection: keep-alive",
          "Host: 0.0.0.0:9090",
          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
          "Accept: */*",
          "Accept-Language: en-US, en;q=0.5",
          "Accept-Encoding: gzip",
          "Referer: http://localhost:8000/",
          "Origin: http://localhost:8000",
          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",
          "application/json"
        ],
        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"
      }
    },
    "rawEvent":{
      "api":{
        "vendor":"com.snowplowanalytics.snowplow",
        "version":"tp2"
      },
      "parameters":{
        "e":"ue",
        "duid":"8a17bb29-e35c-4363-aec3-85b9b363f9bf",
        "vid":"1",
        "co":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\",\"data\":{\"sku\":\"hh456\",\"name\":\"One-size bucket hat\",\"price\":24.49,\"quantity\":\"2\"}},{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\"}}]}",
        "eid":"36c39024-7b1b-4c2c-ae85-e95a8cb8340a",
        "url":"http://localhost:8000/shop/",
        "refr":"http://localhost:8000/",
        "aid":"sh0pspr33",
        "tna":"spmicro",
        "cs":"UTF-8",
        "cd":"24",
        "stm":"1630234190719",
        "tz":"Europe/London",
        "tv":"js-3.1.3",
        "vp":"1000x660",
        "ds":"1003x2242",
        "ue_pr":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\",\"data\":{\"schema\":\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\",\"data\":{\"type\":\"add\"}}}",
        "res":"1920x1080",
        "cookie":"1",
        "p":"web",
        "dtm":"1630234190717",
        "uid":"tester",
        "lang":"en-US",
        "sid":"13c8f5ac-d999-4923-940d-b39f7b74aa94"
      },
      "contentType":"application/json",
      "source":{
        "name":"ssc-2.3.1-stdout$",
        "encoding":"UTF-8",
        "hostname":"0.0.0.0"
      },
      "context":{
        "timestamp":"2021-08-29T10:49:50.727Z",
        "ipAddress":"172.17.0.1",
        "useragent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "refererUri":"http://localhost:8000/",
        "headers":[
          "Timeout-Access: <function1>",
          "Connection: keep-alive",
          "Host: 0.0.0.0:9090",
          "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0",
          "Accept: */*",
          "Accept-Language: en-US, en;q=0.5",
          "Accept-Encoding: gzip",
          "Referer: http://localhost:8000/",
          "Origin: http://localhost:8000",
          "Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6",
          "application/json"
        ],
        "userId":"3734601f-5c3d-47c5-b367-0883e1ed74e6"
      }
    },
    "errors":[
      "Error while validating the event",
      "{\"schema\":\"iglu:com.snowplowanalytics.snowplow.badrows/schema_violations/jsonschema/2-0-0\",\"data\":{\"processor\":{\"artifact\":\"snowplow-micro\",\"version\":\"1.2.1\"},\"failure\":{\"timestamp\":\"2021-08-29T10:49:50.739178Z\",\"messages\":[{\"schemaKey\":\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\",\"error\":{\"error\":\"ValidationError\",\"dataReports\":[{\"message\":\"$.quantity: string found, integer expected\",\"path\":\"$.quantity\",\"keyword\":\"type\",\"targets\":[\"string\",\"integer\"]}]}}]},\"payload\":{\"enriched\":{\"app_id\":\"sh0pspr33\",\"platform\":\"web\",\"etl_tstamp\":\"2021-08-29 10:49:50.731\",\"collector_tstamp\":\"2021-08-29 10:49:50.727\",\"dvce_created_tstamp\":\"2021-08-29 10:49:50.717\",\"event\":\"unstruct\",\"event_id\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\",\"txn_id\":null,\"name_tracker\":\"spmicro\",\"v_tracker\":\"js-3.1.3\",\"v_collector\":\"ssc-2.3.1-stdout$\",\"v_etl\":\"snowplow-micro-1.2.1-common-2.0.2\",\"user_id\":\"tester\",\"user_ipaddress\":\"172.17.0.1\",\"user_fingerprint\":null,\"domain_userid\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\",\"domain_sessionidx\":1,\"network_userid\":\"3734601f-5c3d-47c5-b367-0883e1ed74e6\",\"geo_country\":null,\"geo_region\":null,\"geo_city\":null,\"geo_zipcode\":null,\"geo_latitude\":null,\"geo_longitude\":null,\"geo_region_name\":null,\"ip_isp\":null,\"ip_organization\":null,\"ip_domain\":null,\"ip_netspeed\":null,\"page_url\":\"http://localhost:8000/shop/\",\"page_title\":null,\"page_referrer\":\"http://localhost:8000/\",\"page_urlscheme\":null,\"page_urlhost\":null,\"page_urlport\":null,\"page_urlpath\":null,\"page_urlquery\":null,\"page_urlfragment\":null,\"refr_urlscheme\":null,\"refr_urlhost\":null,\"refr_urlport\":null,\"refr_urlpath\":null,\"refr_urlquery\":null,\"refr_urlfragment\":null,\"refr_medium\":null,\"refr_source\":null,\"refr_term\":null,\"mkt_medium\":null,\"mkt_source\":null,\"mkt_term\":null,\"mkt_content\":null,\"mkt_campaign\":null,\"contexts\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\",\"se_category\":null,\"se_action\":null,\"se_label\":null,\"se_property\":null,\"se_value\":null,\"unstruct_event\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\",\"tr_orderid\":null,\"tr_affiliation\":null,\"tr_total\":null,\"tr_tax\":null,\"tr_shipping\":null,\"tr_city\":null,\"tr_state\":null,\"tr_country\":null,\"ti_orderid\":null,\"ti_sku\":null,\"ti_name\":null,\"ti_category\":null,\"ti_price\":null,\"ti_quantity\":null,\"pp_xoffset_min\":null,\"pp_xoffset_max\":null,\"pp_yoffset_min\":null,\"pp_yoffset_max\":null,\"useragent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"br_name\":null,\"br_family\":null,\"br_version\":null,\"br_type\":null,\"br_renderengine\":null,\"br_lang\":\"en-US\",\"br_features_pdf\":null,\"br_features_flash\":null,\"br_features_java\":null,\"br_features_director\":null,\"br_features_quicktime\":null,\"br_features_realplayer\":null,\"br_features_windowsmedia\":null,\"br_features_gears\":null,\"br_features_silverlight\":null,\"br_cookies\":1,\"br_colordepth\":\"24\",\"br_viewwidth\":1000,\"br_viewheight\":660,\"os_name\":null,\"os_family\":null,\"os_manufacturer\":null,\"os_timezone\":\"Europe/London\",\"dvce_type\":null,\"dvce_ismobile\":null,\"dvce_screenwidth\":1920,\"dvce_screenheight\":1080,\"doc_charset\":\"UTF-8\",\"doc_width\":1003,\"doc_height\":2242,\"tr_currency\":null,\"tr_total_base\":null,\"tr_tax_base\":null,\"tr_shipping_base\":null,\"ti_currency\":null,\"ti_price_base\":null,\"base_currency\":null,\"geo_timezone\":null,\"mkt_clickid\":null,\"mkt_network\":null,\"etl_tags\":null,\"dvce_sent_tstamp\":\"2021-08-29 10:49:50.719\",\"refr_domain_userid\":null,\"refr_dvce_tstamp\":null,\"derived_contexts\":null,\"domain_sessionid\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\",\"derived_tstamp\":null,\"event_vendor\":null,\"event_name\":null,\"event_format\":null,\"event_version\":null,\"event_fingerprint\":null,\"true_tstamp\":null},\"raw\":{\"vendor\":\"com.snowplowanalytics.snowplow\",\"version\":\"tp2\",\"parameters\":[{\"name\":\"e\",\"value\":\"ue\"},{\"name\":\"duid\",\"value\":\"8a17bb29-e35c-4363-aec3-85b9b363f9bf\"},{\"name\":\"vid\",\"value\":\"1\"},{\"name\":\"co\",\"value\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\\\",\\\"data\\\":[{\\\"schema\\\":\\\"iglu:test.example.iglu/product_entity/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"sku\\\":\\\"hh456\\\",\\\"name\\\":\\\"One-size bucket hat\\\",\\\"price\\\":24.49,\\\"quantity\\\":\\\"2\\\"}},{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"id\\\":\\\"fe0dd7c7-fb0b-43a2-b299-75d20baa94ec\\\"}}]}\"},{\"name\":\"eid\",\"value\":\"36c39024-7b1b-4c2c-ae85-e95a8cb8340a\"},{\"name\":\"url\",\"value\":\"http://localhost:8000/shop/\"},{\"name\":\"refr\",\"value\":\"http://localhost:8000/\"},{\"name\":\"aid\",\"value\":\"sh0pspr33\"},{\"name\":\"tna\",\"value\":\"spmicro\"},{\"name\":\"cs\",\"value\":\"UTF-8\"},{\"name\":\"cd\",\"value\":\"24\"},{\"name\":\"stm\",\"value\":\"1630234190719\"},{\"name\":\"tz\",\"value\":\"Europe/London\"},{\"name\":\"tv\",\"value\":\"js-3.1.3\"},{\"name\":\"vp\",\"value\":\"1000x660\"},{\"name\":\"ds\",\"value\":\"1003x2242\"},{\"name\":\"ue_pr\",\"value\":\"{\\\"schema\\\":\\\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"schema\\\":\\\"iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0\\\",\\\"data\\\":{\\\"type\\\":\\\"add\\\"}}}\"},{\"name\":\"res\",\"value\":\"1920x1080\"},{\"name\":\"cookie\",\"value\":\"1\"},{\"name\":\"p\",\"value\":\"web\"},{\"name\":\"dtm\",\"value\":\"1630234190717\"},{\"name\":\"uid\",\"value\":\"tester\"},{\"name\":\"lang\",\"value\":\"en-US\"},{\"name\":\"sid\",\"value\":\"13c8f5ac-d999-4923-940d-b39f7b74aa94\"}],\"contentType\":\"application/json\",\"loaderName\":\"ssc-2.3.1-stdout$\",\"encoding\":\"UTF-8\",\"hostname\":\"0.0.0.0\",\"timestamp\":\"2021-08-29T10:49:50.727Z\",\"ipAddress\":\"172.17.0.1\",\"useragent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"refererUri\":\"http://localhost:8000/\",\"headers\":[\"Timeout-Access: <function1>\",\"Connection: keep-alive\",\"Host: 0.0.0.0:9090\",\"User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0\",\"Accept: */*\",\"Accept-Language: en-US, en;q=0.5\",\"Accept-Encoding: gzip\",\"Referer: http://localhost:8000/\",\"Origin: http://localhost:8000\",\"Cookie: micro=3734601f-5c3d-47c5-b367-0883e1ed74e6\",\"application/json\"],\"userId\":\"3734601f-5c3d-47c5-b367-0883e1ed74e6\"}}}}"
    ]
  }
]
```

### Filters

When querying `/micro/bad` with `POST` (`Content-Type: application/json` needs to be set in the headers of the request), it's possible to specify filters, thanks to a JSON in the data of the HTTP request.

Example of command to query the bad events: 

```bash
curl -X POST -H 'Content-Type: application/json' <IP:PORT>/micro/bad -d '<JSON>'
```

An example of JSON with filters could be:

```json
{
    "vendor":"com.snowplowanalytics.snowplow",
    "version":"tp2",
    "limit": 10
}
```

List of possible fields for the filters:

- `vendor`: vendor for the tracking event.
- `version`: version of the vendor for the tracking event.
- `limit`: limit the number of events in the response (most recent events are returned).

It's not necessary to specify all the fields in each request, only the ones that need to be used for filtering.

## /micro/reset

Sending a request to this endpoint deletes all events stored by Micro.

### HTTP method

`GET`, `POST`

### Response format

Expected:

```json
{
  "total": 0,
  "good": 0,
  "bad": 0
}
```

## /micro/iglu

> **Note:** This is available since version 1.2.0.

The `/micro/iglu` endpoint can be used in order to check whether a schema can be resolved.

Schema lookup should be in format:

```text
/micro/iglu/{vendor}/{schemaName}/jsonschema/{schemaVersion}
```

Or more specifically:

```text
/micro/iglu/{vendor}/{schemaName}/jsonschema/{model}-{revision}-{addition}
```

For example, assuming Micro running on localhost port `9090`:

```bash
curl -X GET http://localhost:9090/micro/iglu/com.myvendor/myschema/jsonschema/1-0-0
```

### HTTP Method

GET

### Response format

The JSON schema itself, if resolved:

```json
{
  "$schema":"http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description":"A template for a self-describing JSON Schema for use with Iglu",
  "self": {
    "vendor":"com.myvendor",
    "name":"myschema",
    "format":"jsonschema",
    "version":"1-0-0"
  },
  "type":"object",
  "properties": {
    "myStringProperty": {
      "type":"string"
    },
    "myNumberProperty":{
      "type":"number"
    }
  },
  "required": ["myStringProperty","myNumberProperty"],
  "additionalProperties":false
}
```

If a schema cannot be resolved, a JSON indicating the Iglu repositories searched:

```json
{
  "value": {
    "Iglu Central": {
      "errors": [{"error":"NotFound"}],
      "attempts":1,
      "lastAttempt":"2021-08-26T13:41:06.905Z"
    },
    "Iglu Client Embedded": {
      "errors": [{"error":"NotFound"}],
      "attempts":1,
      "lastAttempt":"2021-08-26T13:41:06.677Z"
    }
  }
}
```

---

# Snowplow Micro API reference
> Snowplow Micro arguments and environment variables.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-micro/

See [this guide](/docs/testing/snowplow-micro/) for learning about Snowplow Micro and getting started with it.

> **Note:** You can skip this reference page if you are running Micro [through Console](/docs/testing/snowplow-micro/console/).

You can always run Micro with the `--help` argument to find out what is supported:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --help
```

## Arguments

| Argument                              | Description                                                                                                                                                                                   |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--collector-config`                  | Configuration file for collector ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-collector-configuration))                                                          |
| `--iglu`                              | Configuration file for Iglu Client ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-iglu-resolver-configuration))                                                    |
| `-t`, `--output-tsv` _(since 1.4.0)_  | Print events in TSV format to standard output ([usage](/docs/testing/snowplow-micro/local/#exporting-events))                                                                                 |
| `-j`, `--output-json` _(since 2.4.0)_ | Print events in JSON format to standard output ([usage](/docs/testing/snowplow-micro/local/#exporting-events))                                                                                |
| `-d`, `--destination` _(since 2.4.0)_ | Send data to an HTTP endpoint instead of outputting it via standard output. Requires either `--output-tsv` or `--output-json` ([usage](/docs/testing/snowplow-micro/local/#exporting-events)) |
| `--yauaa`                             | Enable YAUAA user agent enrichment ([usage](/docs/testing/snowplow-micro/local/enrichments/#yauaa-yet-another-user-agent-analyzer))                                                           |
| `--no-storage` _(since 4.0.0)_        | Do not store the events anywhere and disable the API                                                                                                                                          |
| `--storage` _(since 4.0.0)_           | Enable PostgreSQL storage backend ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#persisting-events-across-restarts))                                                             |

## Environment variables

| Variable                     | Version | Description                                                                                                                                                                    |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MICRO_IGLU_REGISTRY_URL`    | 1.5.0+  | The URL for an additional custom Iglu registry ([usage](/docs/testing/snowplow-micro/local/schemas/#pointing-micro-to-an-iglu-registry))                                       |
| `MICRO_IGLU_API_KEY`         | 1.5.0+  | An optional API key for an Iglu registry defined with `MICRO_IGLU_REGISTRY_URL`                                                                                                |
| `MICRO_SSL_CERT_PASSWORD`    | 1.7.0+  | The password for the optional SSL/TLS certificate in `/config/ssl-certificate.p12`. Enables HTTPS ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#enabling-https)) |
| `MICRO_POSTGRESQL_PASSWORD`  | 4.0.0+  | The password for the optional PostgreSQL database ([usage](/docs/testing/snowplow-micro/local/advanced-usage/#persisting-events-across-restarts))                              |
| `MICRO_AZURE_BLOB_ACCOUNT`   | 4.0.0+  | The Azure blob storage account name to use for downloading enrichment assets                                                                                                   |
| `MICRO_AZURE_BLOB_SAS_TOKEN` | 4.0.0+  | The Azure blob storage account token to use for downloading enrichment assets                                                                                                  |

---

# Snowplow Mini Control Plane API
> Control and configure a Snowplow Mini instance through the Control Plane API with HTTP authentication.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/control-plane-api/

Snowplow Mini Control Plane API is created for controlling and configuring the Snowplow Mini instance without ssh to it.

You can use control plane from Snowplow Mini dashboard or you can send a request to a specific endpoint of the API with any HTTP client e.g. cURL

### Authentication

The Control Plane uses [HTTP basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).

This means that you need to add "-u username:password" to all `curl` commands, where `username` and `password` are as you specified in the Snowplow Mini setup.

### Current Methods

#### Service restart

```bash
/control-plane/restart-services
```

Example using `curl`:

```bash
$ curl -XPUT http://${snowplow_mini_ip}/control-plane/restart-services \       -u username:password
```

Restarts all the services running on the Snowplow Mini, including Stream Collector, Stream Enrich, Elasticsearch Loader.

This API call blocks until all the services have been restarted.

Return status 200 means that services have been successfully restarted.

#### Resetting Opensearch indices

As of 0.13.0, it is possible to reset Opensearch (or previously Elasticsearch) indices, along with the corresponding index patterns in Opensearch Dashboards, through Control Plane API.

```bash
curl -L \
-X POST '<mini-address>/control-plane/reset-service' \
-u '<username>:<password>' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service_name=elasticsearch'
```

Note that resetting deletes not only indices and patterns but also all events stored so far.

#### Restart services individually

As of 0.13.0, it is possible to restart services one by one.

```bash
curl -L \
-X PUT '<mini-address>/control-plane/restart-service' \
-u '<username>:<password>' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service_name=<service_name>'
```

where `service_name` can be one of the following: `collector`, `enrich`, `esLoaderGood`, `esLoaderBad`, `iglu`, `kibana`, `elasticsearch`.

#### Configuring telemetry

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information on telemetry.

HTTP GET to get current configuration

```bash
curl -L -X GET '<mini-address>/control-plane/telemetry' -u '<username>:<password>'
```

HTTP PUT to set it (use true or false as value of key `disable` to turn it on or off)

```bash
curl -L -X PUT '<mini-address>/control-plane/telemetry' -u '<username>:<password>' -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'disable=false'
```

#### Adding external Iglu Server

```bash
/control-plane/external-iglu
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/external-iglu \
  -d "uri=${external_iglu_uri}&apikey=${external_iglu_server_apikey}&vendor_prefix=${vendor_prefix}&name=${iglu_server_name}&priority=${priority}" \
  -u username:password
```

Adds given pieces of information of the external Iglu Server to iglu resolver json file to use it with Stream Enrich.

Note that a lower priority number means that the repo is ranked higher in terms of priority.

Return status 200 means that pieces of information are added to iglu resolver json file and Stream Enrich is restarted successfully.

**Note**: Apikey must follow the UUID format.

#### Uploading custom enrichments

```bash
/control-plane/enrichments
```

Example using `curl`:

```bash
curl http://${snowplow_mini_ip}/control-plane/enrichments \
  -F "enrichmentjson=@${path_of_the_custom_enrichment_dir}" \
  -u username:password
```

Upload custom enrichment json file to enrichments directory and restarts the Stream Enrich to activate uploaded custom enrichment.

Return status 200 means that custom enrichment json file is placed in the enrichments directory and Stream Enrich is restarted successfully.

#### Adding apikey for local Iglu Server

```bash
/control-plane/local-iglu-apikey
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/local-iglu-apikey \
  -d "local_iglu_apikey=${new_local_iglu_apikey}" \
  -u username:password
```

Adds provided apikey to the section of local Iglu Server in the iglu resolver json config. Restarts to Stream Enrich to activate the changes.

Return status 200 means that apikey is added and Stream Enrich is restarted successfully.

**Note**: Apikey must follow the UUID format.

#### Changing credentials for basic HTTP authentication

As of version 0.13.0, this endpoint doesn't accept new passwords shorter than 8 chars and with a score lower than 4 according to [zxcvbn](https://pkg.go.dev/github.com/trustelem/zxcvbn)

```bash
/control-plane/credentials
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/credentials \
  -d "new_username=${new_username}&new_password=${new_password}" \
  -u username:password
```

Changes the credentials for basic HTTP authentication.

You will get always empty reply from the server because caddy server will be restarted after credentials are submitted and the connection will be lost until caddy server is up and running.

#### Add domain name

```bash
/control-plane/domain-name
```

Example using `curl`:

```bash
curl -XPOST http://${snowplow_mini_ip}/control-plane/domain-name \
  -d "domain_name=${registered_domain_name}" \
  -u username:password
```

Adds domain name for Snowplow Mini instance. After adding the domain name, your connection will be secured with TLS automatically. Make sure that given domain name is resolving to Snowplow Mini instance IP address.

You will get always empty reply from the server because caddy server will be restarted after the domain name is submitted and the connection will be lost until caddy server is up and running.

#### Get Snowplow Mini version

```bash
/control-plane/version
```

Example using `curl`:

```bash
curl -XGET http://${snowplow_mini_ip}/control-plane/version \
  -u username:password
```

Returns version of the running Snowplow Mini instance.

#### Uploading Iglu Server configuration

```bash
/control-plane/iglu-config
```

Example using `curl`:

```bash
curl http://${snowplow_mini_ip}/control-plane/iglu-config \
  -F "igluserverhocon=@${path_of_the_iglu_server_config}" \
  -u username:password
```

Uploads Iglu Server configuration file and restarts the Iglu Server to activate uploaded configuration.

Return status 200 means that configuration is uploaded and Iglu Server is restarted successfully.

---

# Introduction to Snowplow Mini
> Snowplow Mini is a single-instance development environment for testing tracker updates and schema changes.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/

[Snowplow Mini](/docs/api-reference/snowplow-mini/) is a single-instance version of Snowplow that primarily serves as a development environment, giving you a quick way to debug tracker updates and changes to your schema and pipeline configuration.

> **Tip:** For new testing environments, we recommend using [Snowplow Micro](/docs/testing/snowplow-micro/), which you can [deploy through Console](/docs/testing/snowplow-micro/console/) or [run locally](/docs/testing/snowplow-micro/local/). New Snowplow Mini deployments are no longer available through Console.

You might use Snowplow Mini when:

- You've updated a schema in your Development environment and wish to send some test events against it before promoting it to Production
- You want to enable an Enrichment in a test environment before enabling it on Production

## Getting started

New Snowplow Mini instances are no longer available through Console. For new development environments, use [Snowplow Micro](/docs/testing/snowplow-micro/console/) instead.

For Snowplow Self-Hosted, see the setup guides for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/).

## Conceptual diagram

![](/assets/images/image-72cd90387cfe692a867dd033688a5254.png)

The diagram above illustrates how Snowplow Mini (top) works alongside your Production pipeline (bottom).

By pointing your tracker(s) to the Collector on your Snowplow Mini you can send events from your applications development and QA environments to Snowplow Mini for testing.

Once you are happy with the changes you have made you would then change the trackers in your application to point to the Collector on your Production pipeline.[](https://github.com/snowplow/snowplow-mini#features)

## Features of Snowplow Mini

- Data is tracked and processed in real time
- Your Snowplow Mini speaks to your [Schema registries](/docs/fundamentals/schemas/#iglu-schema-repository) to allow events to be sent against your custom schemas
- Data is validated during processing
- Data is loaded into OpenSearch and can be queried directly or through the OpenSearch Dashboard
- Successfully processed events and failed events are in distinct good and bad indexes

## Topology

Snowplow-Mini runs several distinct applications on the same box which are all linked by NSQ topics. In a production deployment each instance could be an Autoscaling Group and each NSQ topic would be a distinct Kinesis Stream.

- Scala Stream Collector:

  - Starts server listening on `http://< sp mini public ip>/` which events can be sent to.
  - Sends "good" events to the `RawEvents` NSQ topic
  - Sends "bad" events to the `BadEvents` NSQ topic

- Stream Enrich:

  - Reads events in from the `RawEvents` NSQ topic
  - Sends events which passed the enrichment process to the `EnrichedEvents` NSQ topic
  - Sends events which failed the enrichment process to the `BadEvents` NSQ topic

- OpenSearch Sink Good:

  - Reads events from the `EnrichedEvents` NSQ topic
  - Sends those events to the `good` OpenSearch index
  - On failure to insert, writes errors to `BadElasticsearchEvents` NSQ topic

- OpenSearch Sink Bad:

  - Reads events from the `BadEvents` NSQ topic
  - Sends those events to the `bad` OpenSearch index
  - On failure to insert, writes errors to `BadElasticsearchEvents` NSQ topic

These events can then be viewed in Kibana at `http://< sp mini public ip>/kibana`.

---

# Set up Snowplow Mini on AWS
> Deploy Snowplow Mini on AWS for a single-instance testing environment.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/setup-guide-for-aws/

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates

All setup for Snowplow Mini is done within the AWS Console and will incur small amounts of running costs, depending on the size of the EC2 instance you select.

We offer Snowplow Mini in 3 different sizes. To decide on which size of Snowplow Mini to choose, read on.

## large & xlarge & xxlarge

Mini is available in 3 different sizes:

- `large` : Opensearch has `4g` heap size and Snowplow apps has `0.5g` heap size. Recommended machine RAM is `8g`.
- `xlarge` : Double the large image. Opensearch has `8g` heap size and Snowplow apps has `1.5g` heap size. Recommended machine RAM is `16g`.
- `xxlarge` : Double the xlarge image. Opensearch has `16g` heap size and Snowplow apps has `3g` heap size. Recommended machine RAM is `32g`.

This service is available as an EC2 image within the AWS Community AMIs in the following regions: `ap-northeast-1`, `ap-northeast-2`, `ap-south-1`, `ap-southeast-1`, `ap-southeast-2`, `ca-central-1`, `eu-central-1`, `eu-west-1`, `eu-west-2`, `sa-east-1`, `us-east-1`, `us-east-2`, `us-west-1` and `us-west-2`.

Version 0.25.1 (recommended) comes with:

- Snowplow Collector NSQ 3.7.0
- Snowplow Enrich NSQ 6.7.1
- Snowplow Elasticsearch Loader 2.1.3
- Snowplow Iglu Server 0.14.0
- Opensearch 3.3.0
- Opensearch Dashboards 3.3.0
- Postgresql 16.10
- NSQ v1.3.0

Note: All services are configured to start automatically so everything should happily survive restarts/shutdowns.

To understand the flow of data please refer to the following diagram:

![This image has an empty alt attribute; its file name is snowplow-mini-topology.jpg](/assets/images/snowplow-mini-topology-95da73899375d477bfe132b2bcdb0e19.jpg)

**IAM**

Create a role with the following configuration

- Step 1: For `Select type of trusted entity` , select `EC2`
- Step 2.1: For `Attach permissions policies` , create a policy with the following

```json
{
  "Version" : "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": ["*"]
    }
  ]
}
```

- Step 2.2: In step 2 of role creation, select the policy created in the previous step
- Step 3: Tags are optional
- Step 4: Fill in the role name and create it.

**CloudWatch**

Create a log group named `snowplow-mini` so that Mini can emit logs to this log group.

Mini will not function properly if a log group with that name isn't found.

## Security Group

In the EC2 Console UI select `Security Groups` from the panel on the left.

Select the `Create Security Group` button and fill in the name, description and what VPC you want to attach it to.

You will then need to add the following InBound rules:

![snowplow-mini-security-group-setup](/assets/images/security-groups-setup-f8d299c2b2b111f0dbd2351e83ec119e.png)

- Custom TCP Rule | Port Range (80)

- CIDR range `0.0.0.0/0`

- Custom TCP Rule | Port Range (443)

- CIDR range `0.0.0.0/0`

- SSH (optional):

  - Custom TCP Rule | Port Range (22)
  - CIDR range `{{ YOUR IP HERE }}/32`

For OutBound you can leave the default to allow everything out.

## Choose AMI

In the EC2 Console UI select the `Launch Instance` button then select the `Community AMIs` button. In the search bar enter `snowplow-mini-0.25.1` to find the needed AMI and then select it.

## Choose Instance Type

AMI names explicitly specify which instance type to use.

- `0.25.1-large` needs `t2.large`
- `0.25.1-xlarge` needs `t2.xlarge`
- `0.25.1-xxlarge` needs `t2.2xlarge`

## Configure Instance

- Select the IAM role created above.

- If you created your Security Group in a different VPC than the default you will need to select the same VPC in the Network field.

**NOTE**: If you select a custom VPC ensure that you select `Enable` for the Auto-assign Public IP option.

## Add Storage

Depending on how long you intend to run Snowplow Mini and how much data you intend to send/store you will need to change the size of the block store accordingly.

For basic testing and debugging;

- 20-50 Gb should suffice for `large`
- 50-100 Gb should suffice for `xlarge`
- 100-200 Gb should suffice for `xxlarge`

We also recommend changing the `Volume Type` to GP2 from Magnetic for a smoother experience.

## Tag Instance

Add any tags you like here.

## Configure Security Group

Select the Security Group you created [above](#security-group).

## Review

Press the `Launch` button and select an existing key-pair, or create a new one, if you want to be able to SSH into the box.

**Telemetry notice**

By default, Snowplow collects telemetry data for Mini (since version 0.13.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to disable telemetry, you can do so via the [API](/docs/api-reference/snowplow-mini/control-plane-api/#configuring-telemetry).

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Set up Snowplow Mini on GCP
> Deploy Snowplow Mini on GCP for a single-instance testing environment.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/setup-guide-for-gcp/

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates (because they can)

Version 0.25.1 (recommended) comes with:

- Snowplow Collector NSQ 3.7.0
- Snowplow Enrich NSQ 6.7.1
- Snowplow Elasticsearch Loader 2.1.3
- Snowplow Iglu Server 0.14.0
- Opensearch 3.3.0
- Opensearch Dashboards 3.3.0
- Postgresql 16.10
- NSQ v1.3.0

Note: All services are configured to start automatically so everything should happily survive restarts/shutdowns.

To understand the flow of data please refer to the following diagram:

![](/assets/images/snowplow-mini-topology-95da73899375d477bfe132b2bcdb0e19.jpg)

## Importing public tarballs to a GCP project

Our offering for GCP is 3 compressed tarballs for 3 different sized Snowplow Mini, produced through `gcloud`'s [`export`](https://cloud.google.com/sdk/gcloud/reference/compute/images/export) command. Simply put, they are Virtual Disk images exported in GCP format, a `disk.raw` file that has been tarred and gzipped.

To use them within GCP console, you need to import a tarball of your choice into your GCP project. You can use `gcloud` CLI utility to do that.

Browse [GCP docs](https://cloud.google.com/sdk/docs/quickstarts) to get started with `gcloud`.

Assuming you have `gcloud` setup and configured for your GCP project, use `gcloud`'s [`create`](https://cloud.google.com/sdk/gcloud/reference/compute/images/create) command to import a tarball of your choice into your GCP project.

A sample usage would be as following.

```bash
gcloud compute images create \
imported-sp-mini \
--source-uri \
https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-25-1-large-1771418956.tar.gz
```

Note that `imported-sp-mini` is a name of your choice for destination image and above URI is for large image, change it with your preferred version of Snowplow Mini.

Version 0.25.1 (recommended)

| L / 2 vCPUs                                                                                        | XL / 4 vCPUs                                                                                         | XXL / 8 vCPUs                                                                                          |
| -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| [large](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-25-1-large-1771418956.tar.gz) | [xlarge](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-25-1-xlarge-1771418987.tar.gz) | [xxlarge](https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-25-1-xxlarge-1771418969.tar.gz) |

You can find more about `gcloud compute images create` command [here](https://cloud.google.com/sdk/gcloud/reference/compute/images/create) for additional parameters.

After importing our tarball of your choice into your project, you should see it under `Images` on `Compute Engine`.

To decide on which size of Snowplow Mini to choose, read on.

## large & xlarge & xxlarge

Mini is available in 3 different sizes:

- `large` : Opensearch has `4g` heap size and Snowplow apps has `0.5g` heap size. Recommended machine RAM is `8g`.
- `xlarge` : Double the large image. Opensearch has `8g` heap size and Snowplow apps has `1.5g` heap size. Recommended machine RAM is `16g`.
- `xxlarge` : Double the xlarge image. Opensearch has `16g` heap size and Snowplow apps has `3g` heap size. Recommended machine RAM is `32g`.

## Create instance

Go to `Compute Engine` on GCP console, select `Images` from menu on the left. You should see your imported image on the list. Select it then you should see `CREATE INSTANCE` button at the top of the page. Click on it.

![](/assets/images/create-instance-cc7dee2edb679cbcfab716d3f068aa49.png)

![](/assets/images/create-instance-2-b23317e188861463d0a697e7e6698441.png)

![](/assets/images/create-instance-3-7397ccd9528d0226acb3bf709bdccf5a.png)

Click `Create`.

These images show setup for `large` image. To setup `xlarge` or `xxlarge`, you should increase memory per above explanation of different sizes.

**Telemetry notice**

By default, Snowplow collects telemetry data for Mini (since version 0.13.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to disable telemetry, you can do so via the [API](/docs/api-reference/snowplow-mini/control-plane-api/#configuring-telemetry).

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

---

# Snowplow Mini usage guide
> Learn how to use Snowplow Mini for testing, debugging trackers, and exploring Snowplow features.
> Source: https://docs.snowplow.io/docs/api-reference/snowplow-mini/usage-guide/

## Overview

Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:

1. Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
2. Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
3. Giving people running Snowplow a quick way to debug tracker updates

Jump to [First Time Usage](#first-time-usage) if it is your first time with a Mini.

## Upgrading

Until version 0.15.0, Snowplow data was loaded to Elasticsearch 6.x in the Mini. However, a [licensing change](https://www.elastic.co/blog/licensing-change) in Elasticsearch prevented us from upgrading it to more recent versions. To make sure we stay up to date with important security fixes, we've decided to replace Elasticsearch with [Opensearch](https://opensearch.org/). Also, Kibana is replaced with [Opensearch Dashboards](https://opensearch.org/docs/latest/dashboards/index/).

[Opensearch](https://opensearch.org/) is a fork of open source Elasticsearch 7.10. Therefore this change shouldn't affect Mini users much. To minimize the impact further, we've tried to make as minimal a change as possible. In Mini, you can still access Opensearch with the `/elasticsearch` endpoint and Opensearch Dashboards with the `/kibana` endpoint.

The only breaking change this migration would bring is the removal of mapping types. It means that you don't have to provide a mapping type in your search queries anymore when accessing your data in good or bad indices. For example, good events count can be found using the following endpoint in previous versions: `/elasticsearch/good/good/_count`. Starting with 0.15.0, it can be found using this endpoint: `/elasticsearch/good/_count`.

## First time usage

This section is dedicated to the steps that need performing when accessing the Snowplow Mini instance for the first time.

### Connecting to the instance for the first time

You can access the Snowplow Mini instance at the `http://[public_dns]/home` address. While accessing Snowplow Mini services, HTTP authentication is required. As a result, you will be prompted for credentials which are `USERNAME_PLACEHOLDER` and `PASSWORD_PLACEHOLDER` by default.

You **should** change these default credentials to something to your liking by going to the Control Plane tab (the last one) and fill the "Change username and password for basic http authentication" form towards the bottom.

**Note that only alphanumeric passwords are supported.**

You will then be prompted for those new credentials.

### Changing the super API key for the local Iglu schema registry

As as second step, you should change the super API key for the Iglu schema registry that is bundled with Snowplow Mini. This API key can be changed via the Control Plane tab.

Given that this API key must be a UUID v4, you will need to generate one by running `uuidgen` at the command line, or by using an online UUID generator like [this one](https://www.uuidgenerator.net/). Make a note of this UUID, you'll need it to upload your own event and context schemas to Snowplow Mini in the next subsection.

### Generating a pair of read/write API keys for the local Iglu schema registry

> **Note:** Mini 0.8.0 comes bundled with Iglu Server 0.6.1 which introduced a couple of changes regarding this section.
>
> - Swagger UI of Iglu Server is deprecated, however Iglu Server still serves at `/iglu-server` endpoint.
> - `POST /api/auth/keygen` no longer supports query parameter to provide vendor prefix. Use POST raw data request instead.

To add schemas to the Iglu repository bundled with Snowplow Mini, you have to create a dedicated pair of API keys. There are 2 options:

- Use igluctl’s [server keygen](/docs/api-reference/iglu/igluctl-2/#server-keygen) subcommand
- Use any HTTP client e.g. cURL

#### Option 1

First, [download igluctl](/docs/api-reference/iglu/igluctl-2/#downloading-and-running-igluctl).

Following is a sample execution where `com.acme` is the vendor prefix for which we'll upload our schemas, `mini-address` is the URL of our mini and `53b4c441-84f7-467e-af4c-074ced53eb20` is an example super API key you would have created in the previous steps.

```bash
/path/to/igluctl server keygen --vendor-prefix com.acme mini-address/iglu-server 53b4c441-84f7-467e-af4c-074ced53eb20
```

#### Option 2

You can also use `cURL` to interact with Iglu Server:

```text
curl --location --request POST 'mini-address/iglu-server/api/auth/keygen' \
  --header 'apikey: 1b5d0459-3492-451c-aab1-7f74cbe12112' \
  --header 'Content-Type: application/json' \
  --data-raw '{"vendorPrefix":"com.acme"}'
```

should return a read key and a write key.

```json
{
  "read":"bfa90866-ab14-4b92-b6ef-d421fd688b54",
  "write":"6175aa41-d3a7-4e4f-9fb4-3a170f3c6c16"
}
```

### Copying your Iglu repository to Snowplow Mini (optional)

To test and send non-standard Snowplow events such as your own custom-contexts and unstructured events you can load them into the Iglu repository local to the Snowplow Mini instance.

1. Get a local copy of your Iglu repository which contains your schemas. This should be modelled after [this folder](https://github.com/snowplow/iglu-central/tree/master/schemas)

2. [Download igluctl](/docs/api-reference/iglu/igluctl-2/#downloading-and-running-igluctl).

3. Run the executable with the following input:

- The address of the Iglu repository: `http://[public_dns]/iglu-server`
- The Super API Key you created previously
- The path to your schemas For example to load the `iglu-central` repository into Iglu Server:

```bash
/path/to/igluctl static push iglu-central/schemas http://[public_dns]/iglu-server 980ae3ab-3aba-4ffe-a3c2-3b2e24e2ffce --public
```

Note: this example assumes the `iglu-central` repository has been cloned in the same directory as where executable is run.

1. After uploading the schemas, you will need to clear the cache with the restart button under the Control Plane tab in the Snowplow Mini dashboard.

### Setting up HTTPS (optional)

If you want to use HTTPS to connect to Snowplow Mini, you need to submit a domain name via the Control Plane. Make sure that the domain name you submit is redirected to the IP of the server Snowplow Mini is running from.

## Sending events to Snowplow Mini

Now that the first time usage steps have been dealt with, you can send some events!

### Example events

An easy way to quickly send a few test events is to use our example web page.

1. Open up the Snowplow Mini UI at: `http://[public_dns]/home`
2. Login with username and password which you choose in step 2.1
3. Select the `Example Events` tab
4. Press the event triggering buttons on the page!

### Events from tracker

You can instrument any other Snowplow tracker by specifying the collector URL as the public DNS of the Snowplow Mini instance.

## Accessing the Opensearch API

Snowplow Mini makes the Opensearch (or previously Elasticsearch) HTTP API available at `http://[public_dns]/elasticsearch`, you can check it's working by:

- Checking the Opensearch API is available:

  - `curl --user username:password http://[public_dns]/elasticsearch`
  - You should see a `200 OK` response

- Checking the number of good events we sent in step 3:

  - `curl --user username:password http://[public_dns]/elasticsearch/good/_count`
  - You should see the appropriate count of sent events

## Viewing the data in Opensearch Dashboards

Data sent to Snowplow Mini will be processed and loaded into Opensearch in real time. In turn, it will be made available in Opensearch Dashboards. To view the data in Opensearch Dashboards, navigate in your browser to `mini-public-address/kibana`.

### Index patterns

Snowplow Mini comes with two index patterns:

- `good` : For good events, indexed on `collector_tstamp`
- `bad` : For bad events, indexed in `data.failure.timestamp`

### Discover your data

Browse to `mini-public-address/kibana` , once Opensearch Dashboards is loaded, you should be able to view most recently sent good events via the discover interface:

You can then inspect any individual event data in the UI by unfolding a payload:

![](/assets/images/Screen-Shot-2020-04-13-at-13.20.22-8410d743b7a7a1261de9528d561c8aa7.jpg)

If you want to inspect bad events, click on `good`, placed towards top left of the screen and select `bad` from drop-down list.

![](/assets/images/Screen-Shot-2020-04-13-at-13.32.26-1b9f1eaae978503d3f587585563148df.jpg)

Unfold any payload to inspect a bad event in detail.

![](/assets/images/Screen-Shot-2020-04-13-at-13.23.16-970d83b883ef507d69792cf0f65de9eb.jpg)

## Resetting Opensearch indices

As of 0.13.0, it is possible to reset Opensearch (or previously Elasticsearch) indices, along with the corresponding index patterns in Opensearch Dashboards, through Control Plane API.

```bash
curl -L \
-X POST '<mini-address>/control-plane/reset-service' \
-u '<username>:<password>' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service_name=elasticsearch'
```

Note that resetting deletes not only indices and patterns but also all events stored so far.

## Restart services individually

As of 0.13.0, it is possible to restart services one by one.

```bash
curl -L \
-X PUT '<mini-address>/control-plane/restart-service' \
-u '<username>:<password>' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'service_name=<service_name>'
```

where `service_name` can be one of the following: `collector`, `enrich`, `esLoaderGood`, `esLoaderBad`, `iglu`, `kibana`, `elasticsearch`.

## Configuring telemetry

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information on telemetry.

HTTP GET to get current configuration

```bash
curl -L -X GET '<mini-address>/control-plane/telemetry' -u '<username>:<password>'
```

HTTP PUT to set it (use true or false as value of key `disable` to turn it on or off)

```bash
curl -L -X PUT '<mini-address>/control-plane/telemetry' -u '<username>:<password>' -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'disable=false'
```

## Uploading custom enrichments

You can add new custom enrichments via the Control Plane tab. The only thing you have to do is submit the enrichment configuration file which you created according to the documentation in [Available Enrichments](/docs/pipeline/enrichments/available-enrichments/). If the enrichment relies on additional schemas these should be uploaded to the Iglu repository.

## Adding a custom schema

Since Mini 0.8.0 deprecated Swagger UI of Iglu Server, we have 2 options:

- Use igluctl’s [static push](/docs/api-reference/iglu/igluctl-2/#static-push) subcommand to put our custom schema into the Iglu Server
- Use any HTTP client e.g. cURL

#### Option 1

First, [download igluctl](/docs/api-reference/iglu/igluctl-2/#downloading-and-running-igluctl).

Following is a sample execution where `path-to-schema(s)` is the path to custom schema(s) , `mini-address` is the URL of our mini and `53b4c441-84f7-467e-af4c-074ced53eb20` is an example super API key you would have created in the previous steps.

```bash
/path/to/igluctl static push path-to-schema(s) mini-address/iglu-server 53b4c441-84f7-467e-af4c-074ced53eb20
```

#### Option 2

You can also use `cURL` to interact with Iglu Server:

```bash
curl mini-address/iglu-server/api/schemas -X POST \
  -H "apikey: YOUR_APIKEY" -d '{"json": YOUR_JSON}'
```

The command will produce a response like this one, if no errors are encountered:

```json
{
  "message": "Schema created",
  "updated": false,
  "location": "iglu:com.acme/ad_click/jsonschema/1-0-0",
  "status":201
}
```

## Adding an external Iglu repository

If you already have an external Iglu repository available, instead of copying it inside the Iglu repository bundled with the Snowplow Mini instance as shown in 2.4, you can add it directly with the Control Plane's `Add an external Iglu repository` form. Note that if you're using a static repository hosted on S3, you can omit providing an API key.

## Runtime metrics

Mini 0.12.0 introduced /metrics endpoint powered by [cAdvisor](https://github.com/google/cadvisor) . You can also find the link to metrics on the home page under Quicklinks header.

It's been possible to observe runtime metrics of a Mini instance by looking at AWS/GCP monitoring dashboards however internal services' individual metrics weren't exposed, making it more difficult to diagnose issues.

Exposing runtime metrics such as CPU, RAM and Network usage of the internal services in real time will make Mini more transparent, hopefully making it easier to understand what's going on under the hood.

## Logs

As of Mini 0.12.0, application logs of the Mini sub-services are exported to Cloudwatch on AWS and Cloud Logging on GCP.

On AWS, each individual service emits its' logs under a specific log stream within `snowplow-mini` log group.

On GCP, we need to make use of filters to see the logs of a specific component. The recommended approach is as following:

- On GCP console, go to Logging > Logs Viewer
- Under Query Builder, select resource
- Under `VM instance`, select the instance Mini is running at
- Click on `Add`

Click on `Run Query` and we should see logs of all services combined.

To see the logs of a specific component, add the following filter to the query:

jsonPayload.container.name="/service-name"

where service-name can be one of the following: `elasticsearch`, `kibana`, `elasticsearch-loader-good`, `elasticsearch-loader-bad`, `nsqlookupd`, `nsqd`, `nsqadmin`, `scala-stream-collector-nsq`, `stream-enrich-nsq`

An example query looks as following:

resource.type="gce\_instance"

resource.labels.instance\_id="3778299199368430127"

jsonPayload.container.name="/elasticsearch"

---

# Collector configuration reference
> Complete configuration reference for the Collector HOCON config file, including common options, sink-specific settings, cookie management, networking, and TLS configuration.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/configure/

This is a complete list of the options that can be configured in the collector HOCON config file. The [example configs in github](https://github.com/snowplow/stream-collector/tree/master/examples) show how to prepare an input file. Some features are described in more detail at the bottom of this page.

### License

The collector is released under the [Snowplow Limited Use License](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

To accept the terms of license and run the collector, set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable. Alternatively, you can configure the `collector.license.accept` option, like this:

```hcl
collector {
  license { accept = true }
}
```

### Common options

| parameter                                                                         | description                                                                                                                                                                                                                                                                                                                                                     |
| --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.interface`                                                             | Required. E.g. `0.0.0.0`. The collector listens for http requests on this interface.                                                                                                                                                                                                                                                                            |
| `collector.port`                                                                  | Required. E.g. `80`. The collector listens for http requests on this port.                                                                                                                                                                                                                                                                                      |
| `collector.ssl.enable`                                                            | Optional. Default: `false`. The collector will also listen for https requests on a different port.                                                                                                                                                                                                                                                              |
| `collector.ssl.port`                                                              | Optional. Default: `443`. The port on which to listen for https requests.                                                                                                                                                                                                                                                                                       |
| `collector.ssl.redirect`                                                          | Optional. Default: `false`. If enabled, the collector redirects http requests to the https endpoint using a `301` status code.                                                                                                                                                                                                                                  |
| `collector.hsts.enable` _(since 3.1.0)_                                           | Default: `false`. Whether to send an [HSTS header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security).                                                                                                                                                                                                                        |
| `collector.hsts.maxAge` _(since 3.1.0)_                                           | Default: `365 days`. The maximum age for the HSTS header.                                                                                                                                                                                                                                                                                                       |
| `collector.paths`                                                                 | Optional. More details about this feature below. This is for customising the collector's endpoints. You can also map any valid (ie, two-segment) path to one of the three default paths.                                                                                                                                                                        |
| `collector.p3p.policyRef`                                                         | Optional. Default: `/w3c/p3p.xml`. Configures the p3p http header.                                                                                                                                                                                                                                                                                              |
| `collector.p3p.CP`                                                                | Optional. Default: `NOI DSP COR NID PSA OUR IND COM NAV STA`. Configures the p3p http header.                                                                                                                                                                                                                                                                   |
| `collector.crossDomain.enabled`                                                   | Optional. Default: `false`. If enabled, `/crossdomain.xml` endpoint returns a cross domain policy file.                                                                                                                                                                                                                                                         |
| `collector.crossDomain.domains`                                                   | Optional. Default: `[*]` meaning the cross domain policy file allows all domains. This can also be changed to a list of domains.                                                                                                                                                                                                                                |
| `collector.crossDomain.secure`                                                    | Optional. Default: `true`. Whether cross domain policy file grants access to just HTTPS or both HTTP and HTTPS sources                                                                                                                                                                                                                                          |
| `collector.cookie.enabled`                                                        | Optional. Default: `true`. The collector sets a cookie to set the user's network user id. Changing this to `false` disables setting cookies. Regardless of this setting, if the collector receives a request with the custom `SP-Anonymous:*` header, no cookie will be set. You can control whether this header is set or not in your tracking implementation. |
| `collector.cookie.expiration`                                                     | Optional. Default: `365 days`. Expiry of the collector's cookie.                                                                                                                                                                                                                                                                                                |
| `collector.cookie.name`                                                           | Optional. Default: `sp`. Name of the collector's cookie.                                                                                                                                                                                                                                                                                                        |
| `collector.cookie.domains`                                                        | Optional. Default: no domains. There is more details about this feature below. This is for fine control over the cookie's domain attribute.                                                                                                                                                                                                                     |
| `collector.cookie.fallbackDomain`                                                 | Optional. If set, the fallback domain will be used for the cookie if none of the `Origin` header hosts matches the list of cookie domains.                                                                                                                                                                                                                      |
| `collector.cookie.secure`                                                         | Optional. Default: `true`. Sets the `secure` property of the cookie.                                                                                                                                                                                                                                                                                            |
| `collector.cookie.httpOnly`                                                       | Optional. Default: `true`. Sets the `httpOnly` property of the cookie. We recommend `true` because `httpOnly` cookies are allowed a longer expiry time by web browsers.                                                                                                                                                                                         |
| `collector.cookie.sameSite`                                                       | Optional. Default: `None`. Sets the `sameSite` property of the cookie. Possible values: `Strict`, `Lax`, `None`.                                                                                                                                                                                                                                                |
| `collector.cookie.clientCookieName` (since 3.4.0)                                 | Optional. Default: not set. If a name is specified (e.g. `sp_client`), the collector sets an extra cookie with that name, `httpOnly=false` and the same value as the main cookie (network user id). This is useful if you need to access the network user id on the client side using JavaScript.                                                               |
| `collector.doNotTrackCookie.enabled`                                              | Optional. Default: `false`. If enabled, the collector respects a "do not track" cookie. If the cookie is present, it returns a `200` status code but it does not log the request to the output queue.                                                                                                                                                           |
| `collector.doNotTrackCookie.name`                                                 | Required when `doNotTrackCookie` feature is enabled. Configures the name of the cookie in which to check if tracking is disabled.                                                                                                                                                                                                                               |
| `collector.doNotTrackCookie.value`                                                | Required when `doNotTrackCookie` feature is enabled. Can be a regular expression. The value of the cookie must match this expression in order for the collector to respect the cookie.                                                                                                                                                                          |
| `collector.cookieBounce.enabled`                                                  | Optional. Default: `false`. When enabled, when the cookie is missing, the collector performs a redirect to itself to check if third-party cookies are blocked using the specified name. If they are indeed blocked, `fallbackNetworkId` is used instead of generating a new random one.                                                                         |
| `collector.cookieBounce.name`                                                     | Optional. Default: `n3pc`. Name of the request parameter which will be used on redirects checking that the third-party cookies work.                                                                                                                                                                                                                            |
| `collector.cookieBounce.fallbackNetworkUserId`                                    | Optional. Default: `00000000-0000-4000-A000-000000000000`. Network user id to use when third-party cookies are blocked.                                                                                                                                                                                                                                         |
| `collector.cookieBounce.forwardedProtocolHeader`                                  | Optional. E.g. `X-Forwarded-Proto`. The header containing the originating protocol for use in the bounce redirect location. Use this if behind a load balancer that performs SSL termination.                                                                                                                                                                   |
| `collector.enableDefaultRedirect`                                                 | Optional. Default: `false`. When enabled, the collector's `/r` endpoint returns a `302` status code with a redirect back to a url specified with the `?u=` query parameter.                                                                                                                                                                                     |
| `collector.redirectDomains` (since _2.5.0_)                                       | Optional. Default: empty. Domains which are valid for collector redirects. If empty then redirects are allowed to any domain. Must be an exact match.                                                                                                                                                                                                           |
| `collector.redirectMacro.enabled`                                                 | Optional. Default: `false`. When enabled, the redirect url passed via the `u` query parameter is scanned for a `placeholder` token. All occurrences of the placeholder are substituted with the cookie's network user id.                                                                                                                                       |
| `collector.redirectMacro.placeholder`                                             | Optional. Default: `${SP_NUID}`.                                                                                                                                                                                                                                                                                                                                |
| `collector.rootResponse.enabled`                                                  | Optional. Default: `false`. Enable custom response handling for the root `"/"` path.                                                                                                                                                                                                                                                                            |
| `collector.rootResponse.statusCode`                                               | Optional. Default: `302`. The http status code to use when root response is enabled.                                                                                                                                                                                                                                                                            |
| `collector.rootResponse.headers`                                                  | Optional. A map of key value pairs to include in the root response headers.                                                                                                                                                                                                                                                                                     |
| `collector.rootResponse.body`                                                     | Optional. The http response body to use when root response is enabled.                                                                                                                                                                                                                                                                                          |
| `collector.cors.accessControlMaxAge`                                              | Optional. Default: `60 minutes`. Configures how long a the results of a preflight request can be cached by the browser. `-1` seconds disables the cache.                                                                                                                                                                                                        |
| `collector.preTerminationPeriod` (since _2.5.0_)                                  | Optional. Default: `10 seconds`. Configures how long the collector should pause after receiving a sigterm before starting the graceful shutdown. During this period the collector continues to accept new connections and respond to requests.                                                                                                                  |
| `collector.prometheusMetrics.enabled` (deprecated since _2.6.0_)                  | Optional. Default: `false`. When enabled, all requests are logged as prometheus metrics and the `/metrics` endpoint returns the report about the metrics.                                                                                                                                                                                                       |
| `collector.prometheusMetrics.durationBucketsInSeconds` (deprecated since _2.6.0_) | Optional. E.g. `[0.1, 3, 10]`. Custom buckets for the `http_request_duration_seconds_bucket` duration prometheus metric.                                                                                                                                                                                                                                        |
| `collector.telemetry.disable`                                                     | Optional. Set to `true` to disable [telemetry](/docs/get-started/self-hosted/telemetry/).                                                                                                                                                                                                                                                                       |
| `collector.telemetry.userProvidedId`                                              | Optional. See [here](/docs/get-started/self-hosted/telemetry/#how-can-i-help) for more information.                                                                                                                                                                                                                                                             |
| `collector.compression.enabled` (since _3.6.0_)                                   | Optional. Default: `false`. Enable compression on the output. Compression should only be enabled with Enrich >=6.1.0.                                                                                                                                                                                                                                           |
| `collector.compression.type` (since _3.6.0_)                                      | Optional. Default: `zstd`. Compression algorithm to use.                                                                                                                                                                                                                                                                                                        |
| `collector.compression.gzipCompressionLevel` (since _3.6.0_)                      | Optional. Default: `6`. The compression level for GZIP compression. It is between 1 and 9. Lower levels have faster compression speed, but worse compression ratio.                                                                                                                                                                                             |
| `collector.compression.zstdCompressionLevel` (since _3.6.0_)                      | Optional. Default: `9`. The compression level for ZSTD compression. It is between 1 and 22. Lower levels have faster compression speed, but worse compression ratio.                                                                                                                                                                                            |

### Kinesis collector options

| parameter                                                                 | description                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                  | Required. Name of the output kinesis stream for successfully collected events.                                                                                                                                                                                                                                                        |
| `collector.streams.bad`                                                   | Required. Name of the output kinesis stream for http requests which could not be written to the good stream. For example, if the event size exceeds the kinesis limit of 1MB.                                                                                                                                                         |
| `collector.streams.useIpAddressAsPartitionKey` (deprecated since _3.5.0_) | Optional. Default: `false`. Whether to use the user's IP address as the kinesis partition key.                                                                                                                                                                                                                                        |
| `collector.streams.{good,bad}.region`                                     | Optional. Default: `eu-central-1`. AWS region of the kinesis streams.                                                                                                                                                                                                                                                                 |
| `collector.streams.{good,bad}.customEndpoint`                             | Optional. Override aws kinesis endpoint. Can be helpful when using localstack for testing.                                                                                                                                                                                                                                            |
| `collector.streams.{good,bad}.threadPoolSize`                             | Optional. Default: `10`. Thread pool size used by the collector sink for asynchronous operations.                                                                                                                                                                                                                                     |
| `collector.streams.{good,bad}.sqsMaxBytes`                                | Optional. Set to the name of a SQS topic to enable buffering of good output events. When messages cannot be sent to Kinesis, (e.g. because of exceeding api limits) then they get sent to SQS as a fallback. Helpful for smoothing over traffic spikes.                                                                               |
| `collector.streams.good.sqsGoodBuffer`                                    | Optional. Set to the name of a SQS topic to enable buffering of good output events. When messages cannot be sent to Kinesis, (e.g. because of exceeding api limits) then they get sent to SQS as a fallback. Helpful for smoothing over traffic spikes.                                                                               |
| `collector.streams.bad.sqsBadBuffer`                                      | Optional. Like the `sqsGoodBuffer` but for failed events.                                                                                                                                                                                                                                                                             |
| `collector.streams.{good,bad}.aws.accessKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_ACCESS_KEY_ID` environment variable.                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.secretKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_SECRET_ACCESS_KEY` environment variable.                                                                                                                                                                 |
| `collector.streams.{good,bad}.maxBytes` (since _2.9.0_)                   | Optional. Default: `1000000` (1 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. If SQS buffer is activated, `sqsMaxBytes` is used instead.                                                                                                    |
| `collector.streams.{good,bad}.sqsMaxBytes`                                | Optional. Default: `192000` (192 kb). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. SQS has a record size limit of 256 kb, but records are encoded with Base64, which adds approximately 33% of the size, so we set the limit to `256 kb * 3/4`. |
| `collector.streams.{good,bad}.startupCheckInterval` (since _2.9.0_)       | Optional. Default: `1 second`. When collector starts, it checks if Kinesis streams exist with `describeStreamSummary` and if SQS buffers exist with `getQueueUrl` (if configured). This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted.        |
| `collector.streams.backoffPolicy.minBackoff`                              | Optional. Default: `3000`. Time (in milliseconds) for retrying sending to kinesis / SQS after failure.                                                                                                                                                                                                                                |
| `collector.streams.backoffPolicy.maxBackoff`                              | Optional. Default: `600000`. Time (in milliseconds) for retrying sending to kinesis / SQS after failure.                                                                                                                                                                                                                              |
| `collector.streams.buffer.byteLimit`                                      | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to Kinesis. This configures the maximum total size of pending events.                                                                                                                                                                |
| `collector.streams.buffer.recordLimit`                                    | Optional. Default: `500`. Configures the maximum number of pending events before flushing to Kinesis.                                                                                                                                                                                                                                 |
| `collector.streams.buffer.timeLimit`                                      | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to Kinesis.                                                                                                                                                                                                            |

### SQS collector options

| parameter                                                                 | description                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                  | Required. Name of the output SQS queue for successfully collected events.                                                                                                                                                                                                                                                             |
| `collector.streams.bad`                                                   | Required. Name of the output SQS queue for http requests which could not be written to the good stream. For example, if the event size exceeds the SQS limit of 256KB.                                                                                                                                                                |
| `collector.streams.useIpAddressAsPartitionKey` (deprecated since _3.5.0_) | Optional. Default: `false`. Whether to use the user's IP address as the Kinesis partition key. This is attached to the SQS message as an attribute, with the aim of using it if the events ultimately end up in Kinesis.                                                                                                              |
| `collector.streams.{good,bad}.region`                                     | Optional. Default: `eu-central-1`. AWS region of the SQS queues.                                                                                                                                                                                                                                                                      |
| `collector.streams.{good,bad}.threadPoolSize`                             | Optional. Default: `10`. Thread pool size used by the collector sink for asynchronous operations.                                                                                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.accessKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_ACCESS_KEY_ID` environment variable.                                                                                                                                                                     |
| `collector.streams.{good,bad}.aws.secretKey`                              | Required. Set to `default` to use the default provider chain; set to `iam` to use AWS IAM roles; or set to `env` to use `AWS_SECRET_ACCESS_KEY` environment variable.                                                                                                                                                                 |
| `collector.streams.{good,bad}.maxBytes` (since _2.9.0_)                   | Optional. Default: `192000` (192 kb). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead. SQS has a record size limit of 256 kb, but records are encoded with Base64, which adds approximately 33% of the size, so we set the limit to `256 kb * 3/4`. |
| `collector.streams.{good,bad}.startupCheckInterval` (since _2.9.0_)       | Optional. Default: `1 second`. When collector starts, it checks if SQS buffers exist with `getQueueUrl`. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted.                                                                                  |
| `collector.streams.backoffPolicy.minBackoff`                              | Optional. Default: `3000`. Time (in milliseconds) for retrying sending to SQS after failure.                                                                                                                                                                                                                                          |
| `collector.streams.backoffPolicy.maxBackoff`                              | Optional. Default: `600000`. Time (in milliseconds) for retrying sending to SQS after failure.                                                                                                                                                                                                                                        |
| `collector.streams.buffer.byteLimit`                                      | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to SQS. This configures the maximum total size of pending events.                                                                                                                                                                    |
| `collector.streams.buffer.recordLimit`                                    | Optional. Default: `500`. Configures the maximum number of pending events before flushing to SQS.                                                                                                                                                                                                                                     |
| `collector.streams.buffer.timeLimit`                                      | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to SQS.                                                                                                                                                                                                                |

### Pubsub collector options

| parameter                                                                                        | description                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                                                                         | Required. Name of the output Pubsub topic for successfully collected events.                                                                                                                                                                          |
| `collector.streams.bad`                                                                          | Required. Name of the output pubsub topic for http requests which could not be written to the good stream. For example, if the event size exceeds the Pubsub limit of 10MB.                                                                           |
| `collector.streams.sink.{good,bad}.googleProjectId`                                              | Required. GCP project name.                                                                                                                                                                                                                           |
| `collector.streams.sink.{good,bad}backoffPolicy.minBackoff` (deprecated since _3.5.0_)           | Optional. Default: `1000`. Time (in milliseconds) for retrying sending to Pubsub after failure.                                                                                                                                                       |
| `collector.streams.sink.{good,bad}.backoffPolicy.maxBackoff` (deprecated since _3.5.0_)          | Optional. Default: `1000`. Time (in milliseconds) for retrying sending to Pubsub after failure                                                                                                                                                        |
| `collector.streams.sink.{good,bad}.backoffPolicy.totalBackoff` (deprecated since _3.5.0_)        | Optional. Default: `9223372036854`. We set this to the maximum value so that we never give up on trying to send a message to pubsub.                                                                                                                  |
| `collector.streams.sink.{good,bad}.backoffPolicy.multipler` (deprecated since _3.5.0_)           | Optional. Default: `2`. Multiplier between two periods.                                                                                                                                                                                               |
| `collector.streams.sink.{good,bad}.backoffPolicy.initialRpcTimeout` (deprecated since _3.5.0_)   | Optional. Default: `10000`. Time (in milliseconds) before a RPC call to Pubsub is aborted and retried.                                                                                                                                                |
| `collector.streams.sink.{good,bad}.backoffPolicy.maxRpcTimeout` (deprecated since _3.5.0_)       | Optional. Default: `10000`. Maximum time (in milliseconds) before RPC call to Pubsub is aborted and retried.                                                                                                                                          |
| `collector.streams.sink.{good,bad}.backoffPolicy.rpcTimeoutMultipler` (deprecated since _3.5.0_) | Optional. Default: `2`. How RPC timeouts increase as they are retried.                                                                                                                                                                                |
| `collector.streams.sink.{good,bad}..maxBytes` (since _2.9.0_)                                    | Optional. Default: `10000000` (10 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead.                                                                             |
| `collector.streams.sink.{good,bad}.startupCheckInterval` (since _2.9.0_)                         | Optional. Default: `1 second`. When collector starts, it checks if PubSub topics exist with `listTopics`. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted. |
| `collector.streams.sink.{good,bad}.retryInterval` (since _2.9.0_)                                | Optional. Default: `10 seconds`. Collector uses built-in retry mechanism of PubSub API. In case of failure of these retries, the events are added to a buffer and every `retryInterval` collector retries to send them.                               |
| `collector.streams.{good,bad}.buffer.byteLimit`                                                  | Optional. Default: `1000000`. Incoming events are stored in an internal buffer before being sent to Pubsub. This configures the maximum total size of pending events                                                                                  |
| `collector.streams.{good,bad}.buffer.recordLimit`                                                | Optional. Default: `40`. Maximum number of pending events before flushing to Pubsub.                                                                                                                                                                  |
| `collector.streams.{good,bad}.buffer.timeLimit`                                                  | Optional. Default: `1000`. Maximum time (in milliseconds) before flushing pending buffered events to Pubsub.                                                                                                                                          |

### Kafka collector options

| parameter                                           | description                                                                                                                                                                                                                         |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.streams.good`                            | Required. Name of the output Kafka topic for successfully collected events.                                                                                                                                                         |
| `collector.streams.bad`                             | Required. Name of the output Kafka topic for http requests which could not be written to the good stream.                                                                                                                           |
| `collector.streams.{good,bad}.brokers`              | Required. A list of host:port pairs to use for establishing the initial connection to the Kafka cluster.                                                                                                                            |
| `collector.streams.{good,bad}.producerConf`         | Optional. Kafka producer configuration. See [the docs](https://kafka.apache.org/documentation/#producerconfigs) for all properties.                                                                                                 |
| `collector.streams.{good,bad}.maxBytes`             | Optional. Default: `1000000` (1 MB). Maximum number of bytes that a single record can contain. If a record is bigger, a size violation failed event is emitted instead.                                                             |
| `collector.streams.{good,bad}.startupCheckInterval` | Optional. Default: `10 second`. When collector starts, it checks if Kafka topics exist. This is the interval for the calls. `/sink-health` is made healthy as soon as requests are successful or records are successfully inserted. |
| `collector.streams.{good,bad}.buffer.byteLimit`     | Optional. Default: `3145728`. Incoming events are stored in an internal buffer before being sent to Kafka. This configures the maximum total size of pending events.                                                                |
| `collector.streams.{good,bad}.buffer.recordLimit`   | Optional. Default: `500`. Configures the maximum number of pending events before flushing to Kafka.                                                                                                                                 |
| `collector.streams.{good,bad}.buffer.timeLimit`     | Optional. Default: `5000`. Configures the maximum time in milliseconds before flushing pending buffered events to Kafka.                                                                                                            |
| `collector.streams.{good,bad}.retryInterval`        | Optional. Default: `10 seconds`. Collector uses built-in retry mechanism of Kafka API. In case of failure of these retries, the events are added to a buffer and every `retryInterval` collector retries to send them.              |

### Setting the domain name

Set the cookie name using the `collector.cookie.name` setting. To maintain backward compatibility with earlier versions of the collector, use the string "sp" as the cookie name.

The collector responds to valid requests with a `Set-Cookie` header, which may or may not specify a `domain` for the cookie.

If no domain is specified, the cookie will be set against the full collector domain, for example `collector.snplow.com`. That will mean that applications running elsewhere on `*.snplow.com` won't be able to access it. If you don't need to grant access to the cookie from other applications on the domain, then you can ignore the `domains` and `fallbackDomain` settings.

In earlier versions, you could specify a `domain` to tie the cookie to. For example, if set to `.snplow.com`, the cookie would have been accessible to other applications running on `*.snplow.com`. To do the same in this version, use the `fallbackDomain` setting but **make sure** that you no longer include a leading dot:

```properties
fallbackDomain = "snplow.com"
```

The cookie set by the collector can be treated differently by browsers, depending on whether it's considered to be a first-party or a third-party cookie. In earlier versions (0.15.0 and earlier), if you had two collector endpoints, one on `collector.snplow.com` and one on `collector.snplow.net`, you could only specify one of those domains in the configuration. That meant that you were only able to set a first-party cookie server-side on either `.snplow.com` or `.snplow.net`, but not on both. From version 0.16.0, you can specify a list of domains to be used for the cookie (**note the lack of a leading dot**):

```properties
domains = [
  "snplow.com"
  "snplow.net"
]
```

Which domain to be used in the `Set-Cookie` header is determined by matching the domains from the `Origin` header of the request to the specified list. The first match is used. If no matches are found, the fallback domain will be used, if configured. If no `fallbackDomain` is configured, the cookie will be tied to the full collector domain.

If you specify a main domain in the list, all subdomains on it will be matched. If you specify a subdomain, only that subdomain will be matched.

Examples:

- `domain.com` will match `Origin` headers like `domain.com`, `www.domain.com` and `secure.client.domain.com`
- `client.domain.com` will match an `Origin` header like `secure.client.domain.com` but not `domain.com` or `www.domain.com`.

### Configuring custom paths

The collector responds with a cookie to requests with a path that matches the `vendor/version` protocol. The expected values are:

- `com.snowplowanalytics.snowplow/tp2` for Tracker Protocol 2
- `r/tp2` for redirects
- `com.snowplowanalytics.iglu/v1` for the Iglu Webhook.

You can also map any valid (ie, two-segment) path to one of the three defaults via the `collector.paths` section of the configuration file. Your custom path must be the key and the value must be one of the corresponding default paths. Both must be full valid paths starting with a leading slash:

```properties
paths {
  "/com.acme/track"    = "/com.snowplowanalytics.snowplow/tp2"
  "/com.acme/redirect" = "/r/tp2"
  "/com.acme/iglu"     = "/com.snowplowanalytics.iglu/v1"
}
```

### TLS port binding and certificate (2.4.0+)

Since 2.4.0 TLS certificates are configured using JVM system parameters. The "`Customizing JSSE`" section in [Java 11 JSSE reference documentation](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-A41282C3-19A3-400A-A40F-86F4DA22ABA9) explains all system properties in detail.

The following JVM properties are the ones to be used most of the time.

| System Property                  | Customized Item                                                                                                                                                                                                                                                           | Default                                                                                                                                                                                                                        | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `javax.net.ssl.keyStore`         | Default keystore; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150)          | None                                                                                                                                                                                                                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `javax.net.ssl.keyStorePassword` | Default keystore password; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150) | None                                                                                                                                                                                                                           | It is inadvisable to specify the password in a way that exposes it to discovery by other users. **Password can not be empty.**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `javax.net.ssl.keyStoreType`     | Default keystore type; see [Customizing the Default Keystores and Truststores, Store Types, and Store Passwords](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-7D9F43B8-AABF-4C5B-93E6-3AFB18B66150)     | `PKCS12`                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `jdk.tls.server.cipherSuites`    | Server-side default enabled cipher suites. See [Specifying Default Enabled Cipher Suites](https://docs.oracle.com/en/java/javase/11/security/java-secure-socket-extension-jsse-reference-guide.html#GUID-D61663E8-2405-4B2D-A1F1-B8C7EA2688DB)                            | See [SunJSSE Cipher Suites](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2__SUNJSSE_CIPHER_SUITES) to determine which cipher suites are enabled by default | Caution: These system properties can be used to configure weak cipher suites, or the configured cipher suites may be weak in the future. It is not recommended that you use these system properties without understanding the risks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `jdk.tls.server.protocols`       | Default handshaking protocols for TLS/DTLS servers. See [The SunJSSE Provider](https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2)                                                                        | None                                                                                                                                                                                                                           | To configure the default enabled protocol suite in the server-side of a SunJSSE provider, specify the protocols in a comma-separated list within quotation marks. The protocols in this list are standard SSL protocol names as described in [Java Security Standard Algorithm Names](https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html). Note that this System Property impacts only the default protocol suite (SSLContext of the algorithms SSL, TLS and DTLS). If an application uses a version-specific SSLContext (SSLv3, TLSv1, TLSv1.1, TLSv1.2, TLSv1.3, DTLSv1.0, or DTLSv1.2), or sets the enabled protocol version explicitly, this System Property has no impact. |

A deployment would need a TLS certificate, preferably issued by a trusted CA, however, for test purposes, a TLS cert could be generated as follows

```bash
ssl_dir=/opt/snowplow/ssl
mkdir -p ${ssl_dir}

sudo openssl req \
  -x509 \
  -newkey rsa:4096 \
  -keyout ${ssl_dir}/collector_key.pem \
  -out ${ssl_dir}/collector_cert.pem \
  -days 3650 \
  -nodes \
  -subj "/C=UK/O=Acme/OU=DevOps/CN=*.acme.com"

sudo openssl pkcs12 \
  -export \
  -out ${ssl_dir}/collector.p12 \
  -inkey ${ssl_dir}/collector_key.pem \
  -in ${ssl_dir}/collector_cert.pem \
  -passout pass:changeme

sudo chmod 644 ${ssl_dir}/collector.p12
```

and then the collector (kinesis as example) could be started as follows

```bash
config_dir=/opt/snowplow/config

docker run \
  -d \
  --name scala-stream-collector \
  --restart always \
  --network host \
  -v ${config_dir}:/snowplow/config \
  -v ${ssl_dir}:/snowplow/ssl \
  -p 8080:8080 \
  -p 8443:8443 \
  -e 'JAVA_OPTS=-Xms2g -Xmx2g -Djavax.net.ssl.keyStoreType=pkcs12 -Djavax.net.ssl.keyStorePassword=changeme -Djavax.net.ssl.keyStore=/snowplow/ssl/collector.p12 -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Dcom.amazonaws.sdk.disableCbor' \
  snowplow/scala-stream-collector-kinesis:2.5.0 \
  --config /snowplow/config/snowplow-stream-collector-kinesis-2.5.0.hocon
```

Note: If you don't have a verified certificate, you need to disable SSL verification on client side. e.g. `cURL`'s `-k, --insecure` flag disables it.

### Setting up an SQS buffer (2.0.0+)

On AWS, the lack of auto-scaling in Kinesis results in throttled streams in case of traffic spikes and the collector starts accumulating events to retry them later. If accumulation continues long enough, the collector will run out of memory. To prevent the possibility of a broken collector, we decided to make it possible to configure an SQS buffer which can provide additional assurance during extreme traffic spikes.

SQS is used to queue any message that the collector failed to send to Kinesis. The [Snowbridge application](/docs/api-reference/snowbridge/) can then read the messages from SQS and write them to Kinesis once Kinesis is ready. In the event of any AWS API glitches, there is a retry mechanism which retries sending to the SQS queue 10 times.

The keys set up for the Kinesis stream are stored as SQS message attributes in order to preserve the information.

> **Warning:** The SQS messages cannot be as big as Kinesis messages. The limit is 256kB per message, but we send the messages as Base64 encoded, so the limit goes down to 192kB for the original message.

#### Setting up the SQS queues

> **Note:** This section only applies to the case when SQS is used as a fallback sink when Kinesis is unavailable. If you are using SQS as the primary sink, then the settings below should be ignored and the `good` and `bad` streams should be configured as normal under `streams.good` and `streams.bad` respectively.

To start using this feature, you will first need to set up the SQS queues. Two separate queues are required for good (raw) events and bad events. The Collector then needs to be informed about the queue names, and this can be done by adding these as entries to `config.hocon`:

```properties
sqsGoodBuffer = {good-sqs-queue-url}
sqsBadBuffer = {bad-sqs-queue-url}
```

### Networking

Since version 3.0.0 networking settings are configured in its own `collector.networking` section:

| parameter                                                    | description                                                                                                                                                                                                      |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `collector.networking.maxConnections` (since _3.0.0_)        | Optional. Default: `1024`. Maximum number of concurrent active connection.                                                                                                                                       |
| `collector.networking.idleTimeout` (since _3.0.0_)           | Optional. Default: `610 seconds`. Maximum inactivity time for a network connection. If no data is sent within that time, the connection is closed.                                                               |
| `collector.networking.bodyReadTimeout` (since _3.7.0_)       | Optional. Default: `25 seconds`. Maximum time from receiving request headers to receiving end of request body. If exceeded returns a 408 Request Timeout.                                                        |
| `collector.networking.responseHeaderTimeout` (since _3.2.0_) | Optional. Default: `30 seconds`. Time from when the request is made until a response line is generated before a 503 response is returned. It is recommended to make this slightly larger than `bodyReadTimeout`. |
| `collector.networking.maxRequestLineLength` (since _3.2.0_)  | Optional. Default: `20480`. Maximum request line to parse. If exceeded returns a 400 Bad Request.                                                                                                                |
| `collector.networking.maxHeadersLength` (since _3.2.0_)      | Optional. Default: `40960`. Maximum data that composes the headers. If exceeded returns a 400 Bad Request.                                                                                                       |
| `collector.networking.maxPayloadSize` (since _3.3.0_)        | Optional. Default: `1048576` (1 MB). Maximum size of an event within payload allowed before emitting an Size Violation event. Returns 200 OK.                                                                    |
| `collector.networking.dropPayloadSize` (since _3.3.0_)       | Optional. Default: `2097152` (2 MB). Maximum body payload size allowed before rejecting the request. If exceeded returns a 413 Payload Too Large.                                                                |

---

# Introduction to Snowplow Collector
> The Snowplow event Collector receives raw Snowplow events from trackers and webhooks, serializes them, and writes them to supported sinks including Kinesis, PubSub, Kafka, NSQ, SQS, and stdout.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/

The collector receives raw Snowplow events sent over HTTP by [trackers](/docs/sources/) or [webhooks](/docs/sources/webhooks/). It serializes them, and then writes them to a sink. Currently supported sinks are:

1. [Amazon Kinesis](http://aws.amazon.com/kinesis/)
2. [Google PubSub](https://cloud.google.com/pubsub/)
3. [Apache Kafka](http://kafka.apache.org/)
4. [NSQ](http://nsq.io/)
5. [Amazon SQS](https://aws.amazon.com/sqs/)
6. `stdout` for a custom stream collection process

The collector supports cross-domain Snowplow deployments, setting a `user_id` (used to identify unique visitors) server side to reliably identify the same user across domains.

## How it works

### User identification

The collector allows the use of a third-party cookie, making user tracking across domains possible.

In a nutshell: the collector receives events from a tracker, sets/updates a third-party user tracking cookie, and returns the pixel to the client. The ID in this third-party user tracking cookie is stored in the `network_userid` field in Snowplow events.

In pseudocode terms:

```text
if (request contains an "sp" cookie) {
    Record that cookie as the user identifier
    Set that cookie with a now+1 year cookie expiry
    Add the headers and payload to the output array
} else {
    Set the "sp" cookie with a now+1 year cookie expiry
    Add the headers and payload to the output array
}
```

## Technical architecture

The collector is written in scala and built on top of [http4s](https://http4s.org).

[GitHub repository](https://github.com/snowplow/stream-collector)

---

# Set up and run Collector
> Instructions for running the Snowplow event Collector using Docker images or jar files, including how to configure the application with HOCON files and perform health checks.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/setup/

A Terraform module is available which deploys the collector on a AWS EC2 without the need for this manual setup.

## Run the collector

The collector is on docker hub with several different flavours. Pull the image that matches the sink you are using:

```bash
docker pull snowplow/scala-stream-collector-kinesis:3.7.0
docker pull snowplow/scala-stream-collector-pubsub:3.7.0
docker pull snowplow/scala-stream-collector-kafka:3.7.0
docker pull snowplow/scala-stream-collector-rabbitmq-experimental:3.7.0
docker pull snowplow/scala-stream-collector-nsq:3.7.0
docker pull snowplow/scala-stream-collector-sqs:3.7.0
docker pull snowplow/scala-stream-collector-stdout:3.7.0
```

The application is configured by passing a hocon file on the command line:

```bash
docker run --rm \
-v $PWD/config.hocon:/snowplow/config.hocon \
-p 8080:8080 \
snowplow/scala-stream-collector-${flavour}:3.7.0 --config /snowplow/config.hocon
```

Alternatively, you can download and run [a jar file from the github release](https://github.com/snowplow/stream-collector/releases).

```bash
java -jar scala-stream-collector-kinesis-3.7.0.jar --config /path/to/config.hocon
```

**Telemetry notice**

By default, Snowplow collects telemetry data for Collector (since version 2.4.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `collector.telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `collector.telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

## Health check

Pinging the collector on the /health path should return a 200 OK response:

```bash
curl http://localhost:8080/health
```

---

# Collector 3.0.x upgrade guide
> Upgrade guide for Collector 3.0.x covering breaking changes including the new license, migration from Akka HTTP to http4s, single endpoint port, and updated configuration requirements.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/

## Breaking changes

### New license

Since version 3.0.0, the collector has been migrated to use the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)).

### New HTTP stack

Version 3.0.0 replaces the Akka HTTP stack with http4s. The same stack is already used in all other Snowplow microservices, so this makes the codebase more uniform and enables us to share more code between applications.

Also, newer versions of Akka HTTP would have required a [commercial license from Lightbend](https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka) for many of Snowplow’s users.

### Single endpoint port

Previously, the collector was able to expose both http and https ports. We’ve found it impractical and therefore the new version is exposing a single port — either http or https. To enable http→https upgrades, use a load-balancer/proxy.

## Upgrading

In order to ease the migration for existing installations, we’ve tried to keep the configurations mostly backwards-compatible.

However, due to changes above, some amendments are required. We’ve also introduced a few tweaks that should simplify configurations between pipeline services. Full configuration examples are available in [snowplow/stream-collector](https://github.com/snowplow/stream-collector/tree/master/examples) repository along with extended commentary.

If you’ve previously used both HTTP and HTTPS ports in the collector, this feature is no longer available. Since version 3.0.0 you can either:

- enable TLS termination in the collector (use HTTPS between load balancer and collector) and enable HTTP upgrade in load balancer running in front of the collector
- disable TLS termination in the collector (use HTTP between load balancer and collector), use TLS termination and HTTP upgrade in the load balancer

### License acceptance

You have to explicitly accept the [Snowplow Limited Use License](/limited-use-license-1.0/) ([FAQ](/docs/licensing/limited-use-license-faq/)). To do so, either set the `ACCEPT_LIMITED_USE_LICENSE=yes` environment variable, or update the following section in the configuration:

```hcl
collector {
  license {
    accept = true
  }
  ...
}
```

### Akka section removal

The top-level `akka` section will no longer be used. Keeping the section in your configuration will _not_ cause collector failures but should be removed for clarity.

### Split sinks

Sink configurations can now be defined similarly as with our other services. Each sink is configured separately to allow for individual config specifics. This change is especially useful for our Kafka collector running on Azure, where EventHubs needs separate `producerConf` sections to account for different EH settings.

Example:

```hcl
collector {
  ...
  streams {
    good {
      name = "good"
      brokers = "localhost:9092,another.host:9092"
      producerConf {
        acks = all
        "key.serializer"     = "org.apache.kafka.common.serialization.StringSerializer"
        "value.serializer"   = "org.apache.kafka.common.serialization.StringSerializer"
      }
      buffer {
        byteLimit = 3145728
        recordLimit = 500
        timeLimit = 5000
      }
    }

    bad {
      name = "bad"
      brokers = "localhost:9092,another.host:9092
      producerConf {
        acks = all
        "key.serializer"     = "org.apache.kafka.common.serialization.StringSerializer"
        "value.serializer"   = "org.apache.kafka.common.serialization.StringSerializer"
      }
      maxBytes = 1000000
      buffer {
        byteLimit = 3145728
        recordLimit = 500
        timeLimit = 5000
      }
    }
  }
  ...
}
```

### Networking

Optional networking settings were previously a part of Akka HTTP configuration section. With the removal of the framework, the relevant settings have been moved into a dedicated `collector.networking` section:

- `collector.networking.maxConnections` - maximum number of concurrent active connection.
- `collector.networking.idleTimeout` - maximum inactivity time for a network connection. If no data is sent within that time, the connection is closed.

Example:

```hcl
collector {
  ...
  networking {
    maxConnections = 1024
    idleTimeout = 610 seconds
  }
  ...
}
```

---

# Collector 3.6.x upgrade guide with compression
> Upgrade guide for Collector 3.6.x introducing payload compression to reduce storage costs and improve throughput, requiring Enrich 6.1.x for compatibility.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/

Collector 3.6.0 introduces payload compression, a new feature that significantly reduces the size (and therefore, cost) of data written to your raw output stream.

The compression feature allows the collector to batch multiple individual collector payloads into a single compressed stream record. This provides several benefits:

- **Reduced storage costs**: compressed payloads take up less space in your output streams
- **Improved throughput**: fewer, larger records reduce the overhead of stream processing
- **Better performance**: downstream consumers can process batches more efficiently

## Enabling compression

> **Warning:** Before enabling compression, you must upgrade to Enrich 6.1.x first. The reason for this is that support for processing compressed payloads is added to Enrich starting with Enrich 6.1.0. Enrich 6.1.0 can process both compressed and uncompressed payloads.
>
> Enrich is currently the only application compatible with compression. Setups with an [S3 loader](/docs/api-reference/loaders-storage-targets/s3-loader/) reading off the raw stream will not be supported.

When upgrading to Collector 3.6.0, compression is an optional feature that can be configured in your [collector settings](/docs/api-reference/stream-collector/configure/). If this feature is not enabled, there will be no changes to the data format or size.

After upgrading Enrich, compression in Collector can be enabled by adding the following config section:

```hocon
compression {
  enabled = true
}
```

> **Tip:** Take note of the `streams.buffer.timeLimit` Collector configuration parameter, which already existed in previous versions. This controls how many events are batched (or, technically, for how long) before appling compression. Bigger values lead to higher compression rates (lower infrastructure costs), but also higher latency. We recommend starting with a value around 300–500ms and fine-tuning it from there.

### Impact on metrics

When compression is enabled, there will be a big decrease in the number of messages sent to the `raw` event stream, i.e. Kinesis, Pub/Sub or Event Hubs, depending on your cloud. You will notice this decrease if you monitor metrics on messages in the `raw` stream.

This is perfectly normal and does not indicate any drop in event volumes. It happens because the compression feature batches together many Snowplow events into a single message sent to the `raw` stream.

---

# Collector upgrade guides
> Guides to help you upgrade the Snowplow event Collector to newer versions with information on breaking changes and migration steps.
> Source: https://docs.snowplow.io/docs/api-reference/stream-collector/upgrade-guides/

This section contains information to help you upgrade to newer versions of Collector.

## [📄️ 3.6.x upgrade guide](/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/)

[Upgrade guide for Collector 3.6.x introducing payload compression to reduce storage costs and improve throughput, requiring Enrich 6.1.x for compatibility.](/docs/api-reference/stream-collector/upgrade-guides/3-6-x-upgrade-guide/)

## [📄️ 3.0.x upgrade guide](/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/)

[Upgrade guide for Collector 3.0.x covering breaking changes including the new license, migration from Akka HTTP to http4s, single endpoint port, and updated configuration requirements.](/docs/api-reference/stream-collector/upgrade-guides/3-0-x-upgrade-guide/)

---

# Links to tracker API documentation
> Generated API documentation for Snowplow tracker SDKs across all supported platforms and programming languages.
> Source: https://docs.snowplow.io/docs/api-reference/trackers/

This section contains links to the generated API documentation for our tracker SDKs.

---

# Snowplow component versions and compatibility matrix
> Latest versions of Snowplow components including collectors, enrichment, loaders, trackers, Iglu, data models, and analytics SDKs with compatibility and upgrade information.
> Source: https://docs.snowplow.io/docs/api-reference/versions/

This page lists the most recent versions of Snowplow components.

Some information about components is relevant only for [Snowplow Self-Hosted](/docs/get-started/#self-hosted) users, as [Snowplow CDI](/docs/get-started/#customer-data-infrastructure) customers won't need to configure all their own components.

In short, almost everything is compatible with almost everything. We rarely change the core protocols that various components use to communicate.

You might encounter specific restrictions when following the documentation, for example, some of our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) might call for a reasonably recent version of the [warehouse loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/).

## Upgrades and deprecation

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to deal with upgrading your pipeline. We'll perform upgrades for you.

Some major upgrades might have breaking changes. In this case, we provide upgrade guides, such as the ones for [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/).

From time to time, we develop better applications for certain tasks and deprecate the old ones. Deprecations are announced on [Community](https://community.snowplow.io/).

***

## Latest versions

### Core pipeline

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to install any of the core pipeline components yourself. We'll deploy your pipeline and keep it up to date.

**AWS:**

| Component                                                                                                        | Latest version |
| ---------------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                                        | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                             | 6.9.0          |
| [RDB Loader (Redshift, Snowflake, Databricks)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) | 6.3.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                          | 0.9.1          |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)            | 0.5.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/)          | 0.4.0          |
| [S3 Loader](/docs/api-reference/loaders-storage-targets/s3-loader/)                                              | 3.1.0          |
| [Snowbridge](/docs/api-reference/snowbridge/)                                                                    | 4.1.0          |
| [Elasticsearch Loader](/docs/api-reference/loaders-storage-targets/elasticsearch/)                               | 2.1.3          |
| [Postgres Loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)                         | 0.3.3          |
| [Dataflow Runner](/docs/api-reference/dataflow-runner/)                                                          | 0.7.6          |

**GCP:**

| Component                                                                                               | Latest version |
| ------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                               | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                    | 6.9.0          |
| [RDB Loader (Snowflake, Databricks)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)  | 6.3.0          |
| [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/)                         | 2.1.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.9.1          |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)   | 0.5.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) | 0.4.0          |
| [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/)                  | 0.5.6          |
| [Snowbridge](/docs/api-reference/snowbridge/)                                                           | 4.1.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.9.1          |
| [Postgres Loader](/docs/api-reference/loaders-storage-targets/snowplow-postgres-loader/)                | 0.3.3          |

**Azure:**

| Component                                                                                               | Latest version |
| ------------------------------------------------------------------------------------------------------- | -------------- |
| [Stream Collector](/docs/api-reference/stream-collector/)                                               | 3.7.0          |
| [Enrich](/docs/api-reference/enrichment-components/)                                                    | 6.9.0          |
| [RDB Loader (Snowflake)](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/)              | 6.3.0          |
| [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/)                                 | 0.9.1          |
| [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)   | 0.5.1          |
| [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) | 0.4.0          |

***

### Iglu (schema registry)

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, you don't need to install Iglu Server yourself. It's also unlikely that you need to use any of the other components in this section. You can manage your data structures [in the UI or via the API](/docs/event-studio/data-structures/).

| Component                                                                      | Latest version |
| ------------------------------------------------------------------------------ | -------------- |
| [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/)         | 0.14.1         |
| [`igluctl` utility](/docs/api-reference/iglu/igluctl-2/)                       | 0.13.0         |
| [Iglu Scala client](/docs/api-reference/iglu/iglu-clients/scala-client-setup/) | 4.0.3          |
| [Iglu Objective-C client](/docs/api-reference/iglu/iglu-clients/objc-client/)  | 0.1.1          |

### Trackers

| Tracker                                                     | Latest version |
| ----------------------------------------------------------- | -------------- |
| [JavaScript (Web and Node.js)](/docs/sources/web-trackers/) | 4.6.8          |
| [iOS](/docs/sources/mobile-trackers/)                       | 6.2.1          |
| [Android](/docs/sources/mobile-trackers/)                   | 6.2.0          |
| [React Native](/docs/sources/react-native-tracker/)         | 4.6.8          |
| [Flutter](/docs/sources/flutter-tracker/)                   | 0.8.0          |
| [WebView](/docs/sources/webview-tracker/)                   | 0.3.0          |
| [Roku](/docs/sources/roku-tracker/)                         | 0.3.1          |
| [Google AMP](/docs/sources/google-amp-tracker/)             | 1.1.0          |
| [Pixel](/docs/sources/pixel-tracker/)                       | 0.3.0          |
| [Golang](/docs/sources/golang-tracker/)                     | 3.1.0          |
| [.NET](/docs/sources/net-tracker/)                          | 1.3.0          |
| [Java](/docs/sources/java-tracker/)                         | 2.1.0          |
| [Python](/docs/sources/python-tracker/)                     | 1.0.3          |
| [Scala](/docs/sources/scala-tracker/)                       | 2.0.0          |
| [Ruby](/docs/sources/ruby-tracker/)                         | 0.8.0          |
| [Rust](/docs/sources/rust-tracker/)                         | 0.2.0          |
| [PHP](/docs/sources/php-tracker/)                           | 0.9.2          |
| [C++](/docs/sources/c-tracker/)                             | 2.0.0          |
| [Unity](/docs/sources/unity-tracker/)                       | 0.8.1          |
| [Lua](/docs/sources/lua-tracker/)                           | 0.2.0          |

### Data models

#### dbt

[Modeling data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/) is our recommended approach.

**Snowplow Unified Digital:**

| snowplow-unified version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ------------------------ | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                    | >=1.10.6 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.4.5                    | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |

**Snowplow Media Player:**

| snowplow-media-player version | snowplow-web version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ----------------------------- | -------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.9.4                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.8.0                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.5.3                         | >=0.14.0 to <0.16.0  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.2                         | >=0.13.0 to <0.14.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.1                         | >=0.12.0 to <0.13.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.3.4                         | >=0.9.0 to <0.12.0   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.1.0                         | >=0.6.0 to <0.7.0    | >=0.20.0 to <1.1.0 | ❌        | ❌          | ✅        | ❌         | ✅        | ❌     |

**Snowplow Normalize:**

| snowplow-normalize version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.4.1                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ✅     |
| 0.3.5                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.3                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.1.0                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

**Snowplow Ecommerce:**

| snowplow-ecommerce version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.9.3                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ✅     |
| 0.8.2                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ❌     |
| 0.3.0                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.1                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

Postgres is technically supported in the models within the package, however one of the contexts’ names is too long to be loaded via the Postgres Loader.

**Snowplow Attribution:**

| snowplow-attribution version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ---------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.6.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ✅     |
| 0.3.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ❌     |

***

See also the [dbt version compatibility checker](/docs/modeling-your-data/modeling-your-data-with-dbt/#dbt-version-compatibility-checker).

#### SQL Runner

> **Note:** We recommend using the dbt models above, as they are more actively developed.

The latest version of [SQL Runner](/docs/modeling-your-data/modeling-your-data-with-sql-runner/) itself is **0.10.1**.

| Model                                                                                               | Redshift | BigQuery | Snowflake |
| --------------------------------------------------------------------------------------------------- | -------- | -------- | --------- |
| [Web](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/)       | 1.3.1    | 1.0.4    | 1.0.2     |
| [Mobile](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/) | 1.1.0    | 1.1.0    | 1.1.0     |

### Testing and debugging

> **Info:** If you are a Snowplow CDI customer, rather than self-hosted, we recommend using [Snowplow Micro through Console](/docs/testing/snowplow-micro/console/) for testing and debugging.

| Application                                                     | Latest version |
| --------------------------------------------------------------- | -------------- |
| [Snowplow Micro](/docs/testing/snowplow-micro/)                 | 4.1.1          |
| [Snowplow Mini](/docs/api-reference/snowplow-mini/usage-guide/) | 0.25.1         |

### Analytics SDKs

| SDK                                                                       | Latest version |
| ------------------------------------------------------------------------- | -------------- |
| [Scala](/docs/api-reference/analytics-sdk/analytics-sdk-scala/)           | 3.0.0          |
| [Javascript](/docs/api-reference/analytics-sdk/analytics-sdk-javascript/) | 0.3.1          |
| [Python](/docs/api-reference/analytics-sdk/analytics-sdk-python/)         | 0.2.3          |
| [.NET](/docs/api-reference/analytics-sdk/analytics-sdk-net/)              | 0.2.1          |
| [Go](/docs/api-reference/analytics-sdk/analytics-sdk-go/)                 | 0.3.0          |

---

# Create event forwarders in Console
> Step-by-step guide to create a Snowplow event forwarder to send events to third-party destinations in real-time with connections, filters, and field mappings.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/creating-forwarders/

Follow the steps below to configure a new event forwarder. See [available integrations](/docs/destinations/forwarding-events/integrations/) for destination-specific guides.

## Step 1: Create a connection

A **connection** is a resource that stores the credentials and endpoint details needed to send events to your destination.

To create a connection from [Snowplow Console](https://console.snowplowanalytics.com), first go to **Destinations** > **Connections**, then select **Set up connection**. From the dropdown, choose **Loader connection**, then select the destination you want to forward events to. Each destination will have specific authentication and endpoint details required.

![Console interface for creating a new destination connection with authentication and endpoint configuration fields](/assets/images/event-forwarding-connection-094b95e8dd4dee4353cd575fab0468b6.png)

When finished, click **Deploy**. Once a connection is deployed, you can use it in one or more forwarders to connect to your destination.

## Step 2: Create a new forwarder

1. Go to **Destinations** > **Destination list**.
2. Navigate to the **Available** tab and select **Configure** on the destination card from the list of available integrations to start setting up the forwarder.
3. Give the forwarder a **name**, select the **pipeline** you want the forwarder to read events from, and choose the **connection** you created in step 1.
4. Optionally, you can choose to **Import configuration from** an existing forwarder. This is helpful when migrating a forwarder setup from development to production.
5. Click **Continue** to configure event filters and data mapping.

## Step 3: Configure event filters and field mapping

Forwarders use JavaScript expressions to define which events to forward and how to map Snowplow data to your destination's required schema.

### Event filtering

Use JavaScript expressions to select which events to forward. Only events that return `true` when evaluated against your filter will be sent to your destination.

Use the `event` object to reference fields on your Snowplow payloads. For example:

```javascript
// Forward page views from website
event.app_id == "website" && event.event_name == "page_view"

// Forward a list of custom events
["add_to_cart", "purchase"].includes(event.event_name)
```

Leave the filter blank to forward all events.

![Event filtering configuration panel showing JavaScript expression field for defining which events to forward](/assets/images/event-forwarding-filters-eb59897842efc2acb3dd85500b569aae.png)

### Field mapping

Define how Snowplow data maps to your destination fields. For each mapping, **Destination Field** represents the property name and **Snowplow expression** is a JavaScript expression used to extract data from your Snowplow event. Snowplow provides default mappings based on common fields, but you can overwrite or delete them as needed.

![Field mapping configuration panel showing key/value pairs with JavaScript property selection expressions](/assets/images/event-forwarding-mapping-0811c90079f2c0d53d555f11294bad1a.png)

### Custom functions

You can also write JavaScript functions for complex data transformations. Examples include converting date formats, transforming enum values, combining multiple fields, or applying other business logic. You can then reference functions in both the event filter and field mapping sections.

![Custom functions editor with JavaScript code for complex data transformations in event forwarding](/assets/images/event-forwarding-custom-functions-7a4098875609b9bcb07169204c459fbc.png)

> **Info:** To learn more about the supported filter and mapping expressions, check out the [filter and mapping reference](/docs/destinations/forwarding-events/reference/).

## Step 4: Test your transformations

Once you've defined your filter and mapping configuration, you can test it against a sample event and preview what the output JSON payload looks like. Choose an event from the **Select sample input event** dropdown and selecting **Run test**.

![Test transformation interface showing sample event input and JSON output preview with run test button](/assets/images/event-forwarding-test-transformations-54e02b5e0465eca443b8f05774912721.png)

Snowplow provides a few out-of-the-box sample events to test with, which you can edit as needed. You can also choose **Custom event** to paste in your own JSON-formatted Snowplow event. You can use [Snowplow Micro](/docs/testing/snowplow-micro/) with the `--output-json` flag to generate your own events to test with.

Select **View generated code** to see the JavaScript function generated from your filters, field mappings, and custom functions. This is exactly what will run when transforming events for your destination, and can be used directly in a [Snowbridge JavaScript transformation](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) for local testing.

If there is an error with your configuration, the generated code will show the line number which contains the error.

## Step 5: Deploy

When you're done, select **Deploy** to save your configuration and create the forwarder. This will deploy the underlying Snowbridge instance to your cloud account and begin forwarding events based on your configuration. It typically takes a few minutes to deploy a new forwarder.

---

# Build custom integrations for event streams
> Build custom consumers for Snowplow event streams using AWS Lambda, GCP Cloud Functions, KCL applications, or Pub/Sub client libraries to integrate with any third-party platform.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/custom-integrations/

Snowplow is underpinned by event streams: AWS Kinesis, GCP PubSub, or Apache Kafka. Before a Snowplow pipeline loads the events to a data warehouse, the enriched events are available on a stream. You can build a custom consumer to consume these events. Below we describe some high level concepts which can be used to consume the enriched event streams.

## Transforming the Enriched Stream to JSON

The Snowplow events in the Enriched stream are in a tab separated format (TSV) by default. Many downstream consumers will prefer this data in JSON format, and the [Snowplow Analytics SDKs](/docs/api-reference/analytics-sdk/) have been built to help with this.

## AWS Lambda and GCP Cloud Functions

[AWS Lambdas](https://aws.amazon.com/lambda/) and [GCP Cloud Functions](https://cloud.google.com/functions/) are server-less platforms that allow you to write applications that can be triggered by events from Kinesis and PubSub respectively. By configuring a function to be triggered by an event, it is possible to write applications that take the Snowplow events, perform transformations and other processing, then relay that event into another system.

Server-less functions are an easy way to approach building real time consumers of the event stream for those use cases which require fast action or decisioning based on incoming events (For example, Ad Bidding, Paywall Optimization, Real-time reporting, etc).

## Kinesis Client Library (KCL) applications

The KCL (Kinesis Consumer Library) allows for applications to be built to consume from AWS Kinesis. It makes use of AWS DynamoDB to keep track of shards in the data stream, and makes it far easier to consume from Kinesis than would otherwise be possible.

There is comprehensive documentation on building Amazon KCL apps within the [AWS Documentation](https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html).

## Pub/Sub client library applications

The Pub/Sub client libraries allow for applications to be built to consume from GCP Pub/Sub. It makes it easy to build against and consume events in Pub/Sub streams, ultimately making it far easier to consume from Pub/Sub than would otherwise be possible.

There is comprehensive documentation on building GCP Pub/Sub client library apps within the [GCP Documentation](https://cloud.google.com/pubsub/docs/reference/libraries).

---

# Monitor and troubleshoot event forwarders
> Monitor event forwarder performance, debug failures, and understand retry logic with cloud metrics, failed event logs, and Console statistics.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/

This page outlines how to monitor event forwarder performance and diagnose delivery issues. Snowplow provides both summary metrics and detailed failed event logs to help you understand failure patterns and troubleshoot specific problems.

## Failure types and handling

Snowplow handles event forwarding failures differently depending on the type of error.

### Data processing failures

These failures occur when there are issues with the event data itself or how in how it's transformed before reaching the destination.

**Transformation failures** occur when Snowplow hits an exception when applying your configured JavaScript transformation. While there are safeguards against deploying invalid JavaScript, transformations may still result in runtime errors. Snowplow treats transformation failures as invalid data and logs them as failed events in your cloud storage bucket without retrying.

**Oversized data failures** results from events exceeding the destination's size limits. Snowplow creates [size violation failed events](/docs/api-reference/failed-events/) for these events and logs them to your cloud storage bucket without retrying.

### Destination failures

These failures occur when the destination's API cannot accept or process the event data.

**Transient failures** are those that are expected to succeed on retry. This includes temporary network errors, HTTP 5xx server errors, or rate limiting. Transient failures are automatically retried.

**Setup failures** result from configuration issues that typically require human intervention to resolve, such as invalid API keys or insufficient permissions. When a setup error occurs, Snowplow will trigger email alerts to the configured list of users and occassionally retry the request to check if the issue has been resolved. For more information on alerting, see [configuring setup alerts](#configuring-setup-alerts) on this page.

**Other unrecoverable failures** are bad requests that won't succeed on retry, such as those with missing or invalid fields. These often map to HTTP 400 response codes. Snowplow will log them as failed events in your cloud storage bucket without retrying.

Failure types are defined per destination based on their expected HTTP response codes. See the list of [available destinations](/docs/destinations/forwarding-events/integrations/) for destination-specific details on retry policies and error handling.

### What happens when events fail

When any type of failure occurs, Snowplow can take one or more of the following actions:

- **Automatic retries**: transient failures are automatically retried according to each destination's retry policy. For all HTTP API destinations, Snowplow will retry up to 5 times with exponential backoff.
- **Failed event logging**: all non-retryable failures are routed to your configured failure destination, which is typically a cloud storage bucket, where you can inspect them further. This includes transformation failures, oversized data failures, unrecoverable failures, and transient failures that have exceeded their retry limit. For how to query these logs, see [Inspecting and debugging failures](#inspecting-and-debugging-failures) on this page.
- **Setup alerts**: just like warehouse loaders, setup failures trigger email alerts to notify configured users of authentication or configuration problems.

## Configuring setup alerts

Once a forwarder is deployed, you can configure one or more email addresses to send alerts when setup failures occur. Follow the steps below to configure the alerts.

1. Navigate to **Destinations** > **Destinations list** from the navigation bar and click the **Details** button on a destination card to open the **Destination details** page.
2. On the table of forwarders, click the three dots next to the forwarder you want to configure alerting for and select **Alerts**.
3. You'll see a modal where you can enter the email addresses you want to be alerted in case of setup errors. Click **Save Changes** to confirm.

![Setup alerts modal dialog for configuring email addresses to receive forwarder error notifications](/assets/images/setup-alerting-screenshot-a84253dd9889a28c38179557139d9425.png)

## Metrics and monitoring

You can monitor forwarders in a few ways:

- **Console metrics**: you can view high-level delivery statistics in Console.
- **Cloud monitoring metrics**: forwarders emit a set of metrics to your cloud provider's observability service.
- **Failed event logs**: for failed deliveries, Snowplow saves detailed logs to your cloud storage bucket. Consume these logs for automated monitoring in your observability platform of choice.

### Console metrics

In Snowplow Console, you can see the number of filtered, failed, and successfully delivered events over the last seven days.

To view these metrics, navigate to **Destinations** > **Destinations list** and select the destination you'd like to view. On the event forwarders overview table, you will see metrics for each forwarder configured for that destination.

![Console metrics dashboard showing forwarder statistics including filtered, failed, and delivered event counts](/assets/images/event-forwarding-console-metrics-a7c681591dad5e6ed07e55980b809c1c.png)

### Cloud monitoring metrics

> **Info:** Forwarder cloud metrics are only available for [CDI Private Managed Cloud](/docs/get-started/#cdi-private-managed-cloud) customers.

Forwarders emit the following metrics in your cloud provider's monitoring service:

- `target_success`: events successfully delivered to your destination
- `target_failed`: events that failed delivery but are eligible for retry
- `message_filtered`: events filtered out based on the forwarder's JavaScript filter expression
- `failure_target_success`: events that failed with unrecoverable errors, such as transformation errors, and logged to your cloud storage bucket

You can find forwarder metrics in the following locations based on which cloud provider you use:

- **AWS**: CloudWatch metrics under `snowplow/event-forwarding` namespace
- **GCP**: Cloud Monitoring metrics with `snowplow_event_forwarding` prefix

To get notified of any issues, you can use these metrics to define [CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) or [Cloud Monitoring alerts](https://cloud.google.com/monitoring/alerts).

## Inspecting and debugging failures

This section explains how to find and query failed event logs.

### Finding failed event logs

To better understand why a failure has occurred, you can directly access and review detailed failed delivery logs in file storage. The logs are automatically saved as [failed events](/docs/monitoring/exploring-failed-events/file-storage/) in your Snowplow cloud storage bucket under the prefix: `/{pipeline_name}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_errors/`

For more details on where to find failed events, see [Accessing failed events in file storage](/docs/monitoring/exploring-failed-events/file-storage/).

Failed event logs are formatted according to the [`event_forwarding_error`](https://iglucentral.com/?q=event_forwarding_error) schema and contain:

- **Original event data**: the complete Snowplow event that failed
- **Error details**: specific error type and message
- **Failure timestamp**: when the error occurred
- **Transformation state**: data state at the point of failure

### Querying failed event logs

You can query failed events using [Athena](https://aws.amazon.com/athena/) on AWS or [BigQuery external tables](https://cloud.google.com/bigquery/docs/external-tables) on GCP.

**AWS:**

**1. Create a table and load the data**

To make the logs easier to query, run the following query to create a table:

```sql
CREATE EXTERNAL TABLE snowplow_event_forwarding_failures (
  data struct<payload:string,
              processor:struct<artifact:string,
                              version:string
              >,
              failure:struct<errorCode:string,
                            errorMessage:string,
                            errorType:string,
                            latestState:string,
                            timestamp:string>
              >
  )
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/'
```

If the table already exists, run the following query to pull in new data:

```sql
MSCK REPAIR TABLE event_forwarding_failures
```

**2. Explore failure records**

Use the query below to view a sample of failure records:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM event_forwarding_failures
LIMIT 10
```

**3. Example queries**

Summarize the most common types of errors:

```sql
SELECT
    data.failure.errorType,
    data.failure.errorCode,
        -- time range for each - is the issue still happening?
    MIN(data.failure.timestamp) AS minTstamp,
    MAX(data.failure.timestamp) AS maxTstamp,
        -- How many errors overall,
    count(*) AS errorCount,
        -- There might just be lots of different messages for the same error
        -- If this close to error count, the messages for a single error might just have high cardinality - worth checking the messages themselves
        -- If it's a low number, we might have more than one issue
        -- If it's 1, we have only one issue and the below message is shared by all
    count(DISTINCT data.failure.errorMessage) AS distinctErrorMessages,
        -- a sample of error message. You may need to look at them individually to get the full picture
    MIN(data.failure.errorMessage) AS sampleErrorMessage
FROM event_forwarding_failures
GROUP BY 1, 2
ORDER BY errorCount DESC -- Most errors first
LIMIT 10
```

View transformation errors:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState,
    data.payload
FROM event_forwarding_failures
WHERE data.failure.errorType = 'transformation'
LIMIT 50
```

View API errors:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState,
FROM event_forwarding_failures
WHERE data.failure.errorType = 'api'
LIMIT 50
```

Filter based on a date and hour:

```sql
-- Note that the times in the paths are for the creation of the file, not the failure time

SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM event_forwarding_failures
WHERE "$path" LIKE '%2025-07-29-16%' -- File paths are timestamped like this, so we can limit our queries this way
LIMIT 50
```

Filter for a range of timestamps:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM event_forwarding_failures
-- Here we need the full path prefix
WHERE "$path" > 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/2025-07-29-16'
AND "$path" < 's3://{BUCKET_NAME}/{PIPELINE_NAME}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/2025-07-29-20'
```

**GCP:**

**1. Create a dataset**

First, create a dataset to organize your failed event tables:

```sql
CREATE SCHEMA snowplow_failed_events
OPTIONS (
  description = "Dataset for Snowplow failed event analysis",
  location = "EU"  -- Should match the location of your bad rows bucket
);
```

**2. Create an external table and load the data**

To make the logs easier to query, run the following query to create an external table:

```sql
  CREATE OR REPLACE EXTERNAL TABLE snowplow_failed_events.event_forwarding_failures (
    schema STRING,
    data STRUCT<
      failure STRUCT<
        errorCode STRING,
        errorMessage STRING,
        errorType STRING,
        latestState STRING,
        timestamp TIMESTAMP
      >,
      payload STRING,
      processor STRUCT<
        artifact STRING,
        version STRING
      >
    >
  )
  OPTIONS (
    description="Event Forwarding failures",
    format="NEWLINE_DELIMITED_JSON",
    ignore_unknown_values=true,
    uris=["gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/*"]
  );
```

**3. Explore failure records**

Use the query below to view a sample of failure records:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM snowplow_failed_events.event_forwarding_failures
LIMIT 10
```

**4. Example queries**

Summarize the most common types of errors:

```sql
SELECT
    data.failure.errorType,
    data.failure.errorCode,
    -- time range for each - is the issue still happening?
    MIN(data.failure.timestamp) AS minTstamp,
    MAX(data.failure.timestamp) AS maxTstamp,
    -- How many errors overall,
    COUNT(*) AS errorCount,
    -- There might just be lots of different messages for the same error
    -- If this close to error count, the messages for a single error might just have high cardinality - worth checking the messages themselves
    -- If it's a low number, we might have more than one issue
    -- If it's 1, we have only one issue and the below message is shared by all
    COUNT(DISTINCT data.failure.errorMessage) AS distinctErrorMessages,
    -- a sample of error message. You may need to look at them individually to get the full picture
    MIN(data.failure.errorMessage) AS sampleErrorMessage
FROM snowplow_failed_events.event_forwarding_failures
GROUP BY 1, 2
ORDER BY errorCount DESC -- Most errors first
LIMIT 10
```

View transformation errors:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState,
    data.payload
FROM snowplow_failed_events.event_forwarding_failures
WHERE data.failure.errorType = 'transformation'
LIMIT 50
```

View API errors:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState
FROM snowplow_failed_events.event_forwarding_failures
WHERE data.failure.errorType = 'api'
LIMIT 50
```

Filter based on a date and hour:

```sql
-- Note that the times in the file paths are for the creation of the file, not the failure time

SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM snowplow_failed_events.event_forwarding_failures
WHERE _FILE_NAME LIKE '%2025/07/29/16%' -- File paths are timestamped like this, so we can limit our queries this way
LIMIT 50
```

Filter for a range of timestamps:

```sql
SELECT
    data.failure.timestamp,
    data.failure.errorType,
    data.failure.errorCode,
    data.failure.errorMessage,
    data.processor.artifact,
    data.processor.version,
    data.failure.latestState, -- these are last because they can be quite large
    data.payload
FROM snowplow_failed_events.event_forwarding_failures
-- Here we need the full path prefix
WHERE _FILE_NAME > 'gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/jsonschema-1/2025/07/29/16'
AND _FILE_NAME < 'gs://{BUCKET}/partitioned/com.snowplowanalytics.snowplow.badrows.event_forwarding_error/jsonschema-1/2025/07/29/20'
```

***

---

# Configure Amplitude Tag for GTM Server Side
> Configure event mapping, user properties, entity rules, and session tracking for the Amplitude Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/amplitude-tag-configuration/

> **Tip:** The [Session ID in Amplitude](https://help.amplitude.com/hc/en-us/articles/115002323627-Track-sessions-in-Amplitude) is the session's start time in milliseconds since epoch, so it cannot be derived directly from the `session_id` of your forwarded Snowplow events, which is a UUID. Therefore, in order to populate the Session ID so that your events are stitched into sessions correctly in Amplitude, your Snowplow events need to have the [`client_session` context entity](/docs/sources/web-trackers/tracking-events/session/) attached. Then the Amplitude Tag will automatically populate the Amplitude Session ID based on the `firstEventTimestamp` property of the session the event belongs to.

## Amplitude API Key (Required)

Set this to the API of your Amplitude HTTP API Data Source.

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAATIAAAB5CAYAAAC6Lq3eAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABfTSURBVHic7d15XNTV/sfx1zDAsLth7kuIBgmaC6K4sIiSYtY1vZp6KyNxKZeyq2n200zJtXBL+91S07BLF70pIWi4XHfDrVQyQ36iqAgqm8Cwzfz+mOvoCCqb4lc+z8djHsJ3Oed8vzLvOed8vzOj0uv1eoQQQsHMqrsBQghRWRJkQgjFkyATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxXvig+xmjo4mH1yr8nKD12ew4VBelZcrhHj8qiTI/rLyJjM2Z1dFUQ/18oqbxF0oBCAxrZjvDj/+MNr/ZwHNp14jNUtnsnzfnwU0+eCa8dFudhrvhWeRrTW8nXXtgVxe/yaj1DJfWXGTTce1xt93nc3HfVYa59OKHt2BCPGUqHSQJacXczaliMhftRQVV0WTHmzOy/a4NTEHIDGtiLBqCLJNx7XYalT8eEJbYl3LemqSFjYgaWEDtr9Xl+vZOj7ZWr6QP3u1iAkbs1g1shat6ptXVbOFeGpVOsg2H9cS6G6FU301O8/mm6xznZnKgphbdPo0DfdZafx4Qsv8bbdo81Eq3UKus//PAgDOphTRZe51xm7IxGPudQKX3iQxrfRUfC88i5MXC1l/MI8J32cSf6UQz3nXAVi9J5fJ/8wybjsnMps5kYYQySvU805YJi4zUxmyOp1rd/WminXwydZsXGem0mP+dXb+bnocd9MW6ok5pWXWS/ZEHCs9RM3NDI9GtdS84WXNiYuFZTiTBqnZOl7/JoPp/e3o0drSuPzExUL6fH4D15mpvBuWSW6BnuNJhbjOTDW+gOw9V0DXkOtlrkuIp0Wlg2zTMS2B7awY0M6KTaU8sfPy9ez+wJGPAu2YsDETjYWKX2fX560eNnz6052eSmp2MW/1sCFupiN9ntfwfnjmA+t93cua5a/V4vnGFhz5yPGh7VyxK4eUTB37pjnyYT87ElLvBOXq/+TwW3IRRz6qz9LXajHp+ywy83SllrP9TD6tnjFncGdrrmToOJty/6Hf9Vs6fjiax/ONy9ar0hbqGbUmg8D2GkZ2tTYuv5Wv5801GbzXx44Ts+pTqDMcT4fmFthoVBw8b3hB2PV7PgFtNWWqS4inSaWC7NdLhaRmF9OztSX922mIjS8gK8/0481GdrPGwVrFS+2t0OnhrR7WWFuo6NtWQ9KNO2FS28aMLs9aADDWx4ZjSYXGuaWqEHM6n3f8bKhvb0anFhZ0d7Ywrtt0TMu7frY4WKvo1MIC96bmHE4svRdlCG4N5mbQt62GTcdMh5cXbhQb58i8F97ATKVi9kD7MrVxYfQtzqYUYW6mMln+nz/yaVZXTX93DVYWKkb3tCE2Ph+VCl50s2LHGUMPcudZCTJRM1VqAmbTMS1922owV0OT2mpcG5mz9VetSW/CWJH6v//+90lqpjIM6UpjZaHCzkrF9Vs6almrSt+onK5n62haR13ququZxYzdkMHt/CjUwYD2JRt345aOvefymfsXQzANaKfh7xFZTO9vZ9y3ZT01B6Y/vIdYmucamhMyyJ6Xlt3Eu42lcWh5NVPH6cuGYSSAHnCwMrwG9XfX8N4/sxjdy4abOXq6OFner3ghnloVDrIiHWw5qeX6LR0RR+/0SizNKTXIyiNbqydbq8fRzozC4rL3yszVoNOVvn19ezNu5pSenA0c1Cz5qwOdWliUuv62LSe1FBZDt3vmoQ4mFJjMZ1XU0C7WOD9jztxBDkz8PpPYKfWoa2tGAwczvJwt2Ti6Tol9ujpZkleoZ/nOHPxdLTF/4m+oEaLqVfjPfu8f+ZipIGlhAy4vNjyOfuzI8aRCLt4s/+XLmzk6tp/Jp7BYz+c7btG5hQX2Vg/ujdlYGnptt4ezzzqqOZpUyPVbOn6/WkTUb3cm7V9007Bmfy7FOriSUcyxpDtDx5faa1gUc4v0XB3puYarjFczSx7DpmNa5v7F3ni8lxc3YGRXayKOlbx6WRmvdrSiu7Ml74cbLlz0bG3JqeQiYk7nU6SDPX8UsGpPDgBqM+jzvIZ/xuXRV4aVooaqcJBtOq5lqIe1SQ+gUS01vV01bK7AE9tOY0bUr1razU7jcGIhS4Y6PHSfTi0taOBghsfcNAB8XTR0bG6B12fXmfR9pvE2DYB3/GyxUKvwmJvG2A2ZNK97Z5g5wc+WNg3M8V10g57zb6A2U9HQwXQYmphmuM3k1Y6mvc3XvWzYdkpLXmHVfvVByCAH/kgpYt2BXGrbmLF2VG1CY28ZrgRH36Jn6zuh1d9dg4Vahc9zEmSiZlI9CV8+cjaliGFfpXNyVv3qbooiHTpfwKo9uawPql3dTRGiWsiMisKl5+pYtSeXVzpYVXdThKg2EmQK5zH3OjaWKl5+QYJM1FxPxNBSCCEqQ3pkQgjFkyATQiieBJkQQvEq9Ralk5eKCP35FrHx+chEmxDVb8+4qr05u6rY2lhTr25tNJpH8xa6CgfZyUtFDFh6Az2G903KJQMhql/rVi2quwmlysjM5mLyVZo3bfRIwqzCQRb68y30wFAPK+a/6oCledW8uVsIUX6P4uPgq1LtWoYPWrhxM4PGjZ6p8vIrPEcWG294r6WEmBCiLGrXsicn99F8onOFe2R6DMNJCTEhRHWTq5ZCCMWTIBNCKJ4EmRBC8STIhBCKJ0EmhFA8CTIhhOJJkAkhFO+xBpmTk9PjrE4IUUNIj0wIoXgSZEIIxavUx/gIIZQrIiKCwsI73+86ZMgQzM3N0Wq1/Pvf/zYu12g0DBo0CIC4uDgSEhKM6zw8PHB2dn58jb4PCTIhaqj8/HwKCgpKLNfr9Wi1pX+uWVFRkcm64uLyfxn3oyBBJkQNkpWVRW5uLmAIrLtdu3YNtVpNfn6+yXKdTkdKSgqAcd/bMjMzjeueeeYZzMyqZ7ZKgkyIGuTUqVOcO3eu1HU7duwodXlBQQHR0dGlrjtz5gxnzpwBYPjw4Wg01fNt9zLZL4Qolb29PebmyujrSJAJIUrl7e2No6NjdTejTCTIhBCKp4x+oxDisVCr1SV+v71Mp9OVuEDwpJAgE0IYDRw40Dhhr9Fo8PHxMd5i8csvv5CYmFidzbsvCTIhhNHdN8IOGDCAo0ePGm+veJLJHJkQQvEkyIQQiidDSyFEqaKjo9HpdNXdjDKRIBOiBnnuuedo3LjxIynbwsLikZRbFo81yJ7UKx5C1BSOjo6Kucm1PGSOTAiheBJkQgjFkyATQiieBJkQQvEkyIQQiidBJoRQvAoHmQowU0FB0ZP5bnghRM1R4SDzf16DTg8fbsqSMBNCPFRGZja2NtaPpGyVvoIfMHTyUhEDlt5Aj6FnppMsE6LaXV7coLqbUKqMzGzSrt+kedNGaDSWVV5+hYMMDGEW+vMtYuPzkRwTovrtGVf617hVN1sba+rVrf1IQgwqGWRCCPEkkKuWQgjFkyATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxZMgE0IongSZEELxJMiEEIonQSaEUDwJMiGE4kmQCSEUT4JMCKF4EmRCCMUzr4pCTv52mqiYnykqLq5YI9RqAl/swwvt3KqiOUKIGqZKemSVCTGAouJiomJ+roqmCCFqoCrpkd0OsVnTP6jQ/p98trhSQSiEqNlkjkwIoXgSZEIIxVNskOXm5uLk5MThw4dNlp8/fx4nJyfOnz//SOotLi7Gw8ODuXPnPpLy77Z27VoCAgIAw3F5eXlx7dq1Kiu/Z8+ehIeHP3S7BQsWsHLlynKVPXPmTJycnEo8Jk6cCMDAgQNZunRpif26dOlCWFhYuep6mmRlZRF39DixO/ew/+BhLl+5Wt1NUoQqmSOrSQ4dOkR6ejo//fQTM2bMwMzs8bwWNGzYkKCgIOrWrQtAZGQks2fP5tixY4+8br1ej5WVVbn369y5M5999pnJMjs7u6pq1lMlKzub78MjOHwkDrVajYODPbm5eWi1Wp5t2YK/jRiG07Mtq7uZTywJsnKKjIxk8ODBREdHc+TIEbp16/ZY6rW1tSUoKOix1HUvvV5foQCysbGhVatWj6BFFafT6arsxaeqykpNS2PBolDUajXjx75Nh/btMDc3R6/X83//d4HNWyIJWbCEccFv0aljhypo+dNHsUPLsoqPj2fIkCG0bdsWX19fIiIiTNavWbMGLy8vOnbsyOTJk8nIyLhvWYWFhWzfvp2BAwfSu3dvIiMjTdYPHDiQuXPnMmzYMFxcXAgICOD06dOsXbsWDw8POnTowMyZM9HpdACEhITw5ptv8u677/LCCy/g5eXF5s2b73scTk5OpKenM2bMGCZNmkR6ejpOTk7s2rWLHTt20L59e5N9hgwZYjJ8+/bbb/Hw8KBLly4sWLCA4nuuFO/bt49+/frh5ubG0KFDOXfuHADe3t506GB4Av3000/4+/vTtm1bBg0axMmTJx90+qvUg+rOy8tjxowZtG/fnu7du7N06VLj8YWEhDB06FDeeustXF1dCQ8Pp0uXLsb/B4Dx48fz0Ucflbus/Px80tPTmThxIu3bt8fT05N58+aVOLf3U1RUROjyVTg42DPr4w9p6+rKP775lnET3mP2p4be7JTJE/Dx7slXX68l+fKVKjmXT5unPshGjx5Nu3btiImJYfz48cyYMYNff/0VgA0bNrBu3Tq++OILwsPDSU1NZdasWfcta8+ePZiZmeHp6cmLL75ITEwMRUVFJttERUUxbtw4IiIiqFOnDiNGjODQoUOsW7eOkJAQIiIiiI6ONm7/yy+/MHjwYA4ePMiYMWOYOnUqSUlJDzymZcuWsWjRIurUqcPp06fx9vZ+6HmIi4vj008/JTg4mPXr16PT6UhJSTGu//333xk3bhxjxowhKioKNzc3goKCKCwsxMvLizZt2pCYmMj777/PhAkTiI6Oxt3dnaCgIPLz8x9af2U9rO4PPviAy5cv869//YslS5YQHh5uMv937NgxfHx82LJlC/379ycrK4sTJ04Ahheoffv20b9//3KXZWlpyZw5c0hLS2Pz5s0sXbqUrVu38s0335TpuHb/Zx83btxk4jtjsbWx4buN/+Tcnwn89dW/UKuWA18s/xKtNp8Rw4bQrGlT/hXx76o6pU+VpzrIcnNzSUlJoVevXjRr1owhQ4awcOFCrK2tAfjHP/7B3//+dzw9PWndujXTp09n27ZtJcLptsjISPr06YNarcbb25uCggL27dtnss2IESPw9vbGzc2N0aNHk52dzbx582jbti39+vXDw8OD06dPG7fv0qULPj4+2NjY8MYbb+Dm5samTZseeFwajQZLS0vAMHxTq9UPPRdhYWH4+voyevRoXFxcmD59Og0aNDCuX7NmDa+88gqvvPIKLVq0YMaMGWRmZnL06FHjNklJSahUKvz8/GjevDnTpk1j0qRJaLXaUuvcv38/Li4uJo+CgoKHtrU0D6o7OTmZmJgYFi1aRJs2bejatSujRo1iy5Ytxv09PT15/fXXcXFxwd7enp49e/Lzz4absA8ePIilpSVdu3Ytd1kqlYrExEQ6dOhAq1at6Nq1K6Ghobi4uJTpuA4ficOrmyd16tQG4PKVq7w2dDC+Pr0IDhpFdvYtLiUno1KpCOwXwKkz8eTk5FToHD7Nnuo5MhsbG4KDgxk3bhw+Pj74+fnRv39/bGxsyMrKIjk5mWnTpjF9+nTAMBdUXFzM1atXadasmUlZubm57Ny5kxUrVgCGMPH19WXr1q34+voatzM3v3NKbW1tAYwT9LfblJube982u7q6cvHixcof/D0uXLiAv7+/ybK72xofH09CQgI//vijcVleXh6XL182/u7l5UWnTp3w9fUlICAAf39/Ro4ced95oo4dO5aY7LewsKhQ+x9U95EjR9Dr9fj5+Rm3LyoqwtHR8b71BgYGsmzZMj788EN27txJ3759UavVxMfHl7us8ePHM2XKFI4fP46fnx8DBw6kYcOGZTqu5MtX6O3nY/z9k/+ZYfx53/6D2NvZ0bRJEwCcnZ3Q6XRcTbmGcyunMpVfUyg2yCwsLDAzM6OwsNBk+e1XfI1GA8C0adMYNmwYsbGxhIWFsXjxYn744Qdq1za8An7++ee4urqalNGoUaMS9cXGxpKXl8fYsWNRqVSAYbLX0tISrVZboat6pbn3eKqKSqUy9uLuZ9SoUbz22msmy+rVq2f8WaPRsHHjRo4fP87u3bv5+OOPadKkCWFhYSaheNuDJvs1Gk2pvbOCggLj/92929+vbgBLS0uioqJM9nlQT7VPnz5Mnz6dP//8k9jYWObPn29cV96yAgIC8PT0JDY2ltjYWEJDQwkNDaVv37733ee2oqJCLEsJ95+2xbAlchtTp0zCxsba2C6gwr3ap5lih5YWFhY0a9bMZJgGcPr0aaytrWncuDEJCQksWbKE5s2bExQUxObNm6lVqxZbt27FwcGB+vXrk5ycTIsWLYwPR0fHUp+UkZGRDBgwgG3bthEVFWV8qFQqdu3aVeHjuHvCGeDUqVO0aNHiofvdDtPb7OzsyMvLMynv7lB89tln+eOPP0z20ev1xp+dnZ1JSEgocS7uvlq5e/dufvjhBzp27MiUKVPYsmULcXFxFZrwd3Z2LvF/d/HiRbKyskoNvwfV7ezsTEFBAdnZ2ca2N23a1KQXdS9bW1t69erFF198gVarxcvLy9iu8pRVUFDAwoULycnJYfDgwaxevZrBgwezfv36Mp2H2rVrk1LKvYFR0Tt4+aX+tHa+cy6upRi2u7uHLwwUG2QAb7/9NqtWrSI8PJyzZ88SGRnJwoULCQoKwszMDDs7O77++mtCQ0O5dOkSBw4cICUlBScnQ7c8ODiY5cuXExkZyaVLl1i5ciWDBw82eYIDZGZmsnfvXgYNGkSrVq2Mj9atW5d69bI8Dh8+TEREBBcvXmTJkiUkJiby6quvPnS/OnXqkJGRwYEDB8jIyKB169aoVCq+/PJLzp07x+rVq/ntt9+M2w8fPpxt27axc+dO8vLyCAsL48qVO1fA3n77bfbs2cPy5ctJSkpi+/bt9OrVy+SCQEFBAbNmzSIqKorLly8TFRWFubl5iWH4bXl5eSQlJZk8bt/Q+8Ybb3Do0CEWLVrEmTNnOHDgABMmTKBz587GK6R3e1DdTk5O9O7dmylTphAXF8f58+eZOHFiiWHtvQIDA4mJiaFPnz7GF6/ylmVpacnu3buZPXs2CQkJxMfHc/LkSePf2MO0c2vLocO/mPzN6fV63vzbcHp4md7ac+DwERwd69GwwTNlKrsmUezQEgxPTpVKxZo1a7hw4QKNGzcmKCiI4OBgwHAT6ddff838+fP56quvqFevHsHBwcarU6NGjSInJ4d58+aRkZGBu7s7S5YsKdHbiYmJwcrKiu7du5doQ2BgIBMmTCA7O7tCx9CuXTsOHTrEvHnzsLa2NvYgH8bT05Nu3boRHBzMsmXL6N27N3PmzCE0NJQ1a9bg5+dH586djdt37tyZadOmMW3aNPR6PX379sXZ2dm43t3dnRUrVvD555+zfPlymjdvzuzZs03megICApg6dSoLFiwgNTWVli1bsnLlSpOLBneLi4szmT8Ew1zXd999h4uLCxs2bGDx4sWsX78eKysr/P39mTFjRqllPazuxYsX88knnxAUFGSc45o8efIDz6G/vz9WVlb069fPZHl5y/rqq6+YPXs2L7/8MhqNBj8/P6ZOnfrAum97sa8/e/cfJDIqhoEDDO3Q6fWs+t9vmDxxPC/Udgcg4Xwiu3bvZeTwoWUqt6ZR6e/tflTAvIVfVPrTK8zVaj6a+l5lm6IoISEhnDt3jnXr1lV3U0Q12rN3P99u2MiA/gG8/FJgiamNY8dP8PXa9bR93pV3xo4u8UIrqqhHFvhinyr5YEUhaiKfXj0wV6vZsDGcfQcO0aF9O+rWrUtuXi7x8We5eCkZn149GDl8qITYfVRJkL3Qzk0+3VWISujRvRvubm3Zs3cfv589xx/n/sTK2hpnZyfefH0Ez7Z8+AWgmqxKhpZCCFGdFH3VUgghQIJMCPEUkCATQiieBJkQQvEkyIQQiidBJoRQPAkyIYTiSZAJIRRPgkwIoXgSZEIIxZMgE0IongSZEELxJMiEEIonQSaEUDwJMiGE4v0/o6b4azBd12AAAAAASUVORK5CYII=)

### Use Amplitude's EU servers

Enable this option to send the data to Amplitude's EU Residency Server [endpoint](https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/#endpoints), instead of the default standard server endpoint.

## Snowplow Event Mapping Options

### Include Self Describing event

Indicates if a Snowplow Self Describing event should be in the `event_properties` object of the Amplitude event.

### Snowplow Event Context Rules

This section describes how the Amplitude tag will use the context Entities attached to a Snowplow Event.

![](/assets/images/02-gtm-ss-amplitude-6f8f90af92e7fefe68e952297bed66cd.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event\_properties

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Amplitude's `event_properties`.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in `event_properties` or `user_properties` of the Amplitude event (default value is `event_properties`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event_properties` is set to `All`), you can specify the context entities you want to exclude from the Amplitude event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in event\_properties

Leaving this option enabled ensures that all Entities on an event will be included within the Event Properties of the Amplitude event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Amplitude event, and can be included in either event properties or user properties. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1` where `com_snowplowanalytics_snowplow` is the event vendor, `web_page` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

##### Include unmapped entities in event\_properties

If remapping or moving some entities to User Properties with the above customization, you may wish to ensure all unmapped entities are still included in the event. Enabling this option will ensure that all entities are mapped into the Amplitude event.

## Additional Event Mapping Options

If you wish to map other properties from a Client event into an Amplitude event they can be specified in this section.

![](/assets/images/03-gtm-ss-amplitude-124150b9c2edaf9f44f0ae0067a19c0d.png)

### Event Property Rules

#### Include common event properties

Enabling this ensures properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Amplitude Event Properties.

#### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Amplitude `eventProperties` object.

### User Property Rules

#### Include common user properties

Enabling this ensures user\_data properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Amplitude Event Properties.

#### Map Snowplow mkt fields (standard UTM parameters) to user properties

Enabling this option automatically maps all the marketing (`mkt_` prefixed) fields of the Snowplow event to the standard UTM parameters in Amplitude's user properties.

#### Additional User Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Amplitude `eventProperties` object.

### Groups Property Rules

> **Note:** This configuration option is relevant **only if** you have set up [account-level reporting in Amplitude](https://help.amplitude.com/hc/en-us/articles/115001765532).

#### Groups Property Mapping Rules

Specify the Property Key from the GTM Event, and the key you would like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0). These keys will populate the Amplitude `groups` object.

## Additional Properties

In this section you can additionally set specified event or user properties to custom values (e.g. through a GTM Server-side variable).

![additional\_properties](/assets/images/05-gtm-ss-amplitude-e77fabbf0070687d579898d733c60e33.png)

### Event Properties

Using the **Additional Event Properties** table allows you to set additional **event** properties in Amplitude payload. To do so add a row and simply specify the property name for Amplitude `event_properties` and then the value you would like to set it to.

### User Properties

Using the **Additional User Properties** table allows you to set additional **user** properties in Amplitude payload. Similarly to the previous table in the section, add a row and specify the property name for Amplitude `user_properties` and then the value you would like to set it to.

### Groups Properties

> **Note:** This configuration option is relevant **only if** you have set up [account-level reporting in Amplitude](https://help.amplitude.com/hc/en-us/articles/115001765532).

Using the **Additional Groups Properties** table allows you to set additional **groups** properties in Amplitude payload. Similarly to the previous tables in the section, add a row and specify the property name for Amplitude `groups` object and then the value you would like to set it to.

## Advanced Event Settings

In this section you can find advanced configuration parameters.

![advanced event settings](/assets/images/04-gtm-ss-amplitude-5cf1cc255705dea76631842bde590aa6.png)

### Forward User IP address

Enabling this will forward the IP Address to Amplitude, otherwise Amplitude will not receive the users IP Address (default: `True`).

### Fallback platform identifier

If there is no Platform property on the Client event, this is the value which the Tag will forward to Amplitude (default: `Web`).

### Amplitude time setting

This option allows you to decide whether the event time of the Amplitude event will be set. The available options are:

- `Do not set` (default): this means the event time will be set automatically by Amplitude.
- `Set to current timestamp`: sets the Amplitude event time to the current timestamp.
- `Set from event property`: sets the Amplitude event time from the client event property. For example, in the image below the Amplitude's event time will be set from the device created timestamp (`dvce_created_tstamp`) of the Snowplow event (`x-sp-` prefix in the client event):

![](/assets/images/07-gtm-ss-amplitude-dda2f8b6d3e860dc2e949e845b21440b.png)

### Device Identifier

#### Inherit Amplitude `device_id` from common event `client_id`

By default the Amplitude tag sets the `device_id` property of the Amplitude event from the `client_id` property of the common event. Unchecking this tick box allows you to override the value for `device_id` in Amplitude event payload.

### User Identifier

#### Inherit Amplitude `user_id` from common event `user_id`

By default the Amplitude tag sets the `user_id` property of the Amplitude event from the `user_id` property of the common event. Unchecking this tick box allows you to override the value for `user_id` in Amplitude event payload.

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Amplitude Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Amplitude GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "Amplitude HTTP API V2", // the name of the tag
    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",    // the "trace-id" header if exists
    "EventName": "xxx"   // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# Amplitude Tag for GTM Server Side
> Forward Snowplow events to Amplitude from GTM Server Side using the Amplitude Tag with HTTP API v2 for product analytics and user tracking.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/

The [Amplitude Tag for GTM SS](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-server-side-amplitude-tag) allows events to be forwarded to Amplitude. This Tag works best with events from the Snowplow Client, but can also construct Amplitude events from other GTM SS Clients such as GAv4.

The tag is designed to work best with Self Describing Events, and atomic events, from a Snowplow Tracker, allowing for events to automatically merged into an Amplitude events properties. Additionally, any other client event properties can be included within the event properties or user properties of the Amplitude event.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “Amplitude HTTP API V2” and select the official “By Snowplow” tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking “Add”

## Amplitude Tag Setup

With the template installed, you can now add the Amplitude Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Amplitude Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Amplitude
3. Enter your Amplitude API Key for a HTTP API integration. This can be retrieved from Amplitude Data Sources within your Amplitude project.
4. Click Save

---

# Configure Braze Tag for GTM Server Side
> Configure authentication, user identifiers, event mapping, and entity rules for the Braze Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/

## Configuration options

### Braze REST API Endpoint (required)

Set this to the URL of your Braze REST [endpoint](https://www.braze.com/docs/api/basics/#endpoints).

### Braze API Key (required)

Set this to your Braze [API Key](https://www.braze.com/docs/api/basics/#app-group-rest-api-keys) that will be included in each request. The minimum permission that you need to assign for this API Key is to access the `/users/track` endpoint.

![key permission](/assets/images/key_permission-0327a99d1c82d569bcce97d5ae5508e4.png)

### Identity settings

#### Braze User Identifier

This section allows you to select which Braze user identifier (external user ID (`external_id`) or [User Alias](https://www.braze.com/docs/api/objects_filters/user_alias_object#user-alias-object-specification) `user_alias`) will be used by the tag. The default value is `external_id`.

![identity settings dropdown](/assets/images/identity_settings-bf117a2d0fb59b9460f358912d794ac5.png)

##### Braze external\_id

This configuration section is enabled if you have selected the `external_id` as the **Braze User Identifier**.

![braze external id](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAe8AAADHCAYAAAAwAdsiAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15eFTl3f/xdxaykpiFEJIgoSGBBBIi+74GAgSw0UcNKikqaLW1rVWqQFv118dKXWsrVikqKvWxigsKsq+iyBZBgQABAiSENStZJzM58/uDZiRNgCQGJ0c/r+uai5lzzn2f70mu8Jn7PmfOuJSUltn5gbHb7fWeG4bh+Ndut2MYBjU2GzWGgc1mczwsFgtWq5V+fXo7pXYREZErcXV2ASIiItI0Cm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTcXd2ASIAx44d4x//+IfjtYeHB+3bt2f06NHEx8c7sbJvzZo1y3Gb3YtNmDCBUaNGOaGiptm/fz8LFy7klltuoW/fvvXWv/POO+zatYs//vGP+Pn51Vu/fft23n//fe6++25iYmK+j5JF5BIU3tKqxMTEEBUVRXV1Nbt372bRokX88pe/pFOnTs4uDYDg4OB6wRcVFXXFduvWrWPVqlU88MADhIeHX63yRORHQuEt9Rw7doxDhw4xduzYBtevWbOGmJgYOnfu3OL7joqKIikpCYDu3bvz0ksvkZmZ2WB4G4aBq+v3e+YnMDDQUd/3yRnHKiKtl8Jb6lm3bh0HDx6kpqaG8ePH11m3YsUKNmzYQG5uLnfddddVrcPHxwcAi8UCwNatW/nwww8ZOnQou3fvZujQoYwePZpdu3axfv16CgoKCAwMJDk5mcTERADmzp1LUVFRvX4ff/xxADIzM1m5ciWFhYWEhoYyceLERo2k/9uWLVtYsmQJN910E/3792fXrl288847TJgwgZycHPbt2wfACy+8wLhx40hKSsJut7Nx40a2bNlCdXU1nTt35oYbbiAgIKDBY+3evTvPP/88I0aMIDc3l5ycHDp06EBaWhqhoaEAnDhxgmXLlpGTk4OPjw/9+vUjOTkZFxeXJh9TZWUlixcv5uDBgwQHB3Pttdc2uQ8RuTr0Vl7qSU9Pp3Pnzqxfv56VK1c6ltcGd+fOnZk6depV2bfFYqG0tJT8/HxWrVqFi4uLI4hr7dixg5iYGCIiIsjOzuadd94hICCAG2+8EV9fX9555x3Onj0LQHJyMqmpqaSmptKrVy8AunbtCkB2djZvvvkmfn5+JCcnYxgGr7/+OufPn79kfTabjaKiIsejdtuBAwcSGhrK2rVrqaqqYuXKlQQFBTFs2DBGjRrl2PfkyZPp2bMncOFN0ooVK4iLi2PkyJHk5OTw1ltv1flK24uPtdbmzZvp0KEDAwcOJC8vj08//RSAqqoqFixYwPnz50lNTaVbt26sW7eOHTt2NOt38emnn7J371569OhBbGws33zzTbP6EZGWp5G31OPh4cGMGTN49dVXWb9+PXBh2nbjxo107tyZGTNm4OHhcVX2vXHjRjZu3Oh4PWDAgHrT86mpqfTp0weAgoICpk2bRlRUFN7e3gQFBfHKK69w7Ngx2rdv79iuqqqKTZs24e/vT2pqKgCbNm3C19eX22+/HXd3d6Kiovj73//Onj17GDJkSIP1HTt2jLlz5zpeh4aG8tBDD+Hq6srkyZN59dVXeeWVVygqKiI9PR13d3c6derEoUOHAOjSpQshISEYhsHmzZvp3r07kyZNAsDd3Z2lS5dy5syZBo/19OnTAAwZMoTJkycDkJOTw4kTJ4AL311/yy23EBoaSrt27ejbty8ZGRkcOXKE/v37N+n3YLPZyMjIICoqittuuw2Atm3bsmzZsib1IyJXh8JbGtRQgF/t4Abo06cP1113HYZhcODAAb788kvCwsIYPHiwYxt/f3/H8+DgYLKzs3n11VcpLCykuroa+HaqvdbHH39McXEx06dPd0zHnzx5krKyMscUeq3/nma/WFhYGMnJyY7Xnp6ejuddu3YlNjaWAwcOEBUVRUJCwiX7KSoqorKykszMTP7whz/UWVdYWNjgsdaqrR/Ay8vLccze3t74+PiwZMkSzpw5Q2VlJYZhONY3RVFRETU1NfzkJz+psy8RaR0U3nJJFwc4cNWDG6Bdu3Z069YNgNjYWL7++mt27txZJ7wvtmfPHhYvXszgwYO56aabyM/PZ9GiRXW22bt3LxkZGQwePNgxZV4rODiYtLS0OssaCsxavr6+9OjR45LrKysrHf/a7fYrnmtOSEhg2LBhdZa1b9++WVPUxcXFvPbaa3Ts2JGpU6fStm1bnn322Sb3AzgujnNzc2tWexG5uhTeclm1AV77/PtktVqx2Wy0adPmktscPnwYgLFjx+Lr60tpaWmd9WVlZXzwwQeEhIQwceLEOutCQ0M5cuQIAQEBBAQEAJCbm0tQUFCz6t29ezfHjx93jL63b9/OgAED6mxTez47MDAQDw8Pzp49S2RkJC4uLtjtdvLy8uqMrJvi+PHjVFdXM2TIECIjI5s14q4VEBCAu7s7p06dciyz2WzN7k9EWpbCW67o+wzto0ePsmHDBmpqasjMzKS6uvqyN2kJCQkB4MMPP6Rjx458/vnnAI6bqbz//vuUl5cTHx9PRkaGo12/fv0YPXo0WVlZ/POf/6R///4UFBSwfft27r333jrTxRcrLi5mw4YNdZZ17tyZjh07snz5ckJCQpg2bRrz5s1j5cqVJCYm4uXl5RjNb9iwgYEDBxIdHc3IkSNZvXo1CxYsIDY2loMHD5Kbm1tvGr2xQkJCcHFxYf369RQXF7Nr1y5qamoavLHMlbi5udG3b1+2bdvGmjVr8PPzY+3atc2qS0RansJbWpWsrCyysrJwc3MjKCiIlJQUhg4desntBw0axNmzZ/n66685ceIEw4YNY/ny5Y7zxtnZ2QBs27atTrvExESioqJIT09n9erVrFq1Cn9/fyZOnHjJ4AbIz89nxYoVdZYlJSVx+PBhiouLSU9Px83NjZSUFBYsWMCaNWscV5hnZGRw4MABOnbsSHR0NKNHj8Zut7N9+3aOHj1Khw4dmDp1arPfLIWHh3PDDTewbt061q1bR79+/aioqKhzDr0pUlJSKCsrY8OGDQQGBtKzZ0+2bNnSrL5EpGW5lJSW2a+8mblc/FGb2ue1ow/DMLDb7RiGQY3NRo1hYLPZHA+LxYLVaqVfn95OqV1ERORK9DlvERERk1F4i4iImIzCW0RExGQU3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRuEtIiJiMgpvERERk1F4i4iImIzCW0RExGQU3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRuEtIiJiMgpvERERk1F4i4iImIzCW0RExGQU3iIiIibj7uwCROTHx2KxcDDrEPn5Bbi5uREW1oHoLlG4umo8IdIYCm8R+d7UGAZLly1nxaq1VFdXc801/tTYaigtKyMwIICbb7qBwQP7O7tMkVZP4S3iBEuWLMHX15exY8c2q/2iRYuIiYlh4MCB9daVlZWxaNEi8vLy+O1vf0twcPB3LbdFWK1W/vr3f5B99Bg3/HQSw4YMwtfXF4D8/AJWrl7LgtfeIDf3BGk33+jkakVaN4W3OM2RI0dYvnw5p0+fxs/Pj+HDhzN48OArttu2bRtRUVGEhIS0SB3nzp0jOzubAQMGtEh/LeWll15i4sSJdO7cuUnttmzZgo+PD4899hju7q3nT3zR2//meE4uf5g1k4iIcD76eBkbNn2GRxsPUn86iam3pdE1JpqX//kaYR1CGT5siLNLFmm1Ws9ftvyoVFRU8MYbb3DTTTfRvXt3zp49y5tvvomfnx8JCQmXbbtt2zb8/f1bLLzz8/PZtm1bk8LbbrcD4OLi0iI1NOSnP/0poaGhTW5XWFhIVFQUbdq0qbfObrdf1ZovJTf3BJu/+JJf/eLndOwYwWebv+DTFatIvX4SpWWlvP7GIsI6hNK/Xx+OHc9h8YdLGDigHx4eHt97rSJmoPAWpygoKMDV1ZXExEQAIiIiGD9+POXl5QAYhsGnn37Kjh07aNu2LZMnTyYuLo6//OUvFBcX8+9//5tx48bVG6lfqt3Zs2d58cUXeeihhwgICGDRokX4+/vTvn17Vq5cicViYe7cucyePRuAnTt3smrVKgzDYPDgwSQlJQHw+OOPExsby759+3jggQc4ePAgR44cwTAMDh8+TKdOnUhPT8fLywubzcZHH33E3r178fLyYuLEifTs2bPRP6N3332X1NRUunTpgtVqZfHixRw4cICIiAhsNluDbZYsWcI333zDvn37OHjwIDNmzGDRokXY7XaOHz/OuHHj6N+/P9u3b2ft2rVYrVYSEhKYPHkybdq0YfPmzRw6dIiamhqOHz9OZGQk48ePZ/HixRQWFtK9e3emTJmCq6srn3zyCT4+PowZM+aKx7J1+046dAild68Lv++jx3MYmzSKyRPHA5CZeYDM/QeJ7hJFyvhkVq5ey77MA/S6rvE/L5EfE13aKU4RFhZG27Ztef/99yksLASgd+/ejnO4n332GXl5ecyePZu0tDTeffddKisrmTVrFuHh4UyZMqXBKfZLtWvfvj0DBgxgxYoVHDt2jKNHj5KcnMygQYOYMmUK4eHhjuDOyclh+fLl3H333fzmN78hIyODw4cPO/bh7e3N7NmzCQoKAuDgwYOMGDGCOXPmUFVVxa5duwBYsWIFVquVRx99lJtuuol33333kqF7JRs2bOD8+fM8/PDDjB8/nnPnzjW4XWpqKgkJCSQnJzNjxgzH8rKyMn71q1/Rt29fsrOzWblyJXfeeSePPPIIxcXFrFq1yrFtbm4uKSkpzJo1i7KyMt5++22mTZvGww8/TG5uLgcOHACgf//+9OrVq1H1n8jLIya6i+P1tKm3MuWW/wEg69Bhzp47R2y3GADatvWlQ2h7TuSdbNoPSeRHROEtTuHu7s4vf/lLvLy8mDdvHi+99JIjFAAyMjIYNWoU3t7eREZGEhERQXZ29hX7vVy7sWPHkp2dzXvvvUdKSgre3t4N9vHVV1/Rt29f2rdvj7+/P3379mX//v2O9YMGDcLHx8cx/RwXF0fnzp3x9vYmKiqKgoICAIYMGUJqaipubm7ExMTg4uJCUVFRs35ee/fuZeTIkbRt25bIyEi6dOly5UYX6d27NwEBAbi6urJr1y769+9PWFgYXl5eTJgwgYyMDMe20dHRRERE0LZtW2JiYujevTvBwcH4+/sTGRnpOL4OHTo0+mI4q9WGRwPT+AcOZvH0c3/jxtTr6RoT7Vju4eGB1VrdpGMU+THRtLk4jY+PD5MmTSIlJYV9+/bx3nvvceONNxIfH8/58+f517/+5QhIwzAaNeV8uXaenp706dOHL7/8kt69e1+yj5KSErKysti6dStw4Txxjx49GnVMLi4u1NTUABfeoCxdupSTJy+MIK1Wq+NceVOVlZURGBjYrLb/rbS0lIiICMfrwMBAysvLsVqt9bb9789du7i4YBhGk/cZFBjA6TNn6i1fvWY93brGMD7526l3wzA4dy7fMbMhIvUpvMUpvvnmG4qKihgxYgSurq4kJCSQl5fHvn37iI+Px8/Pj5tvvpnIyMgm9Xu5dhUVFWzfvp3g4GC2bt16ySvb/f39GTNmDKNGjWrWsdX68MMP6dSpEzfffDOurq48+uijze7Lz8/PcT3AdxUQEEBJSYnjdVFREV5eXg1e4NZSEuJ7sOD1NyksLCIo6Ns3IWOSLswmXGz3N3uoqKykR/fYq1aPiNlp2lycIjAwkHXr1pGZmYnVaqWwsJCDBw8SFhYGQGJiIqtXr6aiooKKigqWLl3qCBwPDw8KCgocI9yLXa7d8uXLSUhIIC0tjdWrV1NWVubor6ysjMrKSgB69uzJli1bOHPmDFarlc8//5ysrKwmH2NZWRkuLi5YLBa2bduGxWJp9si7R48efPHFFxiGQXFxMTk5Oc3qBy5MoW/fvp0zZ85QXV3N6tWrG33u+mJnzpxxXK9wJX379iY4OIiFb/2rzsj9o4+XseXLbY7XpWVl/N+/FzOgX19C2rVrck0iPxYKb3GKa6+9lltvvZW1a9fy2GOP8fLLLxMTE8PQoUMBGDVqFKGhoTz77LM8/fTTuLq64u/vD1y4UGrVqlV8/vnn9fq9VLucnBz27NlDcnIyoaGh9OzZk2XLlgEQGRmJn58fTz75JABdunRhzJgxLFy4kD/96U8cOXKEjh07NvkYJ02axM6dO5k7dy4nT56kU6dOjjcMTTVq1Cjc3Nx48sknefvtt7/TFHqnTp1ISUnhjTfe4IknnnBcCd9UX375JV999VWjtnVzdeUXP59B1qEj/G3ey5wvLQXg97NmOi5cO3Eij7lPPYebqyvpt6c1uR6RHxOXktKy5g0FWrGLRze1z2vf7RuGgd1uxzAMamw2agwDm83meFgsFqxWK/36XPqcqEhLyc3NZf78+fWWP/jggz/Ic75Hjx1n3sv/pKysnOsSE4gID8Nmq+HosWPs3bef6C5R3H/fPVxzjb+zSxVp1RTeCm+R75XVauXzL77k6z17yS8oxM3VlbAOHejfrw+9ruvplJvIiJiNwlvhLSIiJqNz3iIiIiaj8BYRETEZhbeIiIjJKLxFRERMRndYE6exWKopKCymvKLS2aWISCvn6+NNcFAAnp76mlhQeIuTWCzV5Jw4RUi7IMLD2ju7HBFp5YpLSsk5cYpOHcMU4GjaXJykoLCYkHZBBFzj5+xSRMQEAq7xI6RdEAWFxc4upVVQeItTlFdUKrhFpEkCrvHTabb/UHiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZPTFJGJqY8aMITs7u97y//3f/+X22293QkVw7tw5BgwY4Hjt7u5OZGQk6enppKen4+Li4pS6ROSHQ+Etpjd9+nSmTJlSZ1lISMhV219tOK9cuZKuXbtecrsXXniB2NhYampq2L17N3/+858xDIM77rjjqtXWFI09DhFpfRTeYnrBwcF06dLF2WXUc+211zpCMS4ujoKCAl599dV64W0YBq6u398ZrJqaGtzc3L63/YlIy9M5b/nBGjlyJIsWLXK8zsrKokuXLpw+fRqAzZs3M2HCBOLj40lLSyMrKwuAiooKoqKiePfdd5kwYQI9evRg2rRpFBYWsnDhQseU+Pjx43nyyScbXU98fDwnT57EarXy5JNPkpaWxl133UVcXBwWi4WKigpmz57NddddR2JiIo888ggVFRV1apo3bx5JSUnExcUxffp0ioqKHP1XVlYyZ84cEhMTGTJkCH/729+oqakB4Prrr2fmzJlMmDCBtLS0Bo/jvvvu495773X0V1JSQkxMDBs3bmzGT19+zMrLy6murqaqqgqLxVLnUVVVRXV1NeXl5c4u09Q08pYfrJSUFFavXk16ejoAa9eupXfv3nTo0IH9+/dz33338cQTT9CrVy/eeustpk+fzvr16x3t58+fz5///Gfc3d158MEHmT9/Pr/73e9ISkpi5MiRLFmyhLi4uEbXc/r0aQICAmjTpg0AGRkZPProozz88MN4eHhwzz33cPr0ad58800Mw2DOnDn8/ve/569//aujj3379rF48WLOnz/PL37xC5544gmee+45AGbOnElZWRmLFy+msLCQBx98kJCQEG677TYAPvvsM+bOnUtMTAzh4eH1jmPNmjXMnDmTyspKvL29+eyzz/D19WXo0KHf+XchPy4ZGRmsWbMGV1dXDMOos6522dixYxk+fLiTKjQ/jbzF9J5//nliY2Mdj9rz3ykpKWzbto3S0lLgQninpKQA8Prrr5OamkpqaiqRkZHMmTOHkpISdu7c6ej3scceY9CgQfTr14/rr7+effv24e7ujre3NwBeXl6OIL4cu93OgQMHmD9/PhMmTHAsHzBgAD/72c+IjY3l2LFjrFu3jqeffprExER69erFM888wyeffEJeXp6jzT333ENQUBCdO3fm4YcfZunSpVgsFk6cOMHKlSt55pln6Nq1KwMHDuTOO+/k448/drSdOnUqSUlJdOrUqcHjGDVqFK6urmzevBmADRs2MGbMGNzd9R5fmmb48OEMHToUwzDqjbwNw2Do0KEK7u9If5VietOmTatzwZqXlxdwYZo6IiKCDRs2MGjQIPbs2cNLL70EQGZmJocPH2bJkiWOdpWVleTl5ZGYmAiAv7+/Y523t7djCruxpkyZgouLC4ZhYLfbmThxIrNmzXKsvzj49+/fj4+PT52RfHx8PF5eXhw6dIj+/fvX6z82NhabzcapU6fIysrCbrczevRox3qbzUa7du0crz08PC5br7e3N0lJSaxZs4YxY8awadMmnn766SYdswhceMM6btw4XF1d2bhxo+P0jZubG8OHD2fs2LHY7XZ98uI7UHiL6V3ugrWUlBTWrFlDRUUFPXv2JCwszLHuzjvv5NZbb63XV0t57rnniIuLw93dnfDw8MuO0j08POpdtGa32zEMg+rq6gbbWK1Wx3a1fXz66ad1tmnqhWkTJ05k9uzZfPXVV1RXVzNs2LAmtRcBcHFxwW63O0J606ZNAIwYMULB3UIU3vKDNnHiRNLS0jh//rxjyhwgOjqaw4cPExkZ6VhWXl6Or6/vFUfYjf1PJyIiotFXwcfExFBWVsbhw4eJjo4GYO/evVgsFmJjYx3b1QZ17fraNwZ2u53q6mpKS0uJj48HLlxVXhvwjT2OkSNHYrVaeeaZZ0hKSrriaF3kUmoDPDk52THyTk5OVnC3EJ3zFtMrKiri+PHjdR4lJSUAdO/enfbt27N582bGjx/vaDNjxgw2btzIiy++yPHjx1m1ahXDhw93XIl+Of7+/ri6urJ+/fpGbd8YkZGRjB8/npkzZ/LNN9/w9ddfM2vWLJKTk+nUqZNju+eee46srCx27tzJU089xQ033ICnpydRUVEkJSXx0EMPsWPHDo4cOcKvf/1r5s6d26Tj8PDwIDk5mR07dtR5syPSHLUBPmHCBCZMmKDgbkEKbzG9BQsWMGrUqDqPiz8iNnHiRHr27ElERIRjWUJCAvPmzWPp0qWMHTuWZ555hscff5wOHTpccX8eHh78/Oc/Z968ebz55pstdhxPPfUU0dHR3H777dxxxx0kJiby/PPP19mmd+/e3H///aSnpxMdHc2cOXMc65599ll69OjB9OnTSU1Nxd3dnQceeKDJxzFu3Dh8fHx0QZG0iNoAV3C3LJeS0jL7lTczl4unFmuf135cofbiIcMwqLHZqDEMbDab42GxWLBarfTr09sptf9YHDpynJgukVfeUIALn/OOj4/ngw8+oFevXld1Xy+//DKZmZm8+OKLV3U/Is2h/zsu0MhbRIALpx82btzIa6+9RlpamrPLEZHLUHiLCACrV6/m/vvv57bbbtONWURaOV1tLmICPj4+DX57WktKS0vTiFvEJDTyFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFqfw9fGmuKTU2WWIiIkUl5Ti6+Pt7DJaBYW3OEVwUADn8gsV4CLSKMUlpZzLLyQ4KMDZpbQKuj2qOIWnpwedOoZRUFjMufxCZ5cjIq2cr483nTqG4enp4exSWgWFtziNp6cH4WHtnV2GiIjpaNpcRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMkovEVERExG4S0iImIyCm8RERGTUXiLiIiYjMJbRETEZBTeIiIiJqPwFhERMRmFt4iIiMm4O7sAEWmYYRgcyT7KyVOnqbHZCA4Oplu3GLw8PZ1dmog4mcJbpBXatmMn7y3+iILCQtq29cXd3Z2SkvO0adOGcWOT+OnkFNzd9ecr8mOlv36RyygsN0h47Bx5z4Y2q/2B0zamzC9i92MhjW7zwUefsGz5SkaOGMaEcWNoH3KhbXlFBV98sZWPPllG1qHDPPTA/Xh4eDSrLhExN53zFlP519ZKss/VOLuMOtbtt/Cz14pbpK8tW7ezbPlK7p5+B9Om3kpu7gkenv0o9z8wk48/+ZSk0SP445yHOXnqFG+89XaL7FNEzEfhLaby9tZKss/ZmtTGsF+lYv6j/088eGRC2+/cj81m473FHzI2aRSDB/bn1OnT/GP+a8R2i2FSyng+2/wFK1atITysA3ffdQdbtm4n++ix734AImI6Cm9xml05VsY+X0DcH85y/9slVFRfSNlf/V8Jsz8sBWD/KRvdfn+Wk8U1DJ6bT+ZJK796p4SFX1Rcto+FX1Rww0uFjHmugAfeKQEg/tFz/HNTBQOfzKfn4+f48KsqRy1bDlcz9C/5xP3xLHcuLKa0qvGJ/1mWhcc+LnW8Xn/AwuC5+Vz3/87xxn/qbIz9B7MoOX+elAnJABw5cpQuUT/hzmlTGZ88hqFDBpG5/yAAPRN60OnajmzbvrPR/YvID4fCW5yizGLnjteL+e3Ytux6LASrAfPWlwPwx0l+fLyrisNnbfxpaSn3J/kSHuDGltnt6B7ehhdvvYY7h/hctg+AQ2dqePYWf55L83cs25NnZe1Dwfzpp348/smFwM0vM7hzYTEvp1/D14+FUG6xs+Czxofuxc5X2vnFv0qYk9KWDb8LblLbEyfyCGnXjoBrrgFg6JBBzHnkIVxcXCguLmHPvkziYrs6to/uEsWJvJPNqlNEzE3hLU6x6aCFa4PcSEnwxKuNC3cP82FtpgWA9v6u/DrJlzteL+ZEUQ33jvBpch8Aw7p6cN21bWjj5uJY9uskX9p6upDcw5OCMoNyix1fTxeW/jqIhIg2eLi7MDrWk8NnmzY1X+vzw9VEt3dnUqIXgT6u3DGk4dobYrXa8PBoU2/5+dJS/vTnp4gID2NSynjHcg8PD6xWa7PqFBFz09Xm4hSnSgz2HsV98gAABl9JREFU5lmJ+8NZAOyAv9e37yWnDvLmLyvKmJPStk74NqWPhrj8pyu3/2xWY4Cvpwt782z8bvF5yi12CssNBkQ17yru/FKDiEC3ZrUNCgwgP7+Ampoa3Ny+7WPr1h1UWaq49+67cHH59mdx+sxZggIDm7UvETE3hbc4Rai/K4OjPfi/uxsOn/mbyomPcOe1zytIH+yNd5v6AX6lPhprT56Vp1aU8fH9QXS4xpUFn1XwVU7zRrQh/q4UlhvNatujexyW6moyvtpN/359HMu7x3UjPDyszsfCSkrOk7l/Pz+7/dZm7UtEzE3T5uIUw2I82HPCxsq9FmwGbDxYzcsbL5yvPl5Qw6ubK3gl/Rr6RLbhb2u+PY/t4+HC8cIabDWX76Mp8kvt2O1g2O0cOWfj491V1DQvfxka7UHmSRtf51ox7PDpN1VXbvQfgYEBDBk8kH+/9wElJecdy7ftyOD9D5c4Xtvtdha+9Tb+/v4MHNCveYWKiKkpvMUpAnxcWXhnAC+sLSPuD2d5akUZw2Iu3PbzDx+V8rNBPnQMdGN2Slve2FLhOAd920BvnlpRxoLN5ZftoylGdPNgeFcPRj1TwD1vljAwygNbTfM+X+bn5cKLt/lz379KGPqXfE4VN+1dwO1TbsbTy5Mnn36OnNwTAPzPDdfz+B9nA1BaVsaL/5jPvsz9/OLnM3SXNZEfKZeS0rKr/CnY75/dbq/33DAMx792ux3DMKix2agxDGw2m+NhsViwWq3069PbKbVL6xX/6Dkstrp/Lk/e6MfNfb1bdD+lpWW89MoCDmYdontcLF2iOuPu3oaTp06x++s9+Hh788v77qZL1E9adL8iYh4Kb4W3tFK7v/6GbTsyLvpikiASE+IZOmRwg1eli8iPh+bcRFqp6xJ7cl1iT2eXISKtkM55i4iImIzCW0RExGQU3iIiIiaj8BYRETEZXbAmTrM718YLa8pYm2nhB/eRBxET2nhf428q9H3y9fEmOCgAT8/m3bb4h0jhLU6xO9fGpL8VYAdcXcCu9BZxupgukc4uoUHFJaXknDhFp45hCvD/UHiLU7ywpgw7kNbPi7/8jz8e7g1/+YiIXH0RM884u4TLCrjGD4CCwmLCw9o7uZrWQee8xSnWZlpwdUHBLSKNEnCNH+UVlc4uo9XQyFucws6FqXIFt4hI02nkLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8xTSioqKcXYKISKug8BYRETEZhbeIiIjJ6PaoIiLSZO+//z5Wq9Xx+uabb8bd3Z2qqio++ugjx3JPT09uvPFGAHbs2MHhw4cd6/r160d0dPT3V/QPiMJbRESazGKxUF1dXW+53W6nqqrh7wW32Wx11tXU1Fy1+n7oFN4iItIo58+fp6KiArgQ0hc7c+YMbm5uWCyWOssNw+D06dMAjra1SkpKHOvat2+Pq6vO5DaWwltERBplz549ZGVlNbhu9erVDS6vrq5mxYoVDa7bt28f+/btA+C2227D09OzZQr9EdDbHBERaXF+fn64u2t8eLUovEVEpMWNGDGCdu3aObuMHyyFt4iIiMloTkNERFqEm5tbvde1ywzDqHeRmzSfwltERFrE9ddf77jozNPTk5EjRzo+DrZ9+3ays7OdWd4PisJbRERaxMU3Z5k0aRI7d+50fBRMWpbOeYuIiJiMwltERMRkNG0uIiItbsWKFRiG4ewyfrAU3iIi0ijdunUjPDz8qvTdpk2bq9LvD5XCW0xDV6qKOFe7du1045VWQue8RURETEbhLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWp3ABXF2g2qZvGRIRaSqFtzjFmO6eGHaY9cF5BbiIXFFxSSm+Pt7OLqPV0E1axCkeGNuWtZkW3t1RxeKdVRjKbxG5hOKSUs7lF9KpY5izS2k1FN7iFNdd686y3wTzwpoy1mZanF2OiACHjhx3dgkN8vXxplPHMDw9PZxdSquh8Banue5ad964K8DZZYiImI7OeYuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8RURETEbhLSIiYjIKbxEREZNReIuIiJiMwltERMRkFN4iIiImo/AWERExGYW3iIiIySi8RURETEbhLSIiYjIKbxEREZP5/6CzcMYfmXHcAAAAAElFTkSuQmCC)

- **Set external\_id from:**

  Use this option to select how you want to set the `external_id`, either from the Event Property you specify or directly from the Value you provide.

- **external\_id**

  Depending on the previous selection, here you can specify the value or the client event property (e.g. client\_id) that corresponds to the `external_id` user identifier for Braze API.

##### Braze User Alias Object

This configuration section is enabled if you have selected the `user_alias` as the **Braze User Identifier**.

![braze user alias object](/assets/images/user_alias_object-b7392e429e51269631693e8083b3e47a.png)

###### Update existing users only

When enabled(default), this option will only update existing users. Uncheck this box to allow creating alias-only users.

###### User Alias Name

- **Set user alias name from:**

  Using this section you can select how you want to set the name of the user alias object: from an Event Property or directly from a Value you provide.

- **Alias Name**

  Depending on the previous selection, here you can specify the value or the client event property that corresponds to the User Alias Name.

###### User Alias Label

- **Set user alias label from:**

  Using this section you can select how you want to set the label of the user alias object: from an Event Property or directly from a Value you provide.

- **Alias Label**

  Depending on the previous selection, here you can specify the value or the client event property that corresponds to the User Alias Label.

## Snowplow Event Mapping Options

This section includes the mapping rules that concern a Snowplow event as claimed by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/):

### Snowplow Self Describing Event

![snowplow event mapping options](/assets/images/snowplow_event_mapping_options-cfd9b92901b525b1cee1dcb7a6a2b004.png)

#### Include Self Describing event

This option indicates if the Snowplow Self-Describing event data will be included in the event's properties object that will be sent to Braze. By default, this option is enabled.

#### Self Describing Event Location

This section is available only if the [Include Self Describing event](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/#snowplow-self-describing-event) option is enabled. Using this drop-down menu you can indicate the location where Snowplow Self Describing event properties should be added under Braze event properties. The available options are:

- **Nest under schema name** (default): The schema name will be used as a key in Braze event properties with the self-describing data as its value.
- **Merge to root level**: The self-describing properties will be added directly as Braze event properties without nesting.

### Snowplow Event Context Rules

This section describes how the Braze Tag will use the context Entities attached to a Snowplow Event.

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event object

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Braze's `event_object`.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in `event_object` or `user_attributes_object` of the Braze event (default value is `event_object`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event object` is set to `All`), you can specify the context entities you want to exclude from the Braze event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

## Additional Event Mapping Options

If you wish to pick other properties from the Client event and map them onto the Braze event, these can be specified in this section.

### Event Property Rules

#### Include common event properties

Enabled by default, this option sets whether to automatically include the event properties from the [Common Event definition](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) in the properties of the Braze event.

#### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the properties' object key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client. These keys will populate the Braze event's properties object.

#### Include common user properties

Enabled by default, this option sets whether to include the `user_data` properties from the common event definition in the Braze User Attributes object.

#### Additional User Property Mapping Rules

Using this table, you can additionally specify the Property Key from the Client Event, and then the User Attribute key you could like to map it to (or leave the mapped key blank to keep the same name). You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (note the array index 0) or pick non-Snowplow properties if using an alternative Client.

## Advanced Event Settings

![advanced event settings](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAbsAAADICAYAAABxqE/FAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15fJTl3e/xT/aNTEISAgmEhJAAgQREwm4IKCISBAtSooD6KPB4bM9TT33KgVotPdalnmp92VN7jsW6tVoqFhSFAGUTDFvYCZCQBUMSwpJlQrZJZjl/8GRqyBAWEwI33/frNS+d+7rnun73PTDfue5lcKs0VzvsdjsAdrsdh8OB3W7HZrVis9uxWq3Oh8VioampieHD7kRERORW4d7ZBYiIiHQ0hZ2IiBiewk5ERAxPYSciIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ5nZxdwq1mzZg1btmzh4YcfZujQoW2uu3v3blasWMGCBQuIj4+/QRV+f1eqe/HixTT/xNx33X///UyYMOFGlNimnTt38s0331BRUUGXLl0YNmwY99xzDx4eHu02xocffkhNTQ1PP/10u/UpIh1HYXeNDh06BMCBAweuGHZGFhoaSnJycotlsbGxHT7u4sWLSUhI4LHHHnPZ/vXXX/Pll18SGxtLUlISpaWl/POf/8RsNjNr1qzrGuPkyZO8/fbbPPDAA6SkpABgMpnw9NRfH5FbxS33t/XkyZOcOHGCe++912X7hg0biI+PJyYmpt3HLioqoqKigrCwMHJzc6mvr8fPz6/dx7kVdO3alXvuuaezy2hl9+7dBAUFsXDhQtzdLx6lX7ZsGVlZWaSlpeHv798u4zz44IPt0o+I3Bi3XNht3LiRnJwcbDYbkydPbtG2du1aNm/ezKlTp3jiiSfafeyDBw/i4eHB9OnTeffddzly5AjDhw93ttfX1/Ppp5+Sk5NDaGgoUVFRzrbt27fzxRdfsHDhQuLi4gD4zW9+g5eXFz/96U+pqalh9erVHD9+HIB+/frxgx/8AH9/f8rKynjjjTdITU3l1KlTFBUV0aNHD2bPnk337t0BqKqqYtWqVeTl5eHr68vQoUOZPHmy89Dd0aNHycjIoKKigu7du5OWluacibVV97X64IMPyMnJ4Ve/+hVeXl6YzWZeeuklxo4dy/Tp02loaOCLL77gyJEjeHt7k5iYSFpaGl5eXmRmZrJq1SqmTZvGjh07MJvNxMfHk56eTnl5OW+++SYA2dnZLF68mFdffbXV+FarlcbGRhoaGpzB9uCDD1JZWemciV2uhnPnzrUaIy0tjdWrVwOwevVqjh07xsKFC3nrrbeora1lyZIlACxdupSYmBj8/f05dOgQ/v7+TJs2jcTERAAaGxtZsWIFR48eJTg4mDvuuIP169czb948kpKSqKqq4vPPPyc/Px9PT0+SkpKYOnUqXl5e1/1eiMi/3HIXqMybN4+YmBg2bdpERkaGc3lz0MXExDB37tx2H9fhcHDo0CHi4uLo168fJpOJAwcOtFjnq6++4siRIwwaNIgBAwY4D3kCzg+9nJwcAMrLyykvLycpKQmAv/71r2RnZ3PPPfcwfvx4Dh8+zOeff96i/23bttGjRw9GjRpFSUkJX331lbO2Dz/8kPz8fFJTUxk4cCBbt27ln//8JwAFBQV88MEHBAYGMmnSJOx2O3/+85+prq6+Yt2XY7VaqaysdD6a+0pKSsJqtZKXl9die5u3/8MPP+Tw4cOkpKRwxx13sHPnzhbvI8CmTZsYPHgwcXFxZGdns3PnTkJCQnjsscdwc3MjKiqKRx991GVdY8eOpb6+njfeeIPNmzdjNpsJCwsjPj4eb2/vNmtwNcbAgQO5//77ARg+fDj33XffZffJ0aNHqaurIzU1lcbGRj799FNsNhtw8VzvgQMH6NevH4mJiWzfvr3Fa//2t79x8uRJpk6dysiRI9mxY0er/SIi1++Wm9l5e3szf/58li1bxqZNm4CL/w7fli1biImJYf78+c4PtfZUWFiI2Wxm4sSJuLm5MXDgQHbv3k1tbS0BAQFYrVb27t1LbGwsjzzyCABdunThyy+/BCA4OJioqChycnJIS0trFQLjxo3D39+f6OhoAHJzcykoKGhRw9ixY3nggQeAi4dUi4uLgYuHdouLi1ucU7pw4QKFhYUAbN26lYCAAObMmYOnpyexsbG89dZbHD58mJEjR7ZZ9+WcPHmSV155xfm8e/fuPPvssyQkJODh4UFOTg4JCQnk5OQQEBBAnz59KC4uJi8vj7S0NEaPHg1ARUUFe/fudW4XwEMPPcTAgQNpamriF7/4BcXFxaSmpjJo0CDc3NwwmUwMHDjQZV0pKSl4e3uzbt061q5dS0ZGBnfeeSfTp0/H19f3ijW4GqNPnz4A9OjRw/n+uNKzZ09nWFqtVjZv3kx5eTlhYWHs3r2b6OhoZ0h369aN5cuXO19bWlpKVFQUw4cPd47v4+PT5nsgIlfvlgs7cB14HRl0cPEQJlz8UC8vL6dXr17s3LmTQ4cOMXr0aCorK7HZbM4PRgBfX98WfSQmJrJ27VrMZrPzkGFERARw8YN0/fr1fPLJJ9TW1tLU1NRqW757vsnX15fGxkYAzp49C0CvXr2c7d+9gKO0tJSamhqWLl3aor/mWdmV6nYlIiKCSZMmOZ83fzD7+fnRt29fcnNzsdvt5OXlkZSUhLu7O6WlpcDFmWTzrLSZxWJptZ1eXl54eHg4t/NqjRw5kuTkZI4ePUpmZiZ79+7FbDazcOHCK9bwfQLGz88PNzc34F/7sLGxkaqqKqxWK3379nWue+l7O3bsWDZu3Mhrr71GQkICQ4YMaTNYReTa3JJhBy0DD+jQoLPb7Rw+fBiAt99+u0XbwYMHGT16tPNiiLYub09KSmLt2rUcPXqU/Px858wC4L333qO+vp4ZM2YQFhbGihUrOH369FXV1/wB25bQ0FBmz57dYpnJZMLhcFyxblcCAgIYNGiQy7akpCQ+++wz9u/fT319vXP22mzy5Mmtrtxsj3NTtbW15OTkEBERQUREBElJSSQlJfHuu++Sk5OD2Wzu8BpcaX5/2nqf7rvvPvr378+BAwfIzs5m+/btpKWlkZqa2iE1idxubtmwg38FXvP/d5T8/HxqamoYPXq08+ISgD179pCTk0N1dTXBwcF4enq2CCir1dqin7CwMHr06MGmTZtobGx0nq+rqamhrKyMMWPGkJCQAOA813M1wsPDASgpKXHO0DIyMqitrWXmzJl0796d/Px8goODCQ4OBuDUqVOEhIRgs9muWPe1SkxMZOXKlWRkZODr6+u8V69Hjx7AxYtpmq+Wrauro76+3vll4Wo0B/SlrFYry5cvp1+/fjz55JPO5V26dAEuhs3V1uBqjMuNeyVBQUF4eXk5DytDy5ms2Wxm8+bNJCUl8eCDDzJt2jRef/11srKyFHYi7eSWDjvo2JBr1nwhyvjx4+natatzuZubG8ePH+fgwYOkpKSQnJzMrl272LBhA4GBgc4LRL4rKSmJDRs2YDKZnFc9+vv7ExAQwIEDBwgODqakpISioqKr3rbo6GiioqJYv3499fX11NXVkZmZ6fygvPvuu8nNzeWdd95hxIgRlJeXs3v3bp566in69OlzVXVfqqqqis2bN7dYFhMTQ58+fQgICCAmJoaCggLuuOMO56yxd+/exMXFsWvXLux2O926dWtxq8DVMJlMnDx5ko0bNzJhwoQWARUUFERycjJ79uxh2bJl9OnTh4qKCvbt20efPn0wmUyYTKYr1nDpGCaTCbj458Df37/V/YVX4u7uzvDhw8nMzOQvf/kLYWFh7Ny509keEBDAkSNHOHbsGBMnTqSpqQmz2XzZmbOIXLtb7mrMG81ms3HkyBF69OjRIujg4u0Bnp6ezvN5U6ZMYdCgQWzevJlt27YxePDgVv01z+YSExOdh7Xc3d159NFHMZlMbNy4EavVSmJiIo2NjdTW1l6xRjc3N+bNm0dsbCxbtmzhwIEDjBs3znnlYGxsLPPmzcPDw4N169aRm5tLWlqacxZ4NXVf6vz586xdu7bFIzc3t9V2Nv+32dy5cxk6dCiHDx9mw4YNhIaGMnPmzCuO16z5dpPt27e7nGnNnDmTtLQ0qqqq2LhxI7m5uYwYMaLF1ZtXquHSMUJCQhgzZgxnz57lyJEjV13rd02ZMoU77riDY8eOcfjwYeehXTc3Nzw9PVmwYAGhoaGsWrWK9evXk5iYyPTp069rLBFpza3SXO1o/uknu92Ow+HAbrdjs1qx2e1YrVbnw2Kx0NTUxPBhd3Zy2SK3lua/O82HVDds2MCGDRt45plniIyM7OTqRIzvlj+MKXIrWLduHdnZ2YwYMYKmpia2b99Or169nOcQRaRjKexEboBx48Zx4cIFtm7dioeHBwkJCUydOvWaLswRkeunw5giImJ4+lopIiKGp7ATERHDU9iJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihufZ2QWI3Kyqq6vJyc3DbK7G18+XPjHR9IyM6OyyROQ6KOxELlF94QKfLF/Bzl178PDwwGQKpK6unoaGBvrERDNvTjqxfWI6u0wRuQYKO7nlZWZmUlhYyJw5c753X2fPneM3//tNPDw8ePqp+QwdMhhPT08cDgeFhSf5x+erefk3r/PfFj7BsDuHtkP1InIjKOxcsNvtLF68uNXy9PR07rzzznYd69y5cxQUFDBy5MjL1jFt2jTuuusu5/JVq1bh6enJ1KlT27WWq1VUVMTq1aspLS0lKCiI8ePHM2LEiA4dc82aNfj7+zN+/PgOG8NqtfLm7/+IyRTIf/70P3DDjT+9+wGHjhwhvFs3Hpv7MM8+89/5698+5f8te48XnutOr56RHVaPiLQfhV0bnnvuOYKCgjp0jPPnz7Nr1y6XYdds/fr1DB48GJPJ1KG1XI2KigqWLVvG5MmTmTdvHqdPn+bvf/87vr6+DB48+Hv373A4cHNza/V81KhReHh4fO/+27J56zbKyyt49ddLCfD3551l75F7Io8fzvwB+w8e4ne/f5vXXn6ROemzKCw8yacrVvI/fvKjDq1JRNqHwu4affzxx/Tq1Ytx48YB8NFHHxEbG8vYsWMpKiris88+o7KykoSEBGbOnIm3tzeZmZnk5+djt9vJy8ujd+/ezJs3j/3795ORkYHFYuGVV15hyZIlLseMioriiy++YO7cuS7b8/Pz+eyzz6itraVPnz6kp6fj6+vLtm3bOHHiBDabjW+//Zbo6GgmT57Mp59+SkVFBQMHDiQ9PR13d3fsdjtfffUVe/bsoUuXLjzwwAMkJCS0GiszM5OBAwcyZswYAEwmE/fffz9ff/01gwcPvu79c+DAASwWCxEREYwfP5533nmH3r17c/r0aZYsWcLXX39NQEAA9957LwAbN25k+/bt+Pv707NnzxY1ZmVlsW7dOux2O2PGjOGee+65qvd25649jBk9kq5dgwEoKT3Nw7MfYsTwYQxPHsaPn/lPThUX0y8+jrT77+P//PEdamtrCQgIuKr+RaTz6NaDa5SUlMTRo0cBsNlsnDhxgsTERCwWC++//z733nsvzz//PDabjc2bNztfl5OTQ2pqKj//+c9paGhg//79jB49mvT0dCIjIy8bdABTpkwhPz+fnJycVm01NTW8//77zJkzh+eff57Gxka2bdvmbD916hRTpkxh8eLF1NTU8Ne//pXHHnuMRYsWcerUKY4fPw7A119/TUlJCUuWLGH27NksX76c+vr6VuOVlJQQGxvbYllMTAwlJSXfa/+cPXuWhx56iFmzZgFQW1tLUlISzz77bKsa8vLy+Oabb1i4cCELFiygvLzc2VZUVMSaNWtYsGABP/nJT9i7dy95eXmX3bffVVxSSnxcX+fzX73wc0YMHwbAtu2ZBHbpQq//Cta4uFjsdjuny85cVd8i0rkUdm146aWXWLRoEYsWLeJ3v/sdAAMGDKC0tJS6ujoKCwsJDw8nKCiI3NxcQkJCSExMxMvLi5SUFI4dO+bsKyEhgZiYGPz8/IiNjW3xAX0l/v7+TJ06lZUrV2K1Wlu0+fj48OMf/5iePXvi6elJ//79OXfunLM9Li6Onj170qVLF+Lj4xk4cCChoaGYTCaio6Oddezdu5cJEybg5+dHdHQ0PXv2pKCgoFUt9fX1rWYyAQEB2Gw2mpqarnv/xMfHExUV5TxUGRAQwLBhw/D29m5VQ3Z2NsnJyURERBAcHMywYcOcbfv27SM5OZnw8HBMJhPJycktxmmL1dqEt5dXq+VfrsngH5+v5j9+/BT+/n4AzroaGxuvqm8R6Vw6jNkGV+fsvLy8iI+P5/jx4xQXFzvPU5nNZkpKSnjhhRec6/r6+rrs183NDZvNdk21DBs2jKysLDZu3NiqntLSUlasWEFjY6PzUKYr7u4tv9u4ublht9uBi/eU/eUvf3GeL7Pb7S7Pwfn6+lJbW9tiWW1tLR4eHnh6euLm5va998+V1NTUtJpdNjObzeTm5rJz507g4jm/QYMGXVW/wcHBlJ1pPVP7au16pj8wpcWs78x/zehCQkKutXwR6QQKu+swePBgDh06xOnTp1mwYAEAgYGB9O3bl/nz53fYuDNmzOCtt94iKiqKyMiLVwGWlJSQkZHB008/TVBQENu2baOoqOia+w4MDGTWrFlER0e3uV5UVBT5+fktrr48efIkPXv2dAZlR++fwMDAVoHbzGQyMXHiRCZMmHDN/Q5OHMSOnbtJu/8+57Y4HA4en/cI/fvFt1j3m527CAsLpUf38GvfABG54XQY8zokJCRw4sQJfH196dq1K3DxMFxJSQnZ2dnY7XZyc3PZunXrFfvy9vampqbG5fmxS3Xr1o2UlJQW56BqampwOBw4HA7OnTvHwYMHnbO1azFkyBDWr19PXV0ddXV1rF69GrPZ3Gq90aNHc+zYMXbs2EFNTQ15eXlkZGSQkpLiXKc9948rgwYNYt++fdTV1WG1Wp3nCOFi0GZmZnLmzBmamprYvn07ubm5V9Xv5EkTKTtzltVfZTiX2R0O/vjOu5z8zheIvPwCNm3+mqlTJl9X/SJy42lm14aXXnqpxfMZM2YwatQovL29iYuLIyoqytnm7+/P448/zqpVq/jkk08IDw9n5syZVxwjOjqawMBAXn75ZV588cUrrn/33Xdz4MAB5/N+/frRr18/Xn/9dbp27Ur//v05f/78NWzlRRMmTGDNmjX89re/xW63M3z4cJe3OoSEhPDkk0+yevVqVq9eTVBQEJMmTWLIkCHOddpz/7jSt29fhg0bxhtvvIGfnx/du3dv0TZx4kTee+89amtriYuLu+p7I7t3D2fenHQ++Ohjmpoamf5AGp6enry/7I/Odfbu28+y9z7kjiFJpKaMva76ReTGc6s0VzuaZwJ2ux2Hw4HdbsdmtWKz27Farc6HxWKhqamJ4cPa98ZqkZvJ9m928NHHy/Hz82XokMGEhIRQV1/H0aPHKTpVzPhxdzH3kdkdft+fiLQfhZ2IC2ZzNVu+3sax47lUV1fj6+dHn5je3DVmNH1i2j6vKSI3H4WdiIgYni5QERERw1PYiYiI4SnsRETE8BR2IiJieLrP7jIslkbKK6qorbvyzd4icvsK8PcjNCQYH5/Wv+MqNw+FnQsWSyNFxafpFhZCZIR+Dkqks53I/5b4vjfnLR9V5gsUFZ+md68IBd5NTIcxXSivqKJbWAjBQYGdXYqI3OSCgwLpFhZCeUVVZ5cibVDYuVBbV6+gE5GrFhwUqFMeNzmFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKGp3/1QEQMZ8WKFTQ1NTmfz5o1C09PTxoaGli5cqVzuY+PDzNmzABgz5495OXlOduGDx9OXFzcjStaOpTCTkQMx2Kx0NjY2Gq5w+GgoaHB5WusVmuLNpvN1mH1yY2nsBMRQ6iurqaurg64GGrfdebMGTw8PLBYLC2W2+12ysrKAJyvbWY2m51t4eHhuLvrrM+tTGEnIoZw+PBhcnNzXbatX7/e5fLGxkbWrl3rsi07O5vs7GwAHnnkEXx8fNqnUOkU+qoiIredwMBAPD31Xf92orATkdtOamoqYWFhnV2G3EAKOxERMTzN40XktuDh4dHqefMyu93e6qIWMRaFnYjcFqZNm+a8yMTHx4fx48c7by/YvXs3BQUFnVmedDCFnYjcFr57M/nUqVPJyspy3logxqdzdiIiYngKOxERMTwdxhSR287atWux2+2dXYbcQAo7ETGE/v37ExkZ2SF9e3l5dUi/cuMo7ETEEMLCwnSjuFyWztmJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIoansHMhwN+PKvOFzi5DRG4RVeYLBPj7dXYZ0gaFnQuhIcGcO1+hwBORK6oyX+Dc+QpCQ4I7uxRpg35BxQUfH29694qgvKKKc+crOrscEQFO5H/b2SW4FODvR+9eEfj4eHd2KdIGhd1l+Ph4ExkR3tlliIhIO9BhTBERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKGp7ATERHDU9iJiIjhKexERMTwFHYiImJ4CjsRETE8hZ2IiBiewk5ERAxPYSciIobn2dkFiHSW0tNlFJ48SX1dAyZTIP36xREcFNTZZYlIB1DYyW3n22+L+Ojj5eTlF+Dr44N/gD/V1Rew2WyMSB7Gw+kPKfREDMat0lztsNvtANjtdhwOB3a7HZvVis1ux2q1Oh8Wi4WmpiaGD7uzk8sWuT4HDh7iD/93GfF9Y5nxg2n0je2Dm5sbVquVg4eO8Olnq7BYLPzP/3yGHj26d3a5ItJOFHYu2O12Fi9e3Gp5eno6d97Zvtt+7tw5CgoKGDlypMv2Xbt2ERsbS7du3QD4wx/+QFpaGjExMe1ax83s0n1wvU6XlbH0xVcZM2oEj859mNLTZfz5/Y8oKT3NgP79ePzRR/Dx9uH1N3/PhZoaXvzlL/D29mqnrRCRzqQLVNrw3HPP8dprrzkf7R10AOfPn2fXrl2Xbd+1axfnz593Pp8+fTo9e/Zs9zquhcPhwOFw3LAxLt0H12vFPz6nR4/uzJuTjs1m483fv42HhwfpP5zJ6bIylv35A/z8fPmPH/071dXVbNy05XuPKSI3B52zu0Yff/wxvXr1Yty4cQB89NFHxMbGMnbsWIqKivjss8+orKwkISGBmTNn4u3tTWZmJvn5+djtdvLy8ujduzfz5s1j//79ZGRkYLFYeOWVV1iyZEmLsV599VWqqqr429/+xn333ceYMWNYvnw5Dz74IH379uWDDz7AZDJRUFBAZWUlo0aNIjo6ms8//xyr1crdd9/trLOmpoa///3vFBYWEhERwezZswkNDW21fUuXLmXIkCGcOHECm83GtGnTGDRokLNtwIABZGdn88wzzxAcHMyaNWvIysrC19eXlJQU7rrrLgAyMzM5cuQIVquViooKoqOjmT17Nt7e3gBkZWWxbt067HY7Y8aM4Z577nE5xp/+9KcW+8BisVBYWMgTTzwBQEZGBtXV1fzwhz9s831raGjgwMHDLJz/OO7u7pSdOYsbbvz7/H8jNDSELgEB/PGdd3E4HJhMJlLGjmHn7j3cP/ne6/2jIiI3Ec3srlFSUhJHjx4FwGazceLECRITE7FYLLz//vvce++9PP/889hsNjZv3ux8XU5ODqmpqfz85z+noaGB/fv3M3r0aNLT04mMjGwVdACLFy8mMjKS9PR0xowZ47KesrIyFixYwI9+9CN27NjB3r17efbZZ3niiSfIyMigrq4OgOXLlxMeHs4vf/lLkpKSWL58+WW3sWvXrvzsZz9j9uzZLF++nPr6emebn58fS5YsISQkhM2bN3Pq1CkWLVrE/Pnz2bZtG8eOHXOuW15ezrx581iyZAkWi8W5P4qKilizZg0LFizgJz/5CXv37iUvL8/lGJfug6SkJPLy8mhsbATg+PHjzjBuy+myM9hsNuLj+gIQGdGD1175X4SGhmC1WtmxazcD+sfj5uYGQFzfWEpKT3f4DFZEbgyFXRteeuklFi1axKJFi/jd734HwIABAygtLaWuro7CwkLCw8MJCgoiNzeXkJAQEhMT8fLyIiUlpcUHf0JCAjExMfj5+REbG0t5eXm71Dh06FBMJhMRERF0796d5ORk/Pz86N27NwEBAVRWVlJbW8uJEyeYNGkSnp6ejB07luLiYhoaGlz2OXDgQNzc3IiNjaV79+4UFBQ420aPHo2/vz9ubm7s27ePiRMnEhAQQLdu3UhJSSErK8u5bu/evQkMDMTDw4Nx48Y598e+fftITk4mPDwck8lEcnJyi3313TEuFRYWRmhoKDk5OVRXV3P+/Hn69et3xf3U1NQEgLeXd6u2N3//NqdOlfDUgiedy7x9vLHZbNgVdiKGoMOYbXjuuecIuuQSdC8vL+Lj4zl+/DjFxcUMHjwYALPZTElJCS+88IJzXV9fX5f9urm5YbPZ2r1ed3d3PDw8Wjx3OBxUV1fjcDj49a9/3WL9CxcuXLbGZl26dKGmpsZlW01NDV27dnU+79q1KwcOHHC5bmBgoLMfs9lMbm4uO3fuBC6en7ua2VmzpKQksrOzaWhoIC4uDi+vK19EEvJfdZadOUNcl1jn8qJTxRzJPsYvlvyMwMAuzuVlZWcIMpnwcNf3QREjUNhdh8GDB3Po0CFOnz7NggULgIsf5n379mX+/PmdXF1rgYGBeHl5sXTpUtyv8cO7qqqKLl26uGwLCgqiqqrKeZVkZWXlZdf9bpvJZGLixIlMmDDhmmpplpiYyDvvvIPFYrnqkAwLCyWiR3cyd+wiru+/wi44yMR/W/gkffrEOJc5HA527NxN4qCE66pPRG4++tp6ssdB5QAABd1JREFUHRISEjhx4gS+vr7OmU18fDwlJSVkZ2djt9vJzc1l69atV+zL29ubmpqaFufFLm0vLy//XjPBLl26EBUVRUZGBjabjbNnz7Jy5crLno/at28fVquVI0eOcP78eWJjY12ud+edd7Jx40bq6+uprKzkm2++YejQoc72wsJCzp49S0NDA1u2bCEh4WJ4DB48mMzMTM6cOUNTUxPbt28nNzf3svVfug8iIyPx9fXl2LFjzj6vRtqUyWzdtp3jOf8a61RxKX98513nYU6AjPX/5FRxCVMmT7rqvkXk5qaZXRteeumlFs9nzJjBqFGj8Pb2Ji4ujqioKGebv78/jz/+OKtWreKTTz4hPDycmTNnXnGM6OhoAgMDefnll3nxxRdbtY8YMYKVK1dis9lITU297m1JT0/nH//4B7/61a/w8/Nj8uTJLs+JwcXDk6+88srFy/LT0/Hz83O53rhx46itreW3v/0tDoeDcePGMWTIEGd7165dWbFiBWVlZcTHxztncn379mXixIm899571NbWEhcX1+ZtHa72QWJiIkVFRZedSbpy15hRHDlylDffept/e3wuI4cnM2jgAN5f9kcArFYrX65Zx+erv+KR9FlERkZcdd8icnPTTeXSwtKlS3n66acJDw//Xv1kZmZSWFjInDlz2qmyljZs2IC3t/c1fwGw2e18snwFGzdtIapXTwYNTMDf35/KykoOHDpMTU0Nj6T/kPHj7uqQukWkc2hmJ7cUu91OZWUlWVlZPPXUU9f8eg93d+Y+/EPG3TWGr7dnknsij/qGBgIDu5Aydgzjx91F167BHVC5iHQmhZ3cUoqKili2bBmTJk1qcSXoteod1Yu5D7d9I7qIGIcOY4qIiOHpakwRETE8hZ2IiBiewk5ERAxPF6hchsXSSHlFFbV1rm/2FhEBCPD3IzQkGB+f1r+7KjcPhZ0LFksjRcWn6RYWQmTE97vfTES+vxP53xLfN7qzy3CpynyBouLT9O4VocC7iekwpgvlFVV0CwshOCiws0sRkZtccFAg3cJCKK+o6uxSpA0KOxdq6+oVdCJy1YKDAnXK4yansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8PSvHoiI4axYsYKmpibn81mzZuHp6UlDQwMrV650Lvfx8WHGjBkA7Nmzh7y8PGfb8OHDiYuLu3FFS4dS2ImI4VgsFhobG1stdzgcNDQ0uHyN1Wpt0Waz2TqsPrnxFHYiYgjV1dXU1dUBF0Ptu86cOYOHhwcWi6XFcrvdTllZGYDztc3MZrOzLTw8HHd3nfW5lSnsRMQQDh8+TG5ursu29evXu1ze2NjI2rVrXbZlZ2eTnZ0NwCOPPIKPj0/7FCqdQl9VROS2ExgYiKenvuvfThR2InLbSU1NJSwsrLPLkBtIYSciIoanebyI3BY8PDxaPW9eZrfbW13UIsaisBOR28K0adOcF5n4+Pgwfvx45+0Fu3fvpqCgoDPLkw6msBOR28J3byafOnUqWVlZzlsLxPh0zk5ERAxPYSciIoanw5gicttZu3Ytdru9s8uQG0hhJyKG0L9/fyIjIzukby8vrw7pV24chZ2IGEJYWJhuFJfL0jk7ERExPIWdiIgYnsJOREQMT2EnIiKGp7ATERHDU9iJiIjhKexERMTwFHYuBPj7UWW+0NlliMgtosp8gQB/v84uQ9qgsHMhNCSYc+crFHgickVV5gucO19BaEhwZ5cibdAvqLjg4+NN714RlFdUce58RWeXIyLAifxvO7sElwL8/ejdKwIfH+/OLkXaoLC7DB8fbyIjwju7DBERaQc6jCkiIoansBMREcNT2ImIiOEp7ERExPAUdiIiYngKOxERMTyFnYiIGJ7CTkREDE9hJyIihqewExERw1PYiYiI4SnsRETE8BR2IiJieAo7ERExPIWdiIgYnsJOREQMT2EnIiKG9/8BVCP5Oaebrs0AAAAASUVORK5CYII=)

This section offers additional configuration options on the Braze event object:

### Event Name Override

You can use this option to override the name of the Braze event object or leave it blank to inherit from common event properties, which is the default. Please note that the `name` property of the [Braze event object](https://www.braze.com/docs/api/objects_filters/event_object/#event-object), that this option populates, is a required property according to Braze API.

### Event time property

This option enables you to specify the client event property to populate the event time (in ISO-8601 format) or leave it empty to use the current time (default behavior).

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Braze Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Braze GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "Braze",    // the name of the tag
    "Type": "Message",  // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",   // the "trace-id" header if exists
    "EventName": "xxx"  // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# Braze Tag for GTM Server Side
> Forward Snowplow events to Braze from GTM Server Side using the Braze Tag with Track Users API for personalization and marketing automation.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/

The Braze Tag for GTM SS allows events to be forwarded to [Braze](https://www.braze.com/). This Tag works best with the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/), but can also work with other GTM SS Clients such as GAv4.

## Template installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click "Search Gallery" in the Tag Templates section
2. Search for "Braze" and select the official "By Snowplow" tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking "Add"

### Manual Installation

1. Download [the template file](https://github.com/snowplow/snowplow-gtm-server-side-braze-tag/blob/main/template.tpl) `template.tpl` – Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As…"
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## Braze Tag Setup

With the template installed, you can now add the Braze Tag to your GTM SS Container.

1. From the Tag tab, select “New”, then select the Braze Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Braze.
3. Enter the required parameters and then optionally
4. [Configure the tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/braze-tag-configuration/) to customize your Braze Tag.
5. Click Save

---

# Configure HTTP Request Tag for GTM Server Side
> Configure JSON request body construction, entity mapping, custom headers, and post-processing for the HTTP Request Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/http-request-tag-configuration/

In the following short video a complete example configuration of the Snowplow GTM SS HTTP Request Tag is presented.



Scenario: The example assumes that we want to send a POST HTTP Request to an example custom destination endpoint, where we would like the body of the request to have the following structure:

```json
{
  "api-key": "myApiKey",
  "user_identifier": ...
  "event_data": {
    ...
  },
  "user_data": {
    ...
  }
}
```

where, for this example:

- Our endpoint expects the `api-key` inside the request body.

- As our `user_identifier` we want to map the value of the `client_id` from the client event.

- Inside `event_data` we want to include:

  - the common event data
  - the Self-Describing event data
  - the performance timing data from the Snowplow [Performance Timing Context](https://github.com/snowplow/iglu-central/blob/master/schemas/org.w3/PerformanceTiming/jsonschema/1-0-0)), with `performance_timing` as the property name
  - the page view id from the Snowplow [web\_page context](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0), with `page_view_id` as the property name.

- Under `user_data` we want to map the `user_data` from the client event.

You can read on below for more details on each configuration option.

## Configuration options

### Destination URL (required)

Set this to the URL of your custom HTTP endpoint.

### Wrap the request body inside an array

By default, the JSON body is an object. For example:

```json
{ "myProperty": "myValue" }
```

This option allows you to wrap the resulting object of the request body inside an array:

```json
[{ "myProperty": "myValue" }]
```

### Include all event data in the request body

This option allows you to relay the full client event into the body of the request. Enabling this option, consequently disables both the Snowplow and the Additional Event Mapping Options, which allow to cherry-pick event properties and customize the request body.

### Use alternative separator to dot notation

Enable this option to use an alternative separator to dot notation when specifying possibly nested object paths. This setting **applies everywhere dot notation can be used** and it is useful when you want to allow dots or special characters in key names. Enabling this option reveals the text-box where you can specify the character you wish to use to denote nested paths.

#### Example

Let's imagine this property name: `user_data.address.city`

This is by default being interpreted as a nested path, where the dot character denotes a change in nesting level:

```json
  "user_data": {
      "address": {
          "city": "Foobar City"
      }
  }
```

If you wish it to denote a different nesting path where a key name may include a dot, for example:

```json
  "user_data.address": {
      "city": "Foobar City"
  }
```

then you can use this setting to set a different separator for nesting. As an example, if you set:

![first alternative separator example](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZwAAAByCAYAAABneGd/AAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABpySURBVHic7d15WFNn2gbwmzUSFlGhClRpwqJUFAvFKqIWRcUNW5e6daHiYGtbdUangFu134BLhbbTOlpq3abuiK2Oa91aa11QBwVEEawgagSBsIQlJHm+PxxOjUQIVRLE53dduTTvOec9T96ccJ+8ORATeWkZ4X+I7v9Xo9EI/xIRNBoN1CoV1BoNVCqVcKuurkZNTQ38/XzBGGOM1cfU2AUwxhh7NnDgMMYYMwgOHMYYYwbBgcMYY8wgOHAYY4wZhLmxC2isbclViNpZCqWKGl5ZB0tzEywdY4fx/q2ecGWMMcbq89S9w3mcsAEApYoQtbP0CVbEGGNMH09d4DxO2DzJPhhjjDXOUxc4jDHGnk7PROAEupkYuwTGGHvmtfjACXQzwbQ+ZrgQafGn+6ioqIBUKsXp06e12rOzsyGVSpGdnf24Zert8uXLkEqlKC4uNtg+GyM7OxsBAQG4e/eusUthj9C3b19s27bN2GU8EcY+3m7fkeHkqdM4fOQ4ziafh7ykxCh1PC1adODUhk1XJxPsTtXAya7lvdPZs2cP/Pz8ms3+O3TogPDwcLRt29ZoNRnT/PnzMX369CfWX0FBAaRSKTIzM59Yn43R1MdXY8eruRxvOTm5+MeSzzB3wWL8+/ut2HfwEBK+W4+/zonGqm++4+B5hKfusmh9PRw23/yqwZ1SvljgQWq1GmZmZk+0T2tra4SHhz/RPpuLphivB2k0GpiatuhzwCfOGMdbysVLWLl6DTzcpJgf/Xe4SSUwMTGBSqXCxUtp2LHzByz+v6WInDMLHTq0N2htzd1Tf3S7OdR912KssLl8+TLGjRuHrl27IigoCImJiVrL165di4CAAPj6+mLWrFmQy+U6+ykvL8fcuXPh6+uLHj16YObMmSgtrXsp97Rp0zBz5kwUFxdDKpXi6NGjAIDc3Fy8/fbb8Pb2xuDBg3HgwAFhm9DQUMyZMwdDhw7F+PHjhenCbdu2YejQoejatSveeecdFBUVCdscO3YMI0aMQJcuXRAcHCz0p2v/D075JSUloWfPnsJfHweA6dOnY968eQCAyspKzJ07Fz4+PujTpw++/PJLqNVqnWPyn//8B8HBwejatStGjx6NlJQUYVl9/cTGxiIsLAwffvghevTogYCAACQlJek11rVjExMTg4CAAMTFxdW7vo+PDzZv3owDBw5AKpWioqICAHD8+HEMGzYMXbp0QUhICI4fPy7sPzY2FuPHj8eUKVPg5eWF6upqYdm6devwyiuvAABCQkIQGxsr1BUdHY0ePXrAx8cHkZGRwr502bBhA/z9/dGzZ08sW7aszhg/qr5HHV8Pio2NxXvvvYdFixahe/fu6N27t9ZxX1+tjxqvP3u81UpISECfPn3g5eWFt956C7///ruwLDQ0FHFxcZgyZQpefPFFhISE4NKlS48cu4fdkcmwKmEtAgN64e+zZ8LKygr/WPIZ3vvwr/h61bdwc5Pgk/lRaNeuLT7/6l9QKmv07vuZIC8to9pbcUkpFZeUUmGxnAqL5VRQWET59wpJll9At27fody8W3T9Rg5lZmXT5StX6b8XL9HZc+fJkJxny4SbLL+Qvj9RQH2X3BXa3vhXPh25eI9k+YWUcKSA/Bbf1dqm9tYYCoWCJBIJnTp1Sqs9KyuLJBIJZWVlERFRQEAAffrpp5Sbm0vbt28nDw8PSklJISKijRs3Ut++fen06dOUmZlJEydOpBkzZujc38yZM2nMmDF09epVyszMpJEjR9L8+fOJiCg9PZ0kEgkVFRVRVVUVJSYmkq+vLykUClKpVKRQKCgwMJCWLFlCN27coMTERPL09KTMzEwiIho5ciT5+/vT4cOHKScnR3hsQUFB9Ntvv9HZs2cpMDCQYmNjiYjo2rVr5O7uTlu3bqXbt2/Txo0bydPTk27fvq1z/w/WV1ZWRp07d6Zz584REZFSqSRvb2/69ddfiYho+vTp9Pbbb9PVq1fp1KlT1Lt3b9q0aVOd8cjOziYPDw/64YcfKCcnhxYuXEi+vr5UVVXVYD8xMTHk5eVFx44dI4VCQevXryc3Nze6ceNGg2NdOzZjx46l8+fPU35+foPrR0ZG0rRp00ihUBARUUpKCnl4eNC3335L2dnZtGrVKvLw8KBr164J9bm5udGGDRsoIyODNBqN8LhramooJyeHJBIJXbx4kZRKJRERTZ06lUaMGEEpKSl04cIFCgkJoVmzZuk8ls6ePUtubm6UkJBAGRkZFBsbSxKJhLZu3dpgfbqe34fV1r98+XK6cuUKLV++nDp37kxFRUUN1qprvB7neCMiWrNmDb300kt06NAhyszMpBkzZlD//v2FsRs5ciR5eXnRDz/8QOnp6RQWFkahoaE6x06Xf65cTQs/jSW1Wk01NTU0J2o+xSxdQcd+PkEfz11In8V/SUREJSUl9N6Hs2jf/kN69/0seGoDZ8QXd2lP8j2t0NE3bJoicBQKBUmlUjp+/LiwfNeuXXT16lUiIurbty/t3r1bWHbp0iVyd3enmpqaOvvLzMykgoIC4f53331HQ4cOJSKq8wLbvXs3+fr6Cuvu2LGDgoODtfqbMmUKxcfHE9H9F9yXX35Z57E9WPfy5ctp8uTJREQkl8spLS1Nq79u3brRwYMHde7/4fqmTp1KS5YsISKi48ePk6+vL6lUKrp58yZJpVK6e/eusG1CQgK98cYbdcbj6NGj5OnpSaWlpULNGzZsILlc3mA/MTEx9M4772j1N2rUKIqLi2twrGvH5syZM8Ly+tYnIpo3bx69//77wv2ZM2fStGnTtPYfHh5OH3/8sVDfpEmT6jzmWvn5+SSRSITj6Pr16ySRSOjy5cvCOqmpqSSVSikvL6/O9jNnzqSpU6dqtfXq1UsInIbqe/j5fVhMTAy9/vrrwv2qqiqSSqWUnJysV60Pj9fjHG8ajYZ69epFa9eu1arHz8+PkpKSiOj+8b9s2TJh+c8//0weHh6kVqsf+RhrVVZW0pSID+j02WQiIrp1+w79PWoB3btXSEREyecu0JSID4SThk1bttPCxTEN9vsseWo/w7lwk/DNSTUAMwR3vj8z6GRnYrTPbMRiMSIiIvD+++/j1VdfxYABAzBs2DCIxWKUlpYiLy8PkZGRiI6OBnD/y+7UajXu3LmDjh07avXl6OiI+Ph4/PLLLygqKoJKpYKzs7NedWRkZODGjRvw9vYW2pRKJdq0aSPct7S0rLOdnZ2d8H8rKytheqN169Y4d+4cFi1ahOvXr0OlUkGhUGhN/dRn+PDh+Oc//4moqCgcOXIEgwcPhpmZGS5fvgwiwoABA4R1VSoVHBwc6vQREBAAPz8/BAUFYciQIQgODsabb74JU1NTnDlzRu9+anl5eSE3NxeAfmNtYfHHFY6NfW4yMjIwduxYrTZ/f3+tac4H+29IRkYGxGIxvLy8hDZvb2+0atUK165dg4uLi9b6N27cQHBwsFabufkfL3t96mvIg8eOSCSCqakpFAoF8vPzG1Ur8HjHW3FxMe7evYuXX35Zq57u3btrXXTxYL1isRgqlQo1NTUQiUT19n9HdhdqtRoe7m4AAGenDli+5FMA94+5U2fOoktnD5iY3J/md3eT4ujxX0BEQtuz7qkNHEB36DRF2FhYWMDU1BQ1NdrzsUqlEgCEAzUyMhITJkzA4cOHsWnTJqxYsQLbt2+Hvb09ACA+Pl7rxQcATk5OdfYXFRUFlUqFLVu2wMnJCRs3bsTGjRv1rtfPzw/Lli3TarO2ttZ7+welpaVhxowZiI+Px4ABA2BhYdGoq5YGDRqE6OhoXLt2DYcPH8bSpUuFZZaWlti7d6/W+ro+lBeJRNi8eTMuXLiAY8eOYcGCBXBxccGmTZsa1U+tB5/Hxo51Y9e3tLTU+gEP3L84oPbYaSxLS8s6FxbQ/76VV1efJiYmOk8wmqq+x6kVeLzjrfZxPvzcq9XqJ/J4ao8bS4u64/nFV/9Cfv49LJj78R/1iCyhVquhIYIZBw6AFnDRQG3oJOdQk72zsbCwQMeOHZGWlqbVnpaWBisrKzg7OyMrKwtxcXHo1KkTwsPDkZSUhNatW2P37t2ws7ODo6Mj8vLy4OrqKtwcHBzqvNgB4OTJkxg/frwQRvV9IPzwmZObmxuysrLQvn17YT+Ojo71nvHX58yZM5BIJBgyZAgsLCygUqm0XrwNnblZW1ujX79++Pzzz1FVVYWAgAAAgLu7O5RKJcrKyoQ6n3/+eZ11Hjt2DNu3b4evry9mz56NH3/8EcnJyUhJSdGrnwcvWgCA1NRUuLq6AmjcWOuz/sPj4eHhgQsXLmi1nTt3Dl26dKl3P/X1V15ejqysLKEtLS0N1dXVOvuUSCS4evWqVhvRH6+Phup7nDNzfWp9uP/HOd5sbGzQoUMH/Pe//xXaqqurkZaWpvd416ft/2YJZA/9zk/uzTykpWcgYmoYbG1thHaZ7C5a29nBjK88FLSIkagNnaacRps6dSpWrVqFbdu24cqVK9izZw+WL1+O8PBwmJqawsbGBmvWrMEXX3yBmzdv4uTJk5DJZJBKpQCAiIgIfPXVV9izZw9u3ryJlStXYuzYsVov/loSiQT//ve/kZKSgh07dmD16tWPPENr06YN5HI5Tp48Cblcjtdeew0WFhb429/+hszMTKSkpOD111/H/v37/9TjfuGFF5CZmYmkpCScP38eH330ERQKhVDPw/vXZfjw4Thw4AAGDRokBKxUKsXAgQMxe/ZsJCcnIzs7GzNmzMCSJUvqbK9UKvHJJ59g7969uHXrFvbu3Qtzc3N07NhRr35Onz6NxMRE5ObmIi4uDtevX8eYMWMaPdb6rG9vb4+MjAykpqZCpVLhL3/5Cw4ePIh169bh999/R0JCAk6cOIF3331Xr/G3s7ODqakpjh49CplMBldXV4SEhGDOnDm4dOkSLl68iKioKAwePBidOnWqs/2kSZOwb98+HDlyBJWVldi0aRNu374tLG+oPn2e30fRp9aHx+txj7f33nsPn3/+OY4cOYKsrCxERUVBJBJh+PDhjapdFweHdnDq0B6/nTqj1W7f2g7vR4RDInlBaCMinDp9Ft5dtWc0nnUtInCA+6HTlJ/ZTJo0CVFRUVi7di1GjRqF+Ph4hIeHY9asWQDu/wLamjVrcPToUQwaNAiRkZGIiIjAsGHDAADvvvsuwsPDERMTg0GDBuH48eOIi4vTecb22WefoaysDG+++SZ27dqFWbNmQSwW6/xB+Morr6B3796IiIjA+fPnIRaLsX79esjlcoSGhmLatGkYMmQIBg8e/Kce98CBAxEREYGYmBh89NFH6N69O4KCglBWVqZz/7oEBwejVatWGDp0qFb7ihUr0LVrV4SHh+O1116Dubm5MJ4PGjJkCD7++GMsW7YMAwcOxObNm7Fy5Uq0b99er366d++OU6dOYdSoUdi5c6fwTrSxY63P+mPGjIGJiQkmT56MsrIyeHl5YeXKldiyZQtCQkLw448/4ttvv9X6jK0+lpaWmDZtGr7++mts2LABALBs2TK4u7tj8uTJCAsLg4+PD+Lj43Vu//LLLyMyMhKRkZEIDAxEWloa3N3dheUN1afP81ufhmp9eLwe93h76623EBYWhnnz5iE0NBRFRUXYsmULxGJxo2vXZfiwEPx84ldcufrHZ0I3825jVcJ3WlO1Bw4dxs28WxgW8udedy2Viby0TPgpXXu2XTsFodFohDlXtUoFtUYDlUol3Kqrq1FTUwN/P1+DFSyJyn/sv/ZsaW6C35c+94QqYs1ZbGwsMjMzsX79emOXwlqI1QlrkXLxEt4NexOv+L+stUylUuE/+w7ixz17MWnCOAwaGGSkKpunp+6igaVj7J7IF7Axxtif8ZepYdiyLRGrE9Zi776D6PqiF8RiMYqLi5FyKRXl5eV4561JeLVfoLFLbXaeusAZ79+Kv62TMWY0ZqameHPiG+gXGIBffv0NmdeyUFlVBVtbG/TtE4BX+wWiTRt7Y5fZLD11U2qMMcaeTi3mogHGGGPNGwcOY4wxg+DAYYwxZhAcOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDOIp+77cIwlOzsb+/btg0wmg62tLfr164eAgABjlwUAOHPmDKRSKRwdHY1dCmOMPRIHjh4qKiqwfv16jB07Fi+++CLy8/OxYcMG2Nraolu3bk26byKCiYlJveucOXMGdnZ2jQqc2u8+aqhvxhh7Ujhw9FBYWAhTU1P4+PgAAFxcXBASEgKFQgHg/hfV7d27F8nJybCxscHIkSPh5eUFmUyGtWvXwtXVFTk5ObC1tcWECROEYMjOzsbOnTuhUCggkUgwYcIEtGrVCjKZDAkJCejUqRPu3LmD6OjoR667dOlSyOVybN26FUOGDEFAQABycnKwa9cuFBYWwtXVFWPHjoW9/f1vIFy0aBG6dOmC9PR0zJo1C+3atTPOoDLGnjn8GY4enJycYGNjg8TERBQVFQEAfH190atXLwDAL7/8glu3biE6Ohrjx4/Htm3bUFlZCQAoKytDnz59MHfuXHh5eWHHjh0AgPLycqxfvx6TJ0/GggULoFQqceLECWGfCoUC3bp1w+zZs+tdNyoqCs7OzpgwYQICAgJQWVmJdevWISgoCJ988glcXV2xceNG4R0NAFhZWSE6Ohpt27Y1yPgxxhjAgaMXc3NzfPDBB2jVqhW+/vprrFy5EleuXBGWnz9/HkFBQbCysoKrqytcXFxw/fp1APd/uL/wwgsAgP79+yMnJwdVVVUQiUT48MMP4eLiAnNzc3Tu3BkFBQVCn9bW1vDz84OlpWWD6z7oypUrcHR0hI+PD8zNzTFw4EAUFxcjPz9fWKd3794Qi8U8ncYYMyieUtOTWCzGiBEjMGzYMKSnp2P79u0YPXo0vL29UVpaiu+//174Aa7RaNC9e/c601UWFhZo1aoVysvL4eDggNu3byMxMRFKpVKYKtPFwsJC73XLy8vRpk0b4b6pqSlat24NuVyO9u3bP6HRYIyxxuPA0cOlS5dQXFyM/v37w9TUFN26dcOtW7eQnp4Ob29v2NraYty4cXB1ddXaTiaTad2vqqpCVVUVbGxscOvWLRw4cADTp09H69atceLECeTm5urcf2PWtbe3R2pqqnBfo9GgpKQENjY2jzkKjDH2eHhKTQ9t2rTBkSNHcPnyZdTU1KCoqAhXr16Fk5MTAMDHxweHDh1CRUUFKioqsGfPHpSUlAC4f4Vbeno61Go1fvrpJ7i6ugrvcogIRISCggJcvHgRGo1G5/4bWtfS0hKFhYVQq9Xw9PTEvXv3kJaWBrVajZ9//hk2NjZwdnZu+oFijLF68DscPXTs2BETJ07ETz/9hO+//x7W1tZ46aWXEBgYCAAICgrCvn37sGLFCmg0Gvj7+8POzg6VlZUQiURITU3F9u3b0a5dO0ycOBEA4OnpCU9PT8TFxaFNmzbo3Lkz7t27p3P/Da3bs2dP7Nq1C2q1Gv3798eUKVOQlJSEbdu2wcXFBWFhYfx5DWPM6EzkpWXC5Uu1VzLVnj1rNBoQETQaDdQqFdQaDVQqlXCrrq5GTU0N/P18jVN9M1d7efPChQuNXQpjjBkdT6kxxhgzCA4cxhhjBsFTaowxxgyC3+EwxhgzCA4cxhhjBsGBwxhjzCD493D0VF2tRGGRHIqKSmOXwhhrxqzFVmjX1h4ikaWxS2l2OHD0UF2tRG7eHTg6tIWz03PGLoexZ9617Bx4uLk2vKIRyEvKkJt3B52ed+LQeQhPqemhsEgOR4e2sG9ta+xSGGPNnH1rWzg6tEVhkdzYpTQ7HDh6UFRUctgwxvRm39qWp9914MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDMIDhzGGGMGwYHDGGPMIDhwGGOMGQQHDmOMMYPgwGGMMWYQHDiMMcYMggOHMcaYQfBfi2aMtXiJiYmoqakR7o8bNw7m5uaoqqrCrl27hHaRSITRo0cDAJKTk5GVlSUs8/f3h7u7u+GKboE4cBhjLV51dTWUSmWddiJCVVWVzm1UKpXWMrVa3WT1PSs4cBhjLVJpaSkqKioA3A+WB929exdmZmaorq7WatdoNJDJZAAgbFurpKREWPbcc8/B1JQ/kWgsDhzGWIuUmpqKzMxMncsOHTqks12pVGL//v06l6WnpyM9PR0AMGnSJIhEoidT6DOEI7oJpaenY/DgwfDz88OWLVuE9m+++QaLFy82YmWMsQfZ2trC3JzPv5saB04TioyMxKBBg7B48WIsXboUq1evRmlpKfbt2wdfX19jl8cY+5/+/fvDwcHB2GW0eBzpTaSiogJ5eXn44IMPIBaL0bFjR0RERGD58uXo168fhg0bZuwSGWPMoDhwmohYLEZKSopw38fHB7/99htkMhlcXFyMWBljDADMzMzq3K9t02g0dS40YI+PA8eAzMzMOGwYayZCQ0OFD/5FIhFeffVV4dLns2fP4vr168Ysr0XiwGGMPZMe/IXPESNG4Ny5c8Jlz6xp8EUDjDHGDIIDhzHGmEHwlBpj7Jm3f/9+aDQaY5fR4nHgMMZapM6dO8PZ2blJ+rawsGiSfls6DhzGWIvk4ODAv8zZzPBnOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MDRg7XYCvKSMmOXwRh7SshLymAttjJ2Gc0OB44e2rW1R8G9Ig4dxliD5CVlKLhXhHZt7Y1dSrPDf2lADyKRJTo974TCIjkK7hUZuxzGGIBr2TnGLkEna7EVOj3vBJHI0tilNDscOHoSiSzh7PScsctgjLGnFk+pMcYYMwgOHMYYYwbBgcMYY8wgOHAYY4wZBAcOY4wxg+DAYYwxZhAcOIwxxgyCA4cxxphBcOAwxhgzCA4cxhhjBsGBwxhjzCA4cBhjjBkEBw5jjDGD4MBhjDFmEBw4jDHGDIIDhzHGmEFw4DDGGDOI/wcoyo0tIkmfFQAAAABJRU5ErkJggg==)

then the tilde character (`~`) denotes nesting everywhere "dot notation" can be used. You can now denote the example path above as `user_data.address~city`.

It is also possible to use more than one character as alternative separator, for example:

![second alternative separator example](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZsAAABxCAYAAAADMA6oAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABouSURBVHic7d15WFNn2gbwGwTCLipUgSombFJRLBSriFoUFTdsXerWhYqDrW3VGZ0CbnX6DbhUaDutY4daResuYqt1rVtrrQvqoIAoghXcEATCEpaQ5Pn+cDglEiWop0F8fteVS/Kec97z5M0J98mbgzEqLa8gIkKdup81Go3wLxFBo9FArVJBrdFApVIJt5qaGtTW1sLfzxeMMcaYLsaGLoAxxljLx2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0JoYu4FFsSalG1PYyKFXU+Mo6mJkYYckYW4z3N3/ClTHGGNPlqXxn8zhBAwBKFSFqe9kTrIgxxtjDPJVh8zhB8yT7YIwxpp+nMmwYY4w9XZ6ZsAl0NTJ0CYwx9sx6JsIm0NUI0/q0wrlI00fuo7KyEjKZDCdPntRqz8nJgUwmQ05OzuOWqbeLFy9CJpOhpKTkT9tnU+Tk5CAgIAB37twxdCnsAfr27YstW7YYuownwtDH263b+Th+4iQOHjqK0ylnIS8tNUgdzV2LD5u6oOnqaISdaRo42ra8dzi7du2Cn59fs9l/hw4dEB4ejrZt2xqsJkOaP38+pk+f/sT6KywshEwmQ1ZW1hPrsynEPr6aOl7N5XjLzc3DPxd/irkL/oHv1m/Gnv0HkPBtIv46Jxor//Mth859nspLn/V1f9D851cNbpfxhQH1qdVqtGrV6on2aWVlhfDw8CfaZ3MhxnjVp9FoYGzc4s8BnyhDHG+p5y9gxder4O4qw/zov8NVJoWRkRFUKhXOX0jHtu3f4x//twSRc2ahQ4f2f2ptzVWLOKpd7Ru+WzFU0Fy8eBHjxo1D165dERQUhKSkJK3lq1evRkBAAHx9fTFr1izI5XKd/VRUVGDu3Lnw9fVFjx49MHPmTJSVNbxce9q0aZg5cyZKSkogk8lw+PBhAEBeXh7eeusteHt7Y/Dgwdi3b5+wTWhoKObMmYOhQ4di/PjxwhThli1bMHToUHTt2hVvv/02iouLhW2OHDmCESNGoEuXLggODhb607X/+tN8ycnJ6Nmzp/CVFQAwffp0zJs3DwBQVVWFuXPnwsfHB3369MEXX3wBtVqtc0x+/PFHBAcHo2vXrhg9ejRSU1OFZQ/rJzY2FmFhYfjggw/Qo0cPBAQEIDk5Wa+xrhubmJgYBAQEIC4u7qHr+/j4YOPGjdi3bx9kMhkqKysBAEePHsWwYcPQpUsXhISE4OjRo8L+Y2NjMX78eEyZMgVeXl6oqakRlq1ZswYvv/wyACAkJASxsbFCXdHR0ejRowd8fHwQGRkp7EuXtWvXwt/fHz179sTSpUsbjPGD6nvQ8VVfbGws3n33XSxatAjdu3dH7969tY77h9X6oPF61OOtTkJCAvr06QMvLy+8+eab+P3334VloaGhiIuLw5QpU/DCCy8gJCQEFy5ceODY3e92fj5WJqxGYEAv/H32TFhYWOCfiz/Fux/8FV+t/AaurlJ8PD8K7dq1xWdf/htKZa3efbdopeUVJC8rF24lpWVUUlpGRSVyKiqRU2FRMRXcLaL8gkK6ees25d24SVev5VJWdg5dvHSZ/nv+Ap0+c5b+TE6z84VbfkERrT9WSH0X3xHaXv93AR06f5fyC4oo4VAh+f3jjtY2dbemUCgUJJVK6cSJE1rt2dnZJJVKKTs7m4iIAgIC6JNPPqG8vDzaunUrubu7U2pqKhERrVu3jvr27UsnT56krKwsmjhxIs2YMUPn/mbOnEljxoyhy5cvU1ZWFo0cOZLmz59PREQZGRkklUqpuLiYqqurKSkpiXx9fUmhUJBKpSKFQkGBgYG0ePFiunbtGiUlJZGHhwdlZWUREdHIkSPJ39+fDh48SLm5ucJjCwoKot9++41Onz5NgYGBFBsbS0REV65cITc3N9q8eTPdunWL1q1bRx4eHnTr1i2d+69fX3l5OXl6etKZM2eIiEipVJK3tzf9+uuvREQ0ffp0euutt+jy5ct04sQJ6t27N23YsKHBeOTk5JC7uzt9//33lJubSwsXLiRfX1+qrq5utJ+YmBjy8vKiI0eOkEKhoMTERHJ1daVr1641OtZ1YzN27Fg6e/YsFRQUNLp+ZGQkTZs2jRQKBRERpaamkru7O33zzTeUk5NDK1euJHd3d7py5YpQn6urK61du5YyMzNJo9EIj7u2tpZyc3NJKpXS+fPnSalUEhHR1KlTacSIEZSamkrnzp2jkJAQmjVrls5j6fTp0+Tq6koJCQmUmZlJsbGxJJVKafPmzY3Wp+v5vV9d/cuWLaNLly7RsmXLyNPTk4qLixutVdd4Pc7xRkS0atUqevHFF+nAgQOUlZVFM2bMoP79+wtjN3LkSPLy8qLvv/+eMjIyKCwsjEJDQ3WOnS7/WvE1LfwkltRqNdXW1tKcqPkUs2Q5Hfn5GH00dyF9Gv8FERGVlpbSux/Moj17D+jdd0v2VIfNiM/v0K6Uu1qBo2/QiBE2CoWCZDIZHT16VFi+Y8cOunz5MhER9e3bl3bu3Cksu3DhArm5uVFtbW2D/WVlZVFhYaFw/9tvv6WhQ4cSETV4ce3cuZN8fX2Fdbdt20bBwcFa/U2ZMoXi4+OJ6N6L7Ysvvmjw2OrXvWzZMpo8eTIREcnlckpPT9fqr1u3brR//36d+7+/vqlTp9LixYuJiOjo0aPk6+tLKpWKrl+/TjKZjO7cuSNsm5CQQK+//nqD8Th8+DB5eHhQWVmZUPPatWtJLpc32k9MTAy9/fbbWv2NGjWK4uLiGh3rurE5deqUsPxh6xMRzZs3j9577z3h/syZM2natGla+w8PD6ePPvpIqG/SpEkNHnOdgoICkkqlwnF09epVkkqldPHiRWGdtLQ0kslkdOPGjQbbz5w5k6ZOnarV1qtXLyFsGqvv/uf3fjExMfTaa68J96urq0kmk1FKSopetd4/Xo9zvGk0GurVqxetXr1aqx4/Pz9KTk4monvH/9KlS4XlP//8M7m7u5NarX7gY6xTVVVFUyLep5OnU4iI6Oat2/T3qAV0924RERGlnDlHUyLeF04YNmzaSgv/EdNov8+Cp/ozm3PXCf85rgbQCsGe92YEHW2NDPYZjaWlJSIiIvDee+/hlVdewYABAzBs2DBYWlqirKwMN27cQGRkJKKjowHc+1ZUtVqN27dvo2PHjlp9OTg4ID4+Hr/88guKi4uhUqng5OSkVx2ZmZm4du0avL29hTalUok2bdoI983MzBpsZ2trK/xsYWEhTGm0bt0aZ86cwaJFi3D16lWoVCooFAqt6Z6HGT58OP71r38hKioKhw4dwuDBg9GqVStcvHgRRIQBAwYI66pUKtjb2zfoIyAgAH5+fggKCsKQIUMQHByMN954A8bGxjh16pTe/dTx8vJCXl4eAP3G2tT0jysZm/rcZGZmYuzYsVpt/v7+WlOb9ftvTGZmJiwtLeHl5SW0eXt7w9zcHFeuXIGzs7PW+teuXUNwcLBWm4nJHy99feprTP1jRyKRwNjYGAqFAgUFBU2qFXi8462kpAR37tzBSy+9pFVP9+7dtS6wqF+vpaUlVCoVamtrIZFIHtr/7fw7UKvVcHdzBQA4OXbAssWfALh3zJ04dRpdPN1hZHRvat/NVYbDR38BEQltz6qnOmwA3YEjRtCYmprC2NgYtbXa869KpRIAhIM0MjISEyZMwMGDB7FhwwYsX74cW7duhZ2dHQAgPj5e64UHAI6Ojg32FxUVBZVKhU2bNsHR0RHr1q3DunXr9K7Xz88PS5cu1WqzsrLSe/v60tPTMWPGDMTHx2PAgAEwNTVt0tVJgwYNQnR0NK5cuYKDBw9iyZIlwjIzMzPs3r1ba31dH8BLJBJs3LgR586dw5EjR7BgwQI4Oztjw4YNTeqnTv3nsalj3dT1zczMtH65A/cuBKg7dprKzMyswUUE9L+vbtfVp5GRkc6TC7Hqe5xagcc73uoe5/3PvVqtfiKPp+64MTNtOJ6ff/lvFBTcxYK5H/1Rj8QMarUaGiK0esbDpkVcIFAXOCm5JNo7GlNTU3Ts2BHp6ela7enp6bCwsICTkxOys7MRFxeHTp06ITw8HMnJyWjdujV27twJW1tbODg44MaNG3BxcRFu9vb2DV7oAHD8+HGMHz9eCKKHffh7/xmTq6srsrOz0b59e2E/Dg4ODz3Tf5hTp05BKpViyJAhMDU1hUql0nrhNnbGZmVlhX79+uGzzz5DdXU1AgICAABubm5QKpUoLy8X6nz++ed11nnkyBFs3boVvr6+mD17Nn744QekpKQgNTVVr37qX6AAAGlpaXBxcQHQtLHWZ/37x8Pd3R3nzp3Tajtz5gy6dOny0P08rL+KigpkZ2cLbenp6aipqdHZp1QqxeXLl7XaiP54fTRW3+OcketT6/39P87xZm1tjQ4dOuC///2v0FZTU4P09HS9x/th2v5vdiD/vr/pybt+A+kZmYiYGgYbG2uhPT//Dlrb2qIVX2HYMsIG+CNwxJw6mzp1KlauXIktW7bg0qVL2LVrF5YtW4bw8HAYGxvD2toaq1atwueff47r16/j+PHjyM/Ph0wmAwBERETgyy+/xK5du3D9+nWsWLECY8eO1Xrh15FKpfjuu++QmpqKbdu24euvv37gmVmbNm0gl8tx/PhxyOVyvPrqqzA1NcXf/vY3ZGVlITU1Fa+99hr27t37SI+7c+fOyMrKQnJyMs6ePYsPP/wQCoVCqOf+/esyfPhw7Nu3D4MGDRLCVSaTYeDAgZg9ezZSUlKQk5ODGTNmYPHixQ22VyqV+Pjjj7F7927cvHkTu3fvhomJCTp27KhXPydPnkRSUhLy8vIQFxeHq1evYsyYMU0ea33Wt7OzQ2ZmJtLS0qBSqfCXv/wF+/fvx5o1a/D7778jISEBx44dwzvvvKPX+Nva2sLY2BiHDx9Gfn4+XFxcEBISgjlz5uDChQs4f/48oqKiMHjwYHTq1KnB9pMmTcKePXtw6NAhVFVVYcOGDbh165awvLH69Hl+H0SfWu8fr8c93t5991189tlnOHToELKzsxEVFQWJRILhw4c3qXZd7O3bwbFDe/x24pRWu11rW7wXEQ6ptLPQRkQ4cfI0vLtqz2Q8q1pM2AD3AkfMz2gmTZqEqKgorF69GqNGjUJ8fDzCw8Mxa9YsAPf+uGzVqlU4fPgwBg0ahMjISERERGDYsGEAgHfeeQfh4eGIiYnBoEGDcPToUcTFxek8U/v0009RXl6ON954Azt27MCsWbNgaWmp85fgyy+/jN69eyMiIgJnz56FpaUlEhMTIZfLERoaimnTpmHIkCEYPHjwIz3ugQMHIiIiAjExMfjwww/RvXt3BAUFoby8XOf+dQkODoa5uTmGDh2q1b58+XJ07doV4eHhePXVV2FiYiKMZ31DhgzBRx99hKVLl2LgwIHYuHEjVqxYgfbt2+vVT/fu3XHixAmMGjUK27dvF96BNnWs9Vl/zJgxMDIywuTJk1FeXg4vLy+sWLECmzZtQkhICH744Qd88803Wp+pPYyZmRmmTZuGr776CmvXrgUALF26FG5ubpg8eTLCwsLg4+OD+Ph4ndu/9NJLiIyMRGRkJAIDA5Geng43NzdheWP16fP8Pkxjtd4/Xo97vL355psICwvDvHnzEBoaiuLiYmzatAmWlpZNrl2X4cNC8POxX3Hp8h+fAV2/cQsrE77Vmp7dd+Agrt+4iWEhj/a6a2mMSssrqP6Zdd3PddMOGo1GmGNVq1RQazRQqVTCraamBrW1tfD38/3TipZGFTz2/9psZmKE35c894QqYs1ZbGwssrKykJiYaOhSWAvxdcJqpJ6/gHfC3sDL/i9pLVOpVPhxz378sGs3Jk0Yh0EDgwxUZfPyVF4gsGSM7RP58jTGGHsUf5kahk1bkvB1wmrs3rMfXV/wgqWlJUpKSpB6IQ0VFRV4+81JeKVfoKFLbTaeyrAZ72/O37LJGDOYVsbGeGPi6+gXGIBffv0NWVeyUVVdDRsba/TtE4BX+gWiTRs7Q5fZrDyV02iMMcaeLi3qAgHGGGPNE4cNY4wx0XHYMMYYEx2HDWOMMdFx2DDGGBMdhw1jjDHRcdgwxhgTHYcNY4wx0XHYMMYYEx2HDWOMMdFx2DDGGBMdhw1jjDHRcdgwxhgTHYcNY4wx0T2V32djKDk5OdizZw/y8/NhY2ODfv36ISAgwNBlAQBOnToFmUwGBwcHQ5fCGGMNcNjoqbKyEomJiRg7dixeeOEFFBQUYO3atbCxsUG3bt1E3TcRwcjI6KHrnDp1Cra2tk0Km7rvLmqsb8YYe1wcNnoqKiqCsbExfHx8AADOzs4ICQmBQqEAcO9L5nbv3o2UlBRYW1tj5MiR8PLyQn5+PlavXg0XFxfk5ubCxsYGEyZMEEIhJycH27dvh0KhgFQqxYQJE2Bubo78/HwkJCSgU6dOuH37NqKjox+47pIlSyCXy7F582YMGTIEAQEByM3NxY4dO1BUVAQXFxeMHTsWdnb3vjlw0aJF6NKlCzIyMjBr1iy0a9fOMIPKGHtm8Gc2enJ0dIS1tTWSkpJQXFwMAPD19UWvXr0AAL/88gtu3ryJ6OhojB8/Hlu2bEFVVRUAoLy8HH369MHcuXPh5eWFbdu2AQAqKiqQmJiIyZMnY8GCBVAqlTh27JiwT4VCgW7dumH27NkPXTcqKgpOTk6YMGECAgICUFVVhTVr1iAoKAgff/wxXFxcsG7dOtT/RlYLCwtER0ejbdu2f8r4McaebRw2ejIxMcH7778Pc3NzfPXVV1ixYgUuXbokLD979iyCgoJgYWEBFxcXODs74+rVqwDu/WLv3LkzAKB///7Izc1FdXU1JBIJPvjgAzg7O8PExASenp4oLCwU+rSysoKfnx/MzMwaXbe+S5cuwcHBAT4+PjAxMcHAgQNRUlKCgoICYZ3evXvD0tKSp9AYY38KnkZrAktLS4wYMQLDhg1DRkYGtm7ditGjR8Pb2xtlZWVYv3698Mtbo9Gge/fuDaaoTE1NYW5ujoqKCtjb2+PWrVtISkqCUqkUpsd0MTU11XvdiooKtGnTRrhvbGyM1q1bQy6Xo3379k9oNBhjTH8cNnq6cOECSkpK0L9/fxgbG6Nbt264efMmMjIy4O3tDRsbG4wbNw4uLi5a2+Xn52vdr66uRnV1NaytrXHz5k3s27cP06dPR+vWrXHs2DHk5eXp3H9T1rWzs0NaWppwX6PRoLS0FNbW1o85Cowx9mh4Gk1Pbdq0waFDh3Dx4kXU1taiuLgYly9fhqOjIwDAx8cHBw4cQGVlJSorK7Fr1y6UlpYCuHclW0ZGBtRqNX766Se4uLgI726ICESEwsJCnD9/HhqNRuf+G1vXzMwMRUVFUKvV8PDwwN27d5Geng61Wo2ff/4Z1tbWcHJyEn+gGGNMB35no6eOHTti4sSJ+Omnn7B+/XpYWVnhxRdfRGBgIAAgKCgIe/bswfLly6HRaODv7w9bW1tUVVVBIpEgLS0NW7duRbt27TBx4kQAgIeHBzw8PBAXF4c2bdrA09MTd+/e1bn/xtbt2bMnduzYAbVajf79+2PKlClITk7Gli1b4OzsjLCwMP58hjFmMEal5RVU/yqlup/rzpo1Gg2ICBqNBmqVCmqNBiqVSrjV1NSgtrYW/n6+BnkAzV3dJcwLFy40dCmMMWYwPI3GGGNMdBw2jDHGRMfTaIwxxkTH72wYY4yJjsOGMcaY6DhsGGOMiY7/zqYJamqUKCqWQ1FZZehSGGPNmJWlBdq1tYNEYmboUpoNDhs91dQokXfjNhzs28LJ8TlDl8PYM+9KTi7cXV0aX9EA5KXlyLtxG52ed+TA+R+eRtNTUbEcDvZtYdfaxtClMMaaObvWNnCwb4uiYrmhS2k2OGz0pKis4qBhjOnNrrUNT7nXw2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0HDaMMcZEx2HDGGNMdBw2jDHGRMdhwxhjTHQcNowxxkTHYcMYY0x0/L8+M8aeCUlJSaitrRXujxs3DiYmJqiursaOHTuEdolEgtGjRwMAUlJSkJ2dLSzz9/eHm5vbn1d0C8Jhwxh7JtTU1ECpVDZoJyJUV1fr3EalUmktU6vVotXX0nHYMMZarLKyMlRWVgK4Fyr13blzB61atUJNTY1Wu0ajQX5+PgAI29YpLS0Vlj333HMwNuZPIvTFYcMYa7HS0tKQlZWlc9mBAwd0tiuVSuzdu1fnsoyMDGRkZAAAJk2aBIlE8mQKfQZwLDPGGAAbGxuYmPD5t1g4bBhjDED//v1hb29v6DJaLA4bxhhjouP3jIyxZ1arVq0a3K9r02g0DS4qYI+Ow0ZkiYmJ6N27Nzw9PVFZWYlVq1YhLCwMtra2hi6NsWdeaGio8CG/RCLBK6+8IlzefPr0aVy9etWQ5bUoHDYi27hxI8zNzeHp6YmCggKsX78egwcP5rBhrBmo/8ecI0aMwJkzZ4RLm9mTxWEjsvqXV3bu3BmnT582YDWMMWYYfIEAY4wx0fE7G8YYA7B3715oNBpDl9FicdgwxlosT09PODk5idK3qampKP22VBw2jLEWy97env9Qs5ngz2wYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJjsNGT1aWFpCXlhu6DMbYU0JeWg4rSwtDl9FscNjoqV1bOxTeLebAYYw1Sl5ajsK7xWjX1s7QpTQb/D8I6EkiMUOn5x1RVCxH4d1iQ5fDGANwJSfX0CXoZGVpgU7PO0IiMTN0Kc0Gh00TSCRmcHJ8ztBlMMbYU4en0RhjjImOw4YxxpjoOGwYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJjsOGMcaY6DhsGGOMiY7DhjHGmOg4bBhjjImOw4YxxpjoOGwYY4yJ7v8B5QR/rDHF1T0AAAAASUVORK5CYII=)

Now, you can denote nesting by using 2 dots, and the example path above can be denoted as `user_data.address..city`.

## Snowplow Event Mapping Options

This section includes the mapping rules that concern a Snowplow event as claimed by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/):

### Snowplow Atomic Properties Rules

This option indicates if all Snowplow [atomic](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0) properties of the event should be included in the JSON body. By default this option is disabled.

If enabled, an additional text field optionally allows you to specify a key under which those atomic properties will be nested. Leaving it blank adds those properties in the request body without nesting. Dot notation can also be used here.

As an example, this section configured as:

![snowplow atomic properties rules](/assets/images/snowplow_atomic_nest-4975fe7246e9a491c69b3042e45b280c.png)

will result in the following JSON structure:

```json
{
    ...,
    "snowplow_atomic": {
        "app_id": "fooBar",
        "platform": "mobile",
        ...
    },
    ...
}
```

Please, note that some of the Snowplow atomic properties are already mapped to [common event properties](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) by the [Snowplow Client](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/).

### Snowplow Self-Describing Event Rules

This option indicates if the Snowplow Self-Describing data will be included in the request body and it is enabled by default.

Similarly to the above section, you can also specify a key under which the self-describing data will be nested. Leaving it blank adds those properties in the request body without nesting. Dot notation can also be used here.

As an example, this section configured as:

![snowplow self-describing event rules](/assets/images/snowplow_self_desc_no_nest-c7502209e3ea302074ec4427c13b5b6d.png)

will result in the following JSON structure:

```json
{
    ...,
    "self_describing_event_com_acme_test_foo_1": {
        "mySelfDescProp": "exampleValue",
        ...
    },
    ...
}
```

### Snowplow Event Context Rules

This section describes how the HTTP Request tag will use the context Entities attached to a Snowplow Event.

![snowplow event context rules](/assets/images/context_rules-9da59ffc2b9d28034d92e7d5a9013550.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in request body

Using this drop-down menu you can specify whether you want to Include `All` (default) or `None` of the Snowplow context entities in HTTP Request's body.

#### Nest all unmapped Entities under key

This option is available only if the previous option ([Include Snowplow Entities in request body](#include-snowplow-entities-in-request-body)) is set to `All`.

It applies **only** to unmapped entities, i.e. all included entities whose mapping is not edited in the following ([Snowplow Entities to Add/Edit mapping](#snowplow-entities-to-addedit-mapping)) table.

With this setting you can specify a key under which the Snowplow event's unmapped entities will be nested. Alternatively, leaving it blank adds the unmapped entities in the request body without nesting. You can also use dot notation in this value.

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- **Entity Name**: The Entity name to add/edit-mapping (required).¹
- **Destination Mapped Name**: The key you could like to map it to in the request body (optional: leaving the mapped key blank keeps the same name). You can use dot notation here as well to signify further nesting. This value is independent of the [nesting of unmapped entities](#nest-all-unmapped-entities-under-key) setting above.
- **Apply to all versions**: Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if [Include Snowplow Entities in request body](#include-snowplow-entities-in-request-body) is set to `All`), you can specify the context entities you want to exclude from the HTTP Request body. In its columns you can specify:

- **Entity Name**: The Entity name (required).¹
- **Apply to all versions**: Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

This section describes how the HTTP Request tag will use the context Entities attached to a Snowplow Event.

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in request body

Leaving this option enabled (default) ensures that all Entities on an event will be included within the request data.

Optionally, you can also specify a key under which the Snowplow event's contexts will be nested. Alternatively, leaving it blank adds all entities in the request body without nesting.

Disabling this option, reveals the options so that individual entities can be selected for inclusion. Using the "Snowplow Entity Mapping" table, these entities can also be remapped to have different names in the JSON body of the request. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1` where `com_snowplowanalytics_snowplow` is the event vendor, `web_page` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

##### Include unmapped entities in request body

This option enables you to ensure that all unmapped entities (i.e. any entites not found in the "Snowplow Entity Mapping" rules above) will be included in the request body.

Again, optionally, you can also specify a key under which the Snowplow event's unmapped entities will be nested. Alternatively, leaving it blank adds the unmapped entities in the request body without nesting.

## Additional Event Mapping Options

If you wish to pick other properties from the Client event and map them into the request body, this can be specified in this section.

### Event Property Rules

#### Include common event properties

Enabled by default, this option sets whether to include the event properties from the [Common Event definition](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) in the request body. Inclusion of the `user_data` property is not affected by this setting (see next option).

Also, you can optionally specify a key under which the Common Event properties will be nested. Alternatively, leaving it blank adds the the common event properties in the request body without nesting.

#### Include common user properties

Disabled by default, this option sets whether to include the `user_data` properties from the common event definition in the request body.

Again, you can optionally specify a key under which the `user_data` properties from the common event will be nested. Alternatively, leaving it blank adds the `user_data` in the request body without nesting.

#### Additional Event Property Mapping Rules

Using this table, you can additionally specify the Property Key from the Client Event, and then the key you could like to map it to (or leave the mapped key blank to keep the same name). You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts_com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (note the array index 0) or pick non-Snowplow properties if using an alternative Client.

## Additional Request Data

This section allows you to add custom properties in the request body that are "external" to the event, in other words it provides the ability to add custom constant or variable request data.

## Post-processing

![post processing](/assets/images/post_processing-706f515d594c968ce8cfbd595f0bbd6e.png)

This section provides a way to easily configure some basic post-processing of values in the constructed HTTP request payload. The order of the subsections denotes the post-processing order. For more advanced use cases you can still use the **Additional Request Data** section above and provide values through GTM server-side variables.

### JSON Stringify

In this table you can specify the property names or paths of the HTTP request payload whose values you want to transform into JSON strings. Dot notation can also be used to denote nested paths.

### Encode base64url

In this table you can specify the property names or paths of the HTTP request payload whose values you want to encode to base64url. Encoding is only applied to string values. Dot notation can also be used to denote nested paths.

## Request Headers

Similarly to the above, this section allows you to add custom headers to the HTTP request towards your custom endpoint.

## Additional Options

Finally, this section offers two additional configuration options:

- Changing the HTTP request method from POST (default) to PUT.
- Changing the default request timeout (5000 seconds)

## Logs Settings

Through the Logs Settings you can control the logging behavior of the HTTP Request Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the HTTP Request GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "HTTP Request", // the name of the tag
    "Type": "Message",      // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",       // the "trace-id" header if exists
    "EventName": "xxx"      // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# HTTP Request Tag for GTM Server Side
> Send Snowplow events to custom HTTP endpoints from GTM Server Side using the HTTP Request Tag with flexible request configuration and authentication options.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/

The HTTP Request Tag for GTM SS allows events to be forwarded to any JSON HTTP endpoint. This Tag works best with events from the Snowplow Client, but can also work with other GTM SS Clients such as GAv4.

## Template installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “HTTP Request” and select the official “By Snowplow” tag
3. Click "Add to Workspace"
4. Accept the permissions dialog by clicking “Add”

### Manual Installation

1. Download `template.tpl` – Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As…"
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## HTTP Request Tag Setup

With the template installed, you can now add the HTTP Request Tag to your GTM SS Container.

1. From the Tag tab, select “New”, then select the HTTP Request Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to your custom destination.
3. Enter your destination URL
4. [Configure the tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/http-request-tag-for-gtm-ss/http-request-tag-configuration/) to construct the desired JSON request body
5. Click Save

---

# Forward events with Google Tag Manager Server Side
> Send Snowplow events to multiple destinations using Google Tag Manager Server Side with Snowplow-authored tags and vendor/community tags for flexible event routing.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/

To support sending events to adjacent destinations, Snowplow works with Google Tag Manager Server Side. There are both Snowplow Authored and Vendor/Community authored Tags which will allow event data to be forwarded to different destinations.

Before reading this documentation, we recommend you become familiar with [the fundamentals of Server Side tagging](https://developers.google.com/tag-platform/tag-manager/server-side/intro).

Taking an approach with Snowplow and GTM SS offers many additional flexibility and control:

- Full visibility into all transformations on the data
- Ability to evolve sophistication over time
- All data remains in your private cloud until you choose to forward it
- Ease of setup due to rich libraries of tags and familiar GTM UI

## Configuration Options

GTM SS with Snowplow can be setup in two different configurations.

![](/assets/images/gtmssoptions2-b962d1cc761a54c53f1c793e66d8ceec.png)

### Destinations Hub (Post-pipeline)

Use GTM SS to relay enriched events to destinations. Events are sent to GTM SS via Snowbridge after being processed by your Snowplow pipeline.

- For Snowplow CDI, you can [request setup](https://console.snowplowanalytics.com/destinations) through Console
- For Snowplow Self-Hosted, see [Snowbridge](/docs/api-reference/snowbridge/)

> **Note:** Destinations Hub is the recommended way to set up GTM Server-Side because it allows you to take full advantage of the Snowplow pipeline, and forward validated and enriched data to downstream destinations.

### Server Side Tag Manager (Pre-pipeline)

Use GTM SS to relay raw events before the Snowplow pipeline to destinations, including to your Snowplow pipeline. This is useful if your company uses GA but does not want to send data to Snowflake or Databricks via BigQuery.

### Principles for AWS deployment

> **Info:** GTM SS **should** be deployed into a different account to the Snowplow sub-account to maintain full segmentation of the infrastructure that Snowplow manages from that which is managed by the Snowplow customer.
>
> It would be possible to set up a separate VPC within the Snowplow sub-account but it is discouraged. VPC peering would be required to keep the traffic private otherwise traffic would go over public internet.
>
> GTM SS as a Destination Hub is normally intended to be public facing, however as a Server Side Tag Manager it could use both public internet routes and VPC peering to fan out traffic via private routes.

## Deploying Google Tag Manager Server Side

GTM SS must first be deployed before it can be used. This can easily achieved by deploying to GCP App Engine from directly within the GTM User Interface. Alternatively, docker images are available for other deployment options, such as on AWS or with Kubernetes.

- [App Engine Setup Guide](https://developers.google.com/tag-platform/tag-manager/server-side/script-user-guide)
- [Manual Setup](https://developers.google.com/tag-platform/tag-manager/server-side/manual-setup-guide)

## Snowplow Client

To receive events in your GTM SS container, the Snowplow Client must be installed. This works for both events direct from the tracker, or enriched events from the pipeline.

The Snowplow Client populates the common event data so many GTM SS tags will just work, however it also populates a set of additional properties to ensure the rich Snowplow event data is available to Tags which wish to take advantage of this, such as the Snowplow Authored Tags.

## Snowplow Tag

If using GTM SS as a Server Side Tag Manager for Snowplow JavaScript Tracker events, you will want to ensure you forward these events to your Snowplow Collector. The Snowplow Tag will automatically forward any events the Snowplow Client receives once it has been configured with your Collector URL. It can also construct Snowplow events from other GTM SS Clients such as GAv4.

## Snowplow Authored Tags

Snowplow have created a number of GTM SS Tags which work best with the Snowplow Client and make use of the rich data available from Snowplow tracker or Enriched events. See the tags for:

- [Amplitude](/docs/destinations/forwarding-events/google-tag-manager-server-side/amplitude-tag-for-gtm-ss/)
- [Braze](/docs/destinations/forwarding-events/google-tag-manager-server-side/braze-tag-for-gtm-ss/)
- [Iterable](/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/)
- [LaunchDarkly](/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/)

---

# Iterable Tag for GTM Server Side
> Forward Snowplow events to Iterable from GTM Server Side using the Iterable Tag for cross-channel marketing automation and customer engagement.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/

The Iterable Tag for GTM SS allows events to be forwarded to Iterable. This Tag works best with events from the Snowplow Client, but can also construct Iterable events from other GTM SS Clients such as GAv4.

The tag is designed to work best with Self Describing Events, and atomic events, from a Snowplow Tracker, allowing for events to automatically be converted into an Iterable events, include Iterable Identity events. Additionally, any other client event properties can be included within the event properties or user properties of the Iterable event.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

There are two methods to install the Iterable Tag.

### Tag Manager Gallery

1. From the Templates tab in GTM Server Side, click “Search Gallery” in the Tag Templates section
2. Search for “Iterable” and select the official “By Snowplow” tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking “Add”

### Manual Installation

1. Download [template.tpl](https://raw.githubusercontent.com/snowplow/snowplow-gtm-server-side-iterable-tag/main/template.tpl) - Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select “Save Link As…”
2. Create a new Tag in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

## Iterable Tag Setup

With the template installed, you can now add the Iterable Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Iterable Tag as your Tag Configuration
2. Select your desired Trigger for the events you wish to forward to Iterable
3. Enter your Iterable API Key for a Standard Server Side integration. This can be generated from Iterable "Integrations -> API Keys" settings page
4. Click Save

---

# Configure Iterable Tag for GTM Server Side
> Configure user identifiers, identity events, entity mapping, and event properties for the Iterable Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/iterable-tag-for-gtm-ss/iterable-tag-configuration/

## Iterable API Key (Required)

Set this to the API of your Iterable HTTP API Data Source.

Iterable provides four different types of API keys, each of which can access a different subset of Iterable's API endpoints. For the endpoints currently in use (`events/track` and `users/update`) the Javascript type key has enough permissions. The Mobile and Standard key types have more permissions than the Javascript type, so can also be used.

## Identity Settings

Iterable requires users to be identified to work best. The options in this section configure how you wish to identify users to Iterable based on your Snowplow events.

### Identifiers

#### Use client\_id for anonymous users

Specify whether `client_id` is used to create a placeholder email for anonymous users. This is useful for implementations where there is no identifiers for a user besides device identifiers (such as Browser Cookies).

#### email

##### Use email\_address from common user data

For Snowplow Tracking, the common user data can be populated by using the `iglu:com.google.tag-manager.server-side/user_data/jsonschema/1-0-0` context entity. This schema is available on [Iglu Central](https://github.com/snowplow/iglu-central/blob/853357452300b172ebc113d1d75d1997f595142a/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0).

This option is enabled by default. Disabling it allows for any other properties of the event to be selected for the `email` property on the Iterable event.

##### Specify email

As mentioned above, this table is revealed when disabling the "Use email\_address from common user data" configuration option. Using this table allows you to specify key paths to look for the `email` value. You can also set the search priority to denote the preference for the value to use.

The columns of this table are:

- **Search Priority**: The priority of a key path when looking for `email` (higher number means higher priority).
- **Property Name or Path**: The key path to look for in the server-side common event.

#### userId

##### Use user\_id from common user data

Iterable can also accept a User Id, rather than the preferred e-mail address. Enabling this property will use the `user_id` property from the server-side common event as the `userId` identifier of the user.

##### Specify userId

This table is revealed when disabling the "Use user\_id from common user data" configuration option. Using this table allows you to specify key paths to look for the `userId` value. You can also set the search priority to denote the preference for the value to use.

The columns of this table are:

- **Search Priority**: The priority of a key path when looking for `userId` (higher number means higher priority).
- **Property Name or Path**: The key path to look for in the server-side common event.

As an example of how Search Priority works, according to the following setup, in order to set the value for Iterable's `userId`, the Tag will first look for `user_id` in common event. If that is not found, then it will use the value of `user_data.email_address`:

![userId identifier example](/assets/images/user_id_example-c2aa0f9095a8b50674d151370535eb2b.png)

### Identity Events

#### Use the default `identify` event

Iterable allows for user information to be updated once a user has identified themselves (for example, to update their placeholder email to their real email address).

To Identify a user to Iterable, you can send a Self Describing `identify` event. This schema is [available on Iglu Central](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/identify/jsonschema/1-0-0).

For example, using the JavaScript Tracker v3, this would look like:

```javascript
window.snowplow('trackSelfDescribingEvent', {
  schema: 'iglu:com.snowplowanalytics.snowplow/identify/jsonschema/1-0-0',
  data: {
    id: '2c5ba856-ee07-47b5-a3a6-63100026ed63',
    email: 'john.doe@example.com'
  }
})
```

If you would like to specify your own event, disabling this option allows you to select your own event name and properties which can used to fire identity updates to Iterable.

#### Specify identity event(s) by event name

This multi-line text box is revealed when disabling the "Use the default `identify` event" configuration option above.

In general, "identity events" are the event names which will make the Iterable Tag call the `/users/update` [API endpoint](https://api.iterable.com/api/docs#users_updateUser) (create or update a user), using the identifiers and the user\_data specified by the tag configuration. These events might be different than the default Snowplow Identify schema, for example sign\_up, login etc, from your own custom event schemas.

## Snowplow Event Mapping Options

### Include Self Describing event

Indicates if a Snowplow Self Describing event should be in the `dataFields` object.

### Snowplow Event Context Rules

This section describes how the Iterable tag will use the context Entities attached to a Snowplow Event.

![snowplow event context rules](/assets/images/context_rules-3b86d3660566214e58f2854ec459ac3c.png)

#### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

#### Include Snowplow Entities in event properties

Using this drop-down menu you can specify whether you want to Include `All` or `None` of the Snowplow context entities in Iterable's within the Event Data fields of the Iterable event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Iterable event, and can be included in either event data or user data. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_webPage_1` where `com_snowplowanalytics_snowplow` is the event vendor, `webPage` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

#### Snowplow Entities to Add/Edit mapping

Using this table you can specify in each row a specific mapping for a particular context entity. In the columns provided you can specify:

- The Entity name to add/edit-mapping (required).¹
- The key you could like to map it to (optional: leaving the mapped key blank keeps the same name).
- Whether to add in event data fields or user data fields of the Iterable event (default value is `event data`).
- Whether you wish the mapping to apply to all versions of the entity (default value is `False`).¹

#### Snowplow Entities to Exclude

Using this table (which is only available if `Include Snowplow Entities in event properties` is set to `All`), you can specify the context entities you want to exclude from the Iterable event. In its columns you can specify:

- The Entity name (required).¹
- Whether the exclusion applies to all versions of the entity (default value is `False`).¹

> **Note:** ¹ How to specify the **Entity Name** and its relation to **Apply to all versions** option:
>
> Entity Names can be specified in 3 ways:
>
> 1. By their Iglu Schema tracking URI (e.g. `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`)
>
> 2. By their enriched name (e.g. `contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> 3. By their key in the client event object, which is the GTM SS Snowplow prefix (`x-sp-`) followed by the enriched entity name (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1`)
>
> Depending on the value set for the **Apply to all versions** column, the major version number from the 2nd and 3rd naming option above may be excluded. More specifically, this is only permitted if **Apply to all versions** is set to `True`.

**pre-v0.2.0**

#### Snowplow Event Context Rules

##### Extract entity from Array if single element

Snowplow Entities are always in Arrays, as multiple of the same entity can be attached to an event. This option will pick the single element from the array if the array only contains a single element.

##### Include all Entities in event\_properties

Leaving this option enabled ensures that all Entities on an event will be included within the Event Data of the Iterable event.

If disabling this, individual entities can be selected for inclusion. These entities can also be remapped to have different names in the Iterable event, and can be included in either event data or user data. The entity can be specified in two different formats:

- Major version match: `x-sp-contexts_com_snowplowanalytics_snowplow_webPage_1` where `com_snowplowanalytics_snowplow` is the event vendor, `webPage` is the schema name and `1` is the Major version number. `x-sp-` can also be omitted from this if desired
- Full schema match: `iglu:com.snowplowanalytics.snowplow/webPage/jsonschema/1-0-0`

##### Include unmapped entities in event\_properties

If remapping or moving some entities to User Data with the above customization, you may wish to ensure all unmapped entities are still included in the event. Enabling this option will ensure that all entities are mapped into the Iterable event.

### Additional Event Mapping Options

If you wish to map other properties from a Client event into an Iterable event they can be specified in this section.

#### Event Property Rules

##### Include common event properties

Enabling this ensures properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Iterable Event Data.

##### Additional Event Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client.

#### User Property Rules

##### Include common user properties

Enabling this ensures user\_data properties from the [Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) are automatically mapped to the Iterable Event Properties.

##### Additional User Property Mapping Rules

Specify the Property Key from the Client Event, and then the key you could like to map it to or leave the mapped key blank to keep the same name. You can use Key Path notation here (e.g. `x-sp-tp2.p` for a Snowplow events platform or `x-sp-contexts.com_snowplowanalytics_snowplow_web_page_1.0.id` for a Snowplow events page view id (in array index 0) or pick non-Snowplow properties if using an alternative Client.

## Advanced Event Settings

### Merge user dataFields

Enabling this option will merge the user dataFields when updating an Iterable user, instead of replacing with the new user data, which is the default behavior.

## Logs Settings

_(Available since v0.2.0)_

Through the Logs Settings you can control the logging behavior of the Iterable Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Iterable GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "Iterable",  // the name of the tag
    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",    // the "trace-id" header if exists
    "EventName": "xxx"   // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | “Message”                                                      |
| Request     | “RequestMethod”, “RequestUrl”, “RequestHeaders”, “RequestBody” |
| Response    | “ResponseStatusCode”, “ResponseHeaders”, “ResponseBody”        |

---

# LaunchDarkly Tag for GTM Server Side
> Forward Snowplow events to LaunchDarkly from GTM Server Side using the LaunchDarkly Tag with metric import REST API for experiment tracking.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/

The [LaunchDarkly Tag for GTM SS](https://github.com/snowplow/snowplow-gtm-server-side-launchdarkly-tag) allows events to be forwarded to LaunchDarkly using its [metric import REST API](https://docs.launchdarkly.com/home/creating-experiments/import-metric-events). This Tag works best with events from the Snowplow Client, but can also work with events from other GTM SS Clients such as GAv4.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

Coming soon!

## LaunchDarkly Tag Setup

With the template installed, you can now add the LaunchDarkly Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the LaunchDarkly Tag as your Tag Configuration.
2. Select your desired Trigger for the events you wish to use as metrics in LaunchDarkly experiments.
3. [Configure](/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/launchdarkly-tag-configuration/) the Tag.
4. Click Save.

---

# Configure LaunchDarkly Tag for GTM Server Side
> Configure metric type, authentication, context keys, and event creation time for the LaunchDarkly Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/launchdarkly-tag-for-gtm-ss/launchdarkly-tag-configuration/

This is the configuration of the LaunchDarkly Tag.

## Event Name (Required)

This is the value that identifies your metric. When you create your metric in LaunchDarkly, its Event name must exactly match this value.

## Metric Type

![](/assets/images/01-metric-type-35a667b323c01a8c56eb17ec207d2432.png)

This section allows you to select the Metric Type for this Tag. The options are:

1. **Numeric Metric** (default): Choosing this metric type reveals the [Metric Options](#metric-options) configuration section, where you must specify a Metric Value.
2. **Conversion**: Conversion Metrics should just be triggered and the metric name will be sent as a conversion.

## Authentication

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxYAAADPCAYAAABoZ0fdAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d17VFTnvf/xDwwwckcEBUFAGLyAl3ghiVhv1SSmamKtVmNizMXk9PR3fr+2Oa3V2pWY1XpO0npWe3659ZdaY6JRiVZP4l2DJire0GgkGAREoxgVhXCR2wx75veHdSKVROKGDJD3ay3Xmnn2s5/9ncE/+PA8z95eJVdKXYZhyDAMOQ1DhtOQw9Egh90uwzBUV1+vtCGDBQAAAABfxdvTBQAAAABo/wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWt7B582bNnTtXR48eNT3WxYsXNXfuXGVmZrZAZc2zd+9eLVy4UNXV1e1yfAAAALQPBItbOH78uCTp2LFj3/jc119/Xc8++2xLl/S15s2bpzfffNP9PiAgQJ07d5bFYjE99pkzZzR37lzt2bOnVcYHAABA++Xj6QJu5cyZMyooKNA999zT5PEdO3YoOTlZCQkJLX7ts2fPqqysTBEREcrPz1dtba38/f1b/DqtafDgwRo8eHC7HR8AAADtg+VXc3+90OVy6cZ/TqdTTsOQy+VSg2Eopnu0xwr8+9//rkOHDskwDNlstkbHtmzZoszMTFVVVWnQoEEtfu3du3fr/PnzmjFjho4cOaLIyEjFxMS4jz/77LM6deqU+xfrXbt26bXXXlNqaqr++Mc/qqSkRA0NDe7w4+Pjo/379ysyMlJZWVlav369cnNzlZiYqMDAQElSXV2d1q1bp4yMDGVlZamsrExJSUmyWCzat2+fXnrpJfn7+ysjI0NbtmzRuXPn1LdvX5WUlOh3v/udXC6XLl++rMzMTI0bN04bN27UkiVLNHToUPn7+6uhoUEbNmxQRkaGdu7cqeLiYvXs2VNWq1WSdPToUa1cuVIbN27URx99pKCgIEVFRWnPnj16++23JUn5+fk6ffq0hgwZctP4LpdL77//vlavXq2tW7fq1KlTio2NVVBQkCRp4cKFOnPmjPLz87Vq1SodOHBAnTt3VteuXVv85wcAAIBvT5tfCjVr1iwlJCRo586d2rp1q7t9y5Yt2rVrlxISEvTII4+0yrWPHz8um82m3r17KyQk5Bsth3rooYcUHR0tX19fzZ49u9EvzocOHVJkZKTS0tJUXFysjRs3uo+99dZbysnJ0YgRI3THHXfowIEDjT63JO3cuVMDBgyQzWZTbm6uDhw4oPDwcM2ePVteXl7q0aOHHn300Sbr2rp1q7KystSvXz+NGDFC+fn5WrFihSSpqKhIq1atUlhYmKZMmaLAwECtWrVKJSUlSklJ0f333y9JSktL03333dfk+Nu2bdOOHTvUs2dPjR07VhcuXNCSJUtkt9vdfU6cOKGamhqNGjVKdrtda9askWEYzf5uAQAA0Pa0+aVQfn5+mjNnjpYsWaKdO3dKkpxOpz744AMlJCRozpw58vPza/Hrnj59WhUVFRo3bpwkKSUlRYcOHVJ1dbV7duHr9O3bV3v27NEXX3yh1NRUSVJlZaUkacSIEZowYYKka8utiouLJUnFxcUqLCzUhAkTNGzYMElSWVmZjhw5okmTJrnHnjp1qlJSUuRwOPTb3/5WxcXFGjVqlFJTU+Xl5aWQkBClpKTcVJNhGNq3b59SU1P1ox/9SJLk6+urPXv2qKamRqGhoZo9e7YSExPl7++v8PBw/eUvf9GZM2d05513qmfPnpKkqKgoxcfH3zR+Q0ODsrKylJSUpJkzZ0qSYmJitHTpUh0+fFjp6enutscee8x9zq5du1RaWsqsBQAAQDvW5oOF1HS4aM1QIX25Wbtbt24qLS1VbGysDhw4oOPHj7t/6b9dnTp1avS6vr5ekvT5559LkjZt2qRNmzY1Oud6H+nahmnpWiiwWCyNZgO+TllZmRoaGhot5xo9erRGjx7tHreoqEhLlixRWVmZe9wbr32r8evr690BRJKSkpIkSefPn3e33bhP5fp30dzPAAAAgLapXQQLqXG4kNSqocLpdConJ0eS9OqrrzY69vHHH5sOFrcyfvx4JSYmNmrz9fU1Pa6399evfMvJydGaNWuUnp6uqVOn6sqVK1q+fHmzx/fxufbfycvL66ZjTqfzmxULAACAdqXdBAvpy3Bx/XVrOXXqlK5evaphw4Y12jCenZ2tkydPqrKyUiEhIbJaraqrq3Mf/+e/7Ht5ecnlcjX7ulFRUZKk8vJy912uampqVFtbe8tQcKOvumZYWJh8fHzcMyPStZmZo0ePavr06SosLJQk3XPPPQoMDFRVVdU3Ht9qter06dPutuuvo6M9dwMAAAAAtL52FSyk1g0U111fBjV69Gh17tzZ3e7l5aW8vDx9/PHHGjFihGJjY3XixAlt3LhRDQ0NOnjwYKNxgoODVV9fr82bN+uuu+665XXj4uJks9l08OBBOZ1ORUZG6tChQwoNDdXTTz/drNpDQkJ05swZZWZmasyYMY2OWSwWDR8+XLt379a6desUGhqq3bt3KzIyUgEBAYqMjJQkrVu3TrGxsdq7d6+kL2cbQkJC3N9PQECAhg4d2mh8b29vjRw5Ujt27NDKlSsVFRWlrKwsBQQEaMiQIc2qHwAAAO1Tm78r1LfNMAx98sknioqKahQqJKlXr17y8fHRxx9/LEmaOHGi4uLidODAARUXF98UHkaNGqWIiAjt37/fvXH7Vh555BENGjRIOTk52r59u7p06eLeaN0c48ePl3TtidhNzSzcd999Sk9P1/Hjx7Vr1y7ZbDb3HaSGDRumu+++W4WFhTpw4IBGjBgh6dreCUkKDw9Xenq6SkpK9MknnzR5/bFjx2rMmDEqLCxUZmamunbtqp/85CfN2vAOAACA9sur5EqpyzAMGYYhp2HIcBpyOBrksNtlGIbq6uuVNoQHoAEAAAD4asxYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEzz8XQBaH8qKyt1Mr9QFRUV6uTvr54JcYrp3t3TZQEAAMCDCBZotsqqKq3KWKsDB7NlsVgUEhKsmppa1dXVqWdCvGY9PEOJPRM8XSYAAAA8gGCBZim5fFkv/vHPslgs+ulP5mjQwAHy8fGRy+VS0ekzWv/uBv3Hi/+lf336CQ0ZPMjT5QIAAOBbZvnV3F8vdLlcuvGf0+mU0zDkcrnUYBiK6R7t6TpbldPp1K9//Wvt2LFDO3bs0O7du3Xq1CnFxsYqKCjoG4+3ePFiJScnKzAw8LbqMQxDmZmZSkhIkLf3zdtgXn31VVksFkVHX/u5nDx5Uq+88opSU1Nv+5pfp6GhQf/5xz8pMCBAv5n37+oaGaGly5Zr6ZvLlX34Iw0c0F8T7r9PVVevat3/vKdBdwxUSEhwi9cBAACAtovN2zeYP3++XnjhBS1YsEBxcXFatmzZTX1cLtctx/nxj3+s8PDw267DMAzt2LFDhmHcsu/Fixe1atUqPfzww4qMjLzta36dXR/uUWlpqf7P//qJAgMCtGJlhvILTunHP/qhQkND9OeXXlVdXb0enjFNPWJj9c7a9a1SBwAAANoulkLdwNvbW97e3urUqZPGjBmjzMxM1dTU6NixYzp27Jjq6+sVHR2tGTNm6LPPPtP69etVWlqq+Ph4TZ06VWFhYZKk5cuX6/HHH1f37t3ldDq1adMmZWdnKygoSJMmTVLfvn0lSfn5+XrvvfdUUVGhpKQkTZs2TZWVlfrrX/8qSXrhhRf05JNPKiYmpsl6q6qqtHTpUt1///2y2Wzu9sOHD2vbtm1yOp1KT0/X2LFjdfbsWS1ZskTPPfecLBaLCgoKtHbtWs2fP/+W38uBg9lKH3a3One+9vnOf35BD02fqjvThiht6BD9289/qXPFxeqVbNOE++/Ty6+9rurq6laZPQEAAEDbxIxFE+x2u/bt26fQ0FAFBARIkkpKSjR16lRNmzZNtbW1euONNzRmzBg999xzio+P11tvvdXkWLt379b58+c1f/58TZ8+XRkZGaqtrVV1dbVWrFihSZMm6bnnnlNoaKg2btyo6OhozZs3T5I0b968rwwVDodDy5Yt04ABA3TXXXe528+ePavNmzfrqaee0s9+9jMdOXJEhYWFiouLk9VqVVFRkSQpLy9Pqampzfo+is+fV7It0f3++Wd/ozvThkiS9uzdp+CgIMXGXLsrlM2WKKfTqQsXLjVrbAAAAHQMzFjcYNGiRZIkPz8/xcTEaPbs2e5jycnJ6tGjh6Rrv5RHRkZq4MCBkqSxY8dq3759unTpkrp169ZozCNHjmjixIny9/dXfHy8YmJiVFRUJLvdrpiYGPXu3VuSdM899ygvL6/ZtW7btk319fVKSkpq1P7RRx9p6NCh6tq1qyRp6NCh+vTTT2Wz2ZSamqrc3FwlJycrLy9PU6ZMada1GhwN8vP1u6l94+atenfDZs3995+5A5if37V+doe92Z8FAAAA7R/B4gYLFixQaGjoLftdvXpVnTt3dr/39vZWaGioysvLbwoWlZWVWrFihby8vCRd2z8xYMAAORwOhYSEuPsFBQVp6NChza41KipKkydP1ssvv6xevXq5l0JVVFQoPz9fBw4ckHRtT8j1mYl+/frpnXfe0ciRI1VdXa2ePXs261phncN08dLNMxCbtmzXg5N+oGTbl+Hm0sVr/cLDO9/UHwAAAB0XweI2hIWFKScnx/3e6XSqoqKiyTtIBQcHa9q0aYqPj2/UnpOToxMnTrjf2+12Xbhw4aZ+X+X6rMTkyZO1evVq/eIXv1BgYKBCQkI0btw4jRkz5qZzEhMT5XA4tHPnTvXp06fJO041ZUC/VO0/cEgT7r/PHZBcLpcemzVTvXslN+qbtf+gIiK6KOqfAhYAAAA6NvZY3IZevXrpypUr+uSTT2QYhj788EMFBQU1uR9i4MCB2r59u2pqalRTU6MNGzaooqJCNptNn3/+uQoLC+V0OrVjxw5lZWVJknx8fOTt7a3S0lI5nc6vrWXw4MFKSkrSmjVrJEkDBgxwL8tyOBzau3ev8vPzJV2bWUlJSVF2dnaz91dI0vh7x+nipRJt2LTV3eZ0uvTa63/Tmc/OutsKC4u084PdmviD8c0eGwAAAB0DMxa3wWq16oknntC6deuUkZGhmJgYPfbYY032HTNmjDZv3qzFixfL6XQqLS3Nvdzq0Ucf1bvvvquysjL16tVL06dPl3QtANx11116+eWX9fTTT99yFuOHP/yh/vznP2vfvn1KT0/XuHHj9MYbb6i6ulo2m02DBw929+3Xr5+OHj3q3tvRHN26ddWsh2fozeUrZXfYNXnSBPn4+GjZktfcfY58dFRL3nhLdwzsr1Ejhjd7bAAAAHQMXiVXSl2GYcgwDDkNQ4bTkMPRIIfdLsMwVFdfr7Qhg289EtwWLVqkJ554wv0Au7akqKhIH374oR5//PFvfO7erP1avnK1/P39NWjgAIWHh6umtkYnTuTp7LlijR75PT0yc7osFksrVA4AAIC2jBmLFlZRUaGamhr5+/t7upSb1NTU6IMPPtCgQYNu6/zvDR+m/v1S9cHuPfo0L18n8wvUyd9ftqREPfbow+qZ0Lz9IQAAAOh4CBYtqKKiQn/4wx/Uv39/98Py2pJFixYpJSXFfZvc2xEaGqIHJ03Qg5MmtGBlAAAAaO9YCgUAAADANO4KBQAAAMA0ggUAAAAA0wgWAAAAAExj8/Y/1NfbVVpWruqaWk+XAqANCwzwV5fwMFmtfp4uBQCANoVgoWuh4mzxBUVGhKt7dFdPlwN85xWc+kzJSW3z9sXlFVU6W3xBcbHRhAsAAG7AUihJpWXliowIV1hosKdLAdDGhYUGKzIiXKVl5Z4uBQCANoVgIam6ppZQAaDZwkKDWTYJAMA/IVgAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANN8PF0AAJi1du1aORwO9/tp06bJx8dHdXV1Wr9+vbvdarVqypQpkqTs7GwVFha6j6Wlpclms317RQMA0MEQLAC0e/X19bLb7Te1u1wu1dXVNXlOQ0NDo2OGYbRafQAAfBcQLAC0S5WVlaqpqZF0LUDc6NKlS7JYLKqvr2/U7nQ6dfHiRUlyn3tdRUWF+1i3bt3k5eXVWqUDANAhESwAtEs5OTnKz89v8tj27dubbLfb7dqyZUuTx3Jzc5WbmytJmjlzpqxWa8sUCgDAdwSbtwF0eMHBwfLx4e8oAAC0JoIFgA5v1KhRioiI8HQZAAB0aAQLAAAAAKaxNgBAh2SxWNyvvby8ZLFY3G1Op/OmDd8AAMAcggWADumBBx5wb8C2Wq0aPXq0+5ayhw4dUlFRkSfLAwCgwyFYAOiQbnww3sSJE3X48GH37WQBAEDLY48FAAAAANMIFgAAAABMYykUgA5vy5Ytcjqdni4DAIAOjWABoF3q3bu3unfv3ipj+/r6tsq4AAB0ZAQLAO1SREQED70DAKANYY8FAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWAhKTDAX+UVVZ4uA0A7UV5RpcAAf0+XAQBAm0KwkNQlPEyXr5QRLgDcUnlFlS5fKVOX8DBPlwIAQJvCk7clWa1+iouNVmlZuS5fKfN0OQAkFZz6zNMlNCkwwF9xsdGyWv08XQoAAG0KweIfrFY/dY/u6ukyAAAAgHaJpVAAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA0wgWAAAAAEwjWAAAAAAwjWABAAAAwDSCBQAAAADTCBYAAAAATCNYAAAAADCNYAEAAADANIIFAAAAANMIFgAAAABMI1gAAAAAMI1gAQAAAMA0ggUAAAAA03w8XQDQXPX19TqZX6DLV0rlY7EoOjpKtqREeXuTjwEAADyNYIE2zzCc2rBps7Zs2yG73aHQ0BAZDYaqrl5V57AwTZs6Wel33+XpMgEAAL7TCBZo0xwOh/70f19V0ekz+uGDkzRi+DAFBgZKkq5cKdXW7e/rr397U+fOndf0aVM8XC0AAMB3F8FCktPp1Lx5825qnzFjhgYPHnxbY77yyiuaMGGCEhISTFb37Tp48KASExMVGRl507Ft27apqqpKU6dOlSSVl5fr5Zdf1sSJE3XHHXe0Sj1vvb1an509p9/O+6ViYrpr/bsbtevD3fLz9dPkByfqkZnT1SvZptde/5uio7pp5IjhrVIHAAAAvh7B4gYLFixQaGhoi4z14IMPqlu3bi0y1rfp4MGDCgkJaTJY3Ki+vl5Lly7V3Xff3Wqh4uy5Yu3N2q///dN/UWxsjHbvydKmLds0+YGJqqqq0tJlyxXdrZvuTBuiM5+d1Zp1/6O770qTn59fq9QDAACAr0awuIWLFy9q2bJlGjhwoA4ePKigoCA9/PDDio6O1sqVKxUbG6uRI0dKkpYvX67ExEQNHz5cGRkZmjx5spKSkrRw4UL16dNHubm5+vnPf66wsDBt3rxZhw8fVqdOnTRixAh973vfkyTt27dPp06dktPpVGFhoeLi4jRr1ix16tRJe/bsUUFBgQzD0JkzZ5SQkKDx48drzZo1KisrU0pKimbMmCFvb285nU5t2rRJ2dnZCgoK0qRJk9S3b19J0sKFC/X9739fWVlZstvteuCBBzRo0CC98MILKi8v1+rVq3XfffcpPT29ye/E6XTq7bffVnR0tMaNG+duv3r1qt555x2dPn1a0dHRmj59usLCwvT888/rqaeeUo8ePWS327Vw4UL96le/UufOnb/2uz946LCiunXT4EEDJUmnPzurcWNHa9KE8ZKkE5/m6UTeSdlsifrB+Hu1dfv7yj2Rp0F3DDD3QwcAAMA3xu10mqGsrEyhoaFasGCBkpKSlJmZKUnq37+/Tpw4IUkyDEMFBQXq169fk2P4+/tr/vz5Cg8P165du3Tu3DnNnTtXc+bM0Z49e/Tpp5+6+548eVKjRo3Sb37zG9XV1eno0aPuY+fOndOECRM0f/58Xb16VW+//bZmz56tuXPn6ty5c8rLy5Mk7d69W+fPn9f8+fM1ffp0ZWRkqLa21j3O+fPn9cwzz+jBBx/Ue++9J0maN2+eunfvrhkzZnxlqJCk9957T7W1tZo2bVqj9oyMDHXt2lXPPfec+vfvr4yMDFksFneokqSCggJFRkbeMlRIUvH580pOTnK/n/3IQ3rox9eWYeUXFKrk8mX16Z0sSQoKClRUt64qPv/5LccFAABAyyNY3GDRokWaO3eu5s6dqz/96U/u9qCgIKWnp8vX11d9+vRRWVmZJKlPnz76/PPPVVNTo9OnT6tr165fuZRq2LBhCggIkJeXlz766CONGzdOgYGBioyM1IgRI3T48GF33759+yohIUH+/v5KTExUaWmp+5jNZlP37t0VFBSk5ORkpaSkqEuXLgoJCVF8fLy775EjRzRmzBj5+/srPj5eMTExKioqco8zduxYWa1WpaSkqLq6Wna7vVnf0YkTJ5SdnS1JjW7zWl1drYKCAt17773y8fHR8OHDVVxcrLq6ukYBLC8vT6mpqc26lsPRID9f35va807m6w//9d+aMvkB9Uq2udv9/PzkcDTvcwAAAKBlsRTqBs3ZY+Ht7S3DMCRJvr6+Sk5OVl5enoqLizVgQPOW4Fy9erXRX+w7d+6sY8eONdnXy8vLfb2mavnnvk6nU5JUWVmpFStWyMvLS9K1GZWm6rs+xlddo6n+zzzzjFasWKH3339f9957r/t6LpdLv//97xv1r6qqUu/evbV69WqVlZXp5MmTmj17drOuFd45TBcvXbqpffuOnerdK1nj7/1yGZbT6dTly1cUHh7erLEBAADQsggWJg0YMEDHjx/XhQsX9NRTTzXrnNDQUJWXl7s3SH/xxRcKCgpq0bqCg4M1bdo0xcfHt+i4ffr0UZcuXTRz5ky99NJLSk5OVs+ePRUcHCwfHx8tXLiwyQfW9erVSzt37pQkxcTENOta/ful6q9L31RZ2RcKD/8yiI0bO/qm7+vY8RzV1NYqNaWPiU8HAACA28VSKJP69u2rgoICderUqVn7BiRp8ODByszMVG1trb744gtlZWVp0KBBLVrXwIEDtX37dtXU1KimpkYbNmxQRUXFLc/z8/NTaWnpLWcwIiMjNXHiRK1atUq1tbUKCgpSXFyctm7dKsMwVFJSovXr18vlckmS+vXrp+zsbKWkpDT7MwwdMlhduoTrjTdXuGdiJGn9uxu1b/9B9/uqq1e1cvUa3ZU2VJEREc0eHwAAAC2HGYsbLFq0qNH7KVOm3PI5FH5+frLZbOrRo0ezrzNy5EhVV1dr8eLFcrlcGjlypAYOHHg7JX+lMWPGaPPmzVq8eLGcTqfS0tKadSvdO++8U+vXr5dhGBo1atQt+548eVJr167VrFmzNGPGDK1bt07PP/+8/P39NX78ePdSrJSUFHl7ezd7f4UkWSze+um/zNF/vLhY//3ya3ry8UcVEhysBfN+6e5TXHxer/6/JbJ4e2vWw9ObPTYAAABallfJlVKXYRgyDENOw5DhNORwNMhht8swDNXV1yttyO09JA64zuFw6MUXX9T8+fNlsVi+0bmnz3yml197XVevXtUdAwcoJjpaDYah06fP6JMTn8qWlKh/+9enFRoa0krVAwAA4FYIFmh1DodDH3zwgSoqKtxP7b6dMfZm7dfHxz/RldIyWSzeio6K0p1pQzTojgHumREAAAB4BsECrW7JkiWqqqrSk08+qZAQZhUAAAA6IvZYoNXNmTPH0yUAAACglXFXKAAAAACmESwAAAAAmEawAAAAAGAaeyz+ob7ertKyclXX1Hq6FABtWGCAv7qEh8lq9fN0KQAAtCkEC10LFWeLLygyIlzdo7t6uhzgO6/g1GdKTor3dBlNKq+o0tniC4qLjSZcAABwA5ZCSSotK1dkRLjCQoM9XQqANi4sNFiREeEqLSv3dCkAALQpBAtJ1TW1hAoAzRYWGsyySQAA/gnBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkECwAAAACmESwAAAAAmEawAAAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACY5uPpAgDArLVr18rhcLjfT5s2TT4+Pqqrq9P69evd7VarVVOmTJEkZWdnq7Cw0H0sLS1NNpvt2ysaAIAOhmABoN2rr6+X3W6/qd3lcqmurq7JcxoaGhodMwyj1eoDAOC7gGABoF2qrKxUTU2NpGsB4kaXLl2SxWJRfX19o3an06mLFy9Kkvvc6yoqKtzHunXrJi8vr9YqHQCADolgAaBdysnJUX5+fpPHtm/f3mS73W7Xli1bmjyWm5ur3NxcSdLMmTNltVpbplAAAL4j2LwNoMMLDg6Wjw9/RwEAoDURLAB0eKNGjVJERISnywAAoEMjWAAAAAAwjbUBADoki8Xifu3l5SWLxeJuczqdN234BgAA5hAsAHRIDzzwgHsDttVq1ejRo923lD106JCKioo8WR4AAB0OwQJAh3Tjg/EmTpyow4cPu28nCwAAWh57LAAAAACYRrAAAAAAYBpLoQB0eFu2bJHT6fR0GQAAdGgECwDtUu/evdW9e/dWGdvX17dVxgUAoCMjWABolyIiInjoHQAAbQh7LAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAVmTd6QAAATNJREFUAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkEC0mBAf4qr6jydBkA2onyiioFBvh7ugwAANoUgoWkLuFhunyljHAB4JbKK6p0+UqZuoSHeboUAADaFJ68Lclq9VNcbLRKy8p1+UqZp8sBIKng1GeeLqFJgQH+iouNltXq5+lSAABoUwgW/2C1+ql7dFdPlwEAAAC0SyyFAgAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABgGsECAAAAgGkECwAAAACmESwAAAAAmEawAAAAAGAawQIAAACAaQQLAAAAAKYRLAAAAACYRrAAAAAAYBrBAgAAAIBpBAsAAAAAphEsAAAAAJhGsAAAAABg2v8Heg6qijPgOdIAAAAASUVORK5CYII=)

### Project Key (Required)

In this text box you need to provide the key for the **project** your metric events pertain to. You can find it under Environments on the Projects tab on your LaunchDarkly Account settings page.

### Environment Key (Required)

In this text box you need to provide the key for the **environment** your metric events pertain to. You can find it under Environments on the Projects tab on your LaunchDarkly Account settings page.

## Authorization

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAuUAAACDCAYAAAAqCGWQAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABUtSURBVHic7d17cFRlnsbxp9OdNLk3uRIISSABhkDIoMRwWS4RcIRBQATB8YLrouWKVWs5VSg7IzBbs7Wu5U7V6q7FzGJhMeIYhhGMGBAFgnInXEIIkCCEYEK4hnTu6eR07x9MjkTCACN4DPl+qlJJ3tPnfX+nwx9PXn7nxHbZXePzer2SJK/XK5/PJ8MwZBiGvIYhw2uopaVVLR6PDMNQU3OzMu69RwAAAABuDz+rCwAAAAC6OkI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMUI5AAAAYDFC+S3Kzc3VggULdODAge8919mzZ7VgwQJt2rTpNlT2rRUrVuidd965rXO2+eabb/Taa6+puLj4jswPAADQFRHKb9GhQ4ckSQcPHrzlc//whz9o0aJFt7uka4SFhcnlct2Wub5bc0BAgLp37y6n03lb5gcAAIDksLqAW3Xq1CkdP35cEydO7PD4559/rn79+ikpKem2r3369GlVVVUpKipKJSUlamxsVGBg4G1f5+/l9Xrl5+en6dOn37E1YmNj9fLLL9+x+QEAALqiThfKN23apOLiYhmGoQcffLDdsfXr12vLli365ptv9Mwzz9z2tQsKCmS32zVt2jS9++67Onz4sDIyMszjixYtUkJCgubNmydJ2rJli9avX6+XXnpJS5cuVVNTkyRpwYIFeuGFF9StWzdJktvt1rJly1RWVqaYmBjNmTNH0dHRkqSGhgbl5OToyJEjkqRBgwZp2rRp6tatm86fP68333xTQ4cOVUVFhVwul+bNm6e33npL9fX1WrhwofLz87Vq1aprrmX69OkaOXKk6urq9Mknn+jYsWOSpP79++vhhx9WUFCQFi1adE3NjY2NWr58uR599FENGzZMklRZWamcnBydPn1aQUFBGj58uMaPHy9J2rFjh9auXaupU6dq586dcrvd6tevn+bMmcNuOwAAwF91uvaVJ598UklJSdq8ebM2bNhgjrcF8qSkJD3xxBN3ZO1Dhw4pJSVFAwYMUFhY2C21sDz22GOKi4uTv7+/5s6dq5iYGPPYnj17FB0drYyMDJWXl2vdunXmsffee08FBQUaPny4MjIydODAAWVnZ7eb+8CBA4qMjNTAgQOvWTchIUHTp0/X9OnTNW3aNAUFBclut6tPnz6SpJUrV6qoqEjjx4/XuHHjVFhYqI8//viGNbepq6vT0qVLdfHiRU2YMEG9e/fWZ599pq+++qrd6zZv3qwhQ4YoJSVFRUVF2rVr102/dwAAAHe7TrdTHhAQoHnz5mnZsmXavHmzpCttG3l5eUpKStK8efMUEBBw29ctLS2V2+3WhAkTJEmpqanas2eP6uvrFRwcfMPzBw4cqK+++kqXL1/WoEGDJEk1NTWSpNGjR+vnP/+5pCstMuXl5ZKksrIynTp1ShMnTjTbdfz8/LR161adO3dONptNkjRs2DA9+uijHa4bExNjhult27apoaFBkydPVlxcnCRpzJgxCgoKUmJioiSppKREJ0+evG7N37Vnzx41NjZq7ty56tu3ryTp7bff1ubNmzV69GjzdTNnzlRqaqpaWlr061//2rxGAAAAdMKdcunbYN62Y36nA7n07Y2dsbGxunTpkuLj4+X1es0bP7+PtjaWtq+bm5slSWfOnJEkc1dbklJSUiRJFRUV5lh4ePgN1zh//rxyc3OVlJSksWPHmuM9evTQrl279Prrr+u1115TaWmpuf7NqKyslM1ma9fDn5ycrPr6elVXV5tjQUFBkiR/f3/Z7XZ5PJ6bXgMAAOBu1+l2yttcvWMu6Y4Gcq/Xq8LCQkm65lGDBQUFGjFixB1Z1+G48uNp2xFvq+XqzzfD6/Xqww8/lN1u15w5c9rNt3z5cjU2NmrGjBmKiorS6tWrVVlZedNz2+32dvNJks/na/cZAAAAf1unDeXSt8G87es75cSJE6qrq9OIESPMnWpJ2rt3r4qLi1VTU6OwsDA5nU7zxkhJ1+w422y2WwqqsbGxkq60ziQnJ0u68vQZ6coO98364osvVF5erlmzZikiIsIcr6ur09mzZzVy5EizH90wjFuquUePHvJ6vSorKzN39EtLSxUYGHhTO/gAAADo5KFcurNhvE1b68q4cePUvXt3c9xms+nYsWMqKCjQ6NGjFR8fryNHjmjdunVqbW3V7t27280TGhqq5uZm5ebmKjMz84brJiQkKDk5WZs2bZLH45HP59O2bdvUt29fxcfH6/z58zeco7y8XJs3b1ZYWJgMwzBvsIyLi1Pv3r0VHBysgwcPyuVyqaKiQqdPn273nt6o5szMTOXl5emDDz7QyJEjzTkmTpwoP79O2R0FAADwgyM13YBhGDp8+LB69OjRLpBLVx4f6HA4VFBQIEmaMmWKEhIStGvXLpWXl18TYseOHauoqCjt3LnTvMnzRh5//HENHjxYO3fu1J49e5Senq65c+fedP2lpaXyer2qqanRRx99ZH4UFBTIz89PTz31lMLCwrRp0ya1trZq8ODB8ng8qq+vv6maAwMDNW/ePHXv3l0bN25UWVmZHnjgAfOGWAAAANyY7bK7xnd1n7LP55NhGDIMQ17DkOE11NLSqhaPR4ZhqKm5WRn33mNx2QAAAMDdg51yAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBiDqsLQNfl9Xp14mSpzlSeVWtrq6IiIzVgQD91czqtLg0AAOAHRSiHJXbvzdeqP6/RpaoqhYQEy+FwyO2ukb+/v342cbymPTRZDgf/PAEAQNdA6unilixZohdeeEExMTE/2Jp/WZOjdbkbNG7saE362QTFREdLkuobGrR9+y6tyVmnkuNf65cvvaiAgIAfrC4AAACrEMpv4J133lFcXJwefvhhq0u5ri1btmj9+vUdHnvllVcUGRn5A1d0fTt27da63A169p/mauTwTO3bf0Bv/u5tNTQ2aOTwTM2eNUODB6fqP974L723YqWem/ePVpcMAABwxxHK/4bLly/r3LlzunDhgqZOnSq73W51SR3KyspSVlaWJGnZsmUaMmSI7rvvPourulZra6tW/XmNJo7P0sjhmao8e1bv/P5djRyRqV4947T243UKCwvVlMkP6tlnntbv/vt/NGF8lvr2SbK6dAAAgDuKUP437N+/X2lpaTp37pyOHTumQYMGmceqqqq0atUqlZeXKyYmRo888oh69eolSdq6dau2bt0qr9erYcOGacqUKZKkuro6rVq1SqWlpYqLi9Ps2bPNXezdu3fr888/V0tLi9LS0jRjxgz5+fmptrZW2dnZKisrk8vl0owZM9SnT59buo6ysjKtWbNGly5dUmJiombOnCmXy9XuNT6fT++9955CQ0M1c+ZMeb1effrpp9q7d69CQkL00EMPaeDAgZKutLzcf//92r59uzwej6ZOnaqhQ4fesI6jxSVy19Ro8qQHJEknTpQquW8fPTP3CdlsNl28eElHjhZryuQHNSRtkHr3jtfuPfmEcgAAcNfjkYh/w4EDB5SWlqYhQ4Zo//797Y6tXLlSycnJWrJkiUaPHq0VK1ZIkoqLi7V9+3bNnz9fCxYs0KlTp5Sfny9Jys7OVkxMjBYvXqy0tDRlZ2dLktxutz799FPNnz9fr776qs6cOWOe89lnnykmJka/+c1vlJWVpQ8++OCWrqGxsVHLly9XVlaWFi9erMTERLPWq+Xk5Mjj8ZhtOl9++aUqKiq0cOFCzZ49W9nZ2WpsbDRfX1FRoZdfflnTpk1TTk7OTdVSXl6h6KgoucLDJUn/MGqE/vWVX8pms6m62q3CoiMa+JP+5uv7JfdVecWZW7peAACAzohQfh3l5eWqra1VSkqK0tLSdPToUTOUut1uVVZW6v7775fD4dDQoUM1ZswYtbS0qLCwUPfdd58iIyMVFBSk6dOnKzw8XPX19Tp+/LgeeOABORwOjRo1SuXl5WpqapLdbpfP59P58+fldDr13HPPaciQIZIkf39/ud1u1dXV6Z577tGLL754S9dx7NgxRUdHKz09XQ6HQ+PHjzfbctrs3r1bxcXFeuqpp8wWnX379ikrK0uBgYFKTExUr169dPLkSfOc8ePHy+l0KjU1VfX19fJ4PDespaWlVQEB/teM19TW6t/+/T/Vq2ecpkx+0BwPCAhQS0vLLV0vAABAZ0T7ynXs379fqampstvtcrlciouL06FDh5SZmamGhgYFBwe36zEfNWqUJKm+vl4JCQnmeHx8vCSpsrJSPp9Pv/3tb9utU1tbq+joaM2dO1dffPGF/vSnPyk9Pd1seZk8ebI2bNigt956S2FhYZo0aZLC/7rTfDPq6urUvXt383s/Pz+Fh4erurpasbGxkqRdu3YpKCio3Xk1NTV6//33ZbPZJEmGYZi/KFzNz8/PPH4jEd1dunjxkgzDaPfe7dq1V03NTXr+2WfM9STp7LnziriqdgAAgLsVobwDXq9XBw8eVF1dnfbt22eO2+12ZWZmKiwsTPX19e3C5cmTJ5WQkKDw8HDV1NSY51RXV8vj8Sg0NFQOh0NLliwxg2yburo6RURE6Pnnn1dDQ4P++Mc/aufOnRozZowuXLigSZMm6aGHHlJRUZHef/99LV68+Jo5rsflcqmwsLDdtbndboWEhJhjTz75pPLz85WTk6PZs2dLkkJDQzVr1iwlJibe+ht4HYNSB6rZ49G+/Qd1X8a95njqwAHq2TOu3eMP3e4aHTl6VE89/thtWx8AAODHivaVDpSUlMhms+n111/XG2+8oTfeeEO/+tWvdPr0aVVVVSk4OFgJCQnKy8uT1+vVkSNH9OGHH8rPz09paWnau3evqqqq1NTUpOzsbJ08eVIhISFKSEjQhg0bZBiGzp8/rzVr1sjn8+nMmTNaunSp3G63nE6nunXrptbWVknSmjVrtHXrVvl8PoWFham1tVU+n++mr6V///66ePGiDh8+LMMwtHXrVoWEhJg3pUpSRESEZsyYoZKSEhUVFUmS0tPTtXHjRjU0NKihoUGffPKJ3G7393pfu3d3adTI4fpw1V/kdn/7i8vuvfu0+qO15vc+n0/LV6xUWFiYhmdmfK81AQAAOgP7qwsXLmkLeVd/vvrD6/XKaxjy+XxqNQz16hlnZc133MaNG5WSkqJ+/fqZY926dVNFRYXq6+vVt29fpaSkaMeOHfr4449VWVmpp59+WiEhIYqIiJDdbtfq1auVl5en9PR0jR07VpLUr18/5efna+3atSosLNSwYcMUFxenyMhI+Xw+ZWdnKy8vT9HR0Zo0aZLsdrv69++vvLw85eTk6Pjx43rkkUfMtpOO7N+/X7GxsWbodjgcSk5OVm5urnJzc+XxePT444+b7Sp5eXnKyMiQy+VSTEyMVq9erWHDhiklJUWVlZX66KOPtG3bNsXHxystLa3dOcHBwfJ6vdq0aZOysrLk739tv/h3DfxJf23fsVtfbt+hAf37KTw8TKkDB2jc2NGSpNq6Ov1+2XIdLjqql/9lviIjI/6+HyIAAEAnYrvsrvF5vV5JV1obfD6fDMOQYRjyGoYMr6GWlla1eDwyDENNzc3KuPcei8tGZ1ZbW6f/Xfp/Ki45rtSBP1Fy3yQ5HP46U1mpgwWFCgoM1Px/flbJfW/t0Y8AAACdFaEcljlYcEi79+7TmcqzMlpbFRkZoSFpgzV61MgOn9ICAABwt+JGT1jmp+lD9NP0a5/oAgAA0NVwoycAAABgMUI5AAAAYDFCOQAAAGAxQjkAAABgMW70vI7mZo8uVVWrvqHR6lIA/IgFBwUqMsIlpzPgxi8GAOA6COUdaG726HR5paKjItQzLsbqcoAu7/iJMvVLTrS6jA5Vu2t1urxSCfFxBHMAwN+N9pUOXKqqVnRUhFzhoVaXAuBHzhUequioCF2qqra6FABAJ0Yo70B9QyOBHMBNc4WH0uoGAPheCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUI5QAAAIDFCOUAAACAxQjlAAAAgMUcVhcAALfb6tWr1dLSYn4/a9YsORwONTU1ac2aNea40+nUjBkzJEl79+7V119/bR7LyMhQSkrKD1c0AKBLI5QDuOs0NzfL4/FcM+7z+dTU1NThOa2tre2OGYZxx+oDAOC7COUA7go1NTVqaGiQdCV8X+3cuXOy2+1qbm5uN+71enX27FlJMs9t43a7zWOxsbGy2Wx3qnQAAAjlAO4OhYWFKikp6fDYxo0bOxz3eDxav359h8eKiopUVFQkSfrFL34hp9N5ewoFAKAD3OgJoMsJDQ2Vw8GeBADgx4NQDqDLGTt2rKKioqwuAwAAE6EcAAAAsBj/fwugS7Db7ebXNptNdrvdHPN6vdfcHAoAwA+JUA6gS5g6dap5s6bT6dS4cePMxx7u2bNHJ0+etLI8AEAXRygH0CVc/UeDpkyZovz8fPORhwAAWI2ecgAAAMBihHIAAADAYrSvAOhy1q9fL6/Xa3UZAACYCOUA7goDBgxQz54978jc/v7+d2ReAADaEMoB3BWioqL4g0AAgE6LnnIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoTyDgQHBaraXWt1GQA6iWp3rYKDAq0uAwDQiRHKOxAZ4dKFi1UEcwA3VO2u1YWLVYqMcFldCgCgE+MvenbA6QxQQnycLlVV68LFKqvLASDp+Ikyq0voUHBQoBLi4+R0BlhdCgCgEyOUX4fTGaCecTFWlwEAAIAugPYVAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYoRyAAAAwGKEcgAAAMBihHIAAADAYv8PvbgynTL/D5sAAAAASUVORK5CYII=)

### Access Token (Required)

In this text box you need to provide the access token to be used for authorizing the requests to the LaunchDarkly API. This can be either a personal or service token. The access token must have a role that allows the `importEventData` environment action. It is strongly recommended to use a dedicated access token with this permission.

## Metric Options

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw0AAAB+CAYAAABmv/0SAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABvoSURBVHic7d1/dNT1ne/xJ8lMwjdxAhlwgASYYAYY0hlkogQbe4MFa/RCbVzBI9xKT3Gvei/bleuyXm2Xdbt2+2OX3eq23mt7r7QFhfaCGhWEqIkSa5SgRJhZGEoiTDEBpjAxGchA5kfuH0kgQDKEIobA63EOB/L9+f5+J5zzeX0/n893hjS3tHYkEgkAOjo6SCQSJOJx4okE8XicWCxGPB4n2t5OLB6jvT3KDZ5piIiIiIjI1SFloAsQEREREZHLm0KDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJmQa6gEFh/wZ+/Gw1IXIpXfows0efuTr03i9Z8Uo9scwiFi+fh/PzimL7K3j6N3Xkzn+YeQXGn3GAGMHtm9n8ex8Nf4pgGpbLlBmzKL3ZgeViarzoukRERERkMFFPw4VINOLzNZ21MITft5/YBR3Hx9onHuW7z9cl3y/NwJJtxZJuvuBSAYLvruSZ/1dNQyIHT5EHhyXE9tdW8otX/EQuptaLrEtEREREBpdB19MQ2V9JxT4HpV+1c+4z7giBtyuon1DK7LzP/wm4Kd3EYZ+Xplk55HTHrWYfvkAMU7rpwoJDf+SUsPivS/68fY/XsbmqnujYUv7qodnkmADCONc8xdqtFVRPd1I6dgDqEhEREZFBZ5CFhhDe6hpqdlXTGL2Pxbc5egSHCPVvrGR1VQBTQQ6evCKsn+u5TeRNdNDo9+I9VEpOTldFPj/7Ux04JzTiO9Bz+zD1b2+g4gM/jW0mrBM83P6N23GdrOTpf6+kMQHsXMt3n/Cz6IkF2Let5EcvB3FOt9Ps82K6ZSlLRlfzxHPbyb93OYsKDSBG09YNbHi3jv0hyMpxUnxHGSX5vcSnvV4ajptwzCnuCgwAFjzFHip2VuPf1UTp2BzCW1fyo5dDOGfkEt7lpbE9i1zXLObfVYQtWNFrra6968+pK7hzMxuq6ggcjWAa4aTotrmUFliBGL41f8+qgJvSYvD93svh9izyispYMMeJBSAWpG5jOZUf7ycUM07fqxGD7NdTRERE5Ao1yIYnWSm6dxGl+SYCVatZ+UZ91zCbHoEhv5T77/28A0Mn0wQnzvQQ3p3dQ5RC+Lz7YYILR8aZ2za9tYqVFfWYnSXcPstN1sFq1qyupGmYhznfnI0jHUx5JSxaOBN796eQCOHzBcl2FeHOOTcIhHeu47mXa2m1Tef2O4rJafey4fl11LacW2v4SDORFCs221nHGWHDlgKHj4RO94wkgjQ0GnjmLKTsRivBj9azqiJALDtJrT1E9pSzck01jWlTKLltNlPSAlQ+v5INDT36XprrqPkPM66vzKZwRIT6d8vZsv/0vVq7tRVbSRnz/7OHrMZq1qzeQlMi6cchIiIiIl+QwfcoN83O7G8tgt+soqJqNSsT85nJFta90xUYvjWbnLRLc+qYyYF7shnvLi9Nt+WQ0+zD+yk4/sKJ4d/QY0M/Ne8HMBfMY8GcQgzAlRZkxWtevEdLKS3IZVsKkJWLc3IOJiAMgIHnGw+wYFpXQ39Pz7OH8W71ErYWsWDhXBwmKLHDMy/U8ccDMYqGnflRRk5GATPG2fci3YwpFYhFTy9LseApLaN4IjAthxMHVrBhp5f9d8zF0UutZwrjfW87oUwPi749D5cBFFppW7GWbe95mZ3v7jqvi7nfnofHgFhOM7uf207TwTDkGRw+EgLDTeFNRbgMsFut+IJZmBIMulgrIiIiciUafKEBzgoOq1gFWCZe2sAAQMxE/lQ3xk4v2z+dzdB9XhrNTuZPMYj6emwXaqQxAhHfep78u/U9VhiEWpMcP8WEkdnHXIzEYYLBGKbcHHK7P7XxpSx5vLTXzc3pZiBKpP2sFSejxOKAqa9JzFZyxhjwaZDQSSA9Sb1ddTUdjmHKzcfeXXqmA0eOCV+wkWCiKzSkmDF3HcuUPhQDiMaigAXn9CJse2pYsyKIw5mPY3Ih02/OwVBgEBEREbksDM7QAGcEhxqKL31gAKKxGIbDxRRjO/4d2zECjZgnzseZGcPbcyhN17+thfNYcNOo08tTwHLtRRRwAY3o7JHZmBJ+gsEIjO8RRI4GCSZg1Ehrnx9+rPtaLnB40OnjRTv37ef+xuQylj7iYvtHPvx/8LF5WzVbXPNYsujSDDMTERERkQszuJ/lptmZ/a1lLPsCAsPpczpwFRgEt2+k+lMzTpcTAzqf3nezjsKWDq1Hwhjj7djz7NjzrJhNVqwG/FlZLWUUtpEmYsEmGrunCgRrWb9qFZUN5763ych348yMUf9BDU2nVkfwfeAjhA1nQc4Z28eiXRslmmg8EAaLjVH9qTVlFDmjTMQO/pFAd69GpInAoRgmW+f8iaQSQeo2b2BzQwZFt5ax6L8vY/GXrYT9H+PvZa6GiIiIiHzxBm9PQ7c0o5dXr15KJhxT3Vi31RAyPLgnG3TPSDhdk5PiGbl4qypZ+esw0ydaieytoabRzsJlC3ClG1gyIRbYxua3zRTNdPXjGiy4Z7ipXFNL+RqDognQ+FENdS35zPtGLx+jxcPtt2yjYVMFz/2vIJ5JVmJNXmr9rVhnLKKk5+tWE2G2v74Gy1EHpgPbqGwyYZvl7pz0nDi3VtvZdd1USNWqWtb9xsTMAish3xbqIlaKb3af/7pSDAjVUf1+AyeOl+C0hPHuawWLE1vmeW+KiIiIiHwBBndPwwAxTXDhHAbGZDfOXlvFJuy3LWbxHW4sh+uo3FSJr83O7Hvmdk4UTrFTNMuDLbafmg/8hPr5BQ+WqfO5/65Csg7VsHlTDYE0N2Xfnk/RsN63t828jwfvKcFOgG3vbcEbysJ9xyIeuMt5ZmM+xSBvQgb1v6+gcneE3BllLLrV3rXu/LUaBWUsvreE3LCXzRsr8UZyKVn4AHMn9ifOWfDcdT9lUw32v7OONesracwspOy+UhyDP9KKiIiIXBGGNLe0diQSnYPPOzo6SCQSJOJx4okE8XicWCxGPB4n2t5OLB6jvT3KDZ5pA1y2fF46v6ehkaL7l1M2caCrEREREZHLkXoaREREREQkKYUGERERERFJSsOTREREREQkKfU0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUgoNIiIiIiKSlEKDiIiIiIgkpdAgIiIiIiJJKTSIiIiIiEhSCg0iQjQ60BWIiIjI5cw00AXIVSIRou7F1WzYGcJ+12MsKjQGuqLLS7iWlf/WQNHyBbi+iCh/8gg73trM5q1e9h5sJZoAMkbicE6j5PavMTM/6wsoQkRERAYLhYbexHys+v4qfCd7LjThWviPLJr2Od+yo36q9xlMv9HOoGhGx5qoe+8wo272kHMBtyLWUMWGgJ35jy7Babm4e9j0xtM89Y6JuX+zhJIRPddE8K35Mav2OFn0xPka3xECH24jMqEE54jet6h/bQUbjYU8fGtO/woLVvLM/25k+rJFFGWeXhzeupIV2xws+asSbP070iUVbXyP537+O95vGcH1xbdw352jyUyBaPMBdnzwHqt//B7Vtz/I39w9mYyBLlZEREQuCwoNfUlxUPb4AxRnX+LzHPGxpdaGa7CEhuhh6qrq8BRdWGiItrQSG+3BcZGBoZsp0UhdXZCSW3s0w8NeavdG+nmECIHaLQQtfYeGvOL5lKVcQDPf5sF9bTX+PRGKTvWkhNm9O4Dt+rLLIjBwZBvP/uvz7BpxK//zsTu51v87nl39KvVtZq5138lD/+MH3Pnhr/i3Nc/wY5ay/O7rMA90zSIiIjLgUh97/PF/6OjoOLWgo6PjjD+JRKLz73icREeCeDxBzpjRA1jyFyARZEf1Aaw33cC4M1ryMXy//SfKm93cYDdO/xwu5IZx6UQ+rWH9r3/N+g3V7GwawhinneGpEH7/l/zz20c4UVfO6hcr2NrQzugCB2xfxdPlXlqaA+yoO07uTZOwDulxukOVPP3sO3z6SQ2vb6ygsq4Jwz6F3GtSOtf97C3+sPdNXqr8jOtumsTwY/VU/m41L7z0KtV1AU5YHThGpnfWueb7rPMe5ePK13n9zWr2REYyyTGSdIBECN/ra1m1dh0VtQ0cHzaBSTYDEvWU/8tKavbVseHFbQwZEabiNxV80naEvdv9JK5zc+SlH1Decj1FeQYQpub//JC3h9zE9WNONzUjH6/l6df8hP+0l9oPDzGi0I3NHCP44au88PxaXnxjK/4jaYydmIslld6vrcd9CTdspd48hkTgM3JnnF4X+nAjlSdsjDlmJm+mG9sQ4Hg9lb9dzQvrN1G96yjG+EnkZnzChp+u5PeHwjT6PyKQXoAnt/HMa/1SEfEPnuW1Pzm56bpMIELg3fX8Zs16Xq3aSkN4OBMm2jB6fl4YWE7s5c36dIqmjulsbB/fQcWmNtxf/wrjDIgF63h1zQuse7WCau8hzOOmMM6SAu2N1L3fTO5MN7YOP+t/+DuOfKkIewYQqWPlDzbBTR7GmPu4pmv6M6aplff/78/Z1H4zSx+9h4Jjb/CTp7aQuPEeFn0tl9AHr1Jx8DruuusObsz8hI0vf0B06i0UDO/X/xoRERG5gmki9AUx4XQ5CPn9hAFi9fj3jcLltEDEz8Y1NZhnLWH5E0soMdWy7s3AqT0jgUaMWx5k+XcfZHq0hqrtIawzFvHYvYVYxs5mybK5OHr7NMIRsm95kGWPP86DnjAVL9UQ7F7X1kpW0QMsXzoXe0qI2nVr8Y8oZenyH7L0G7nUr19LTXP3xlHCKfks+OtlLF9ahtVXTsWuGADBd9ey4bCDhY/+I4/dm0/TaxuoDXftFg8TGVnK0u8toWTqbB54pAxHuoOyR5Ywe6zBFJedoN9PCOD4bvyH7DgnntlnYkxbwMNzHBiTy3j80QW4DIg1bGZVVSvTvvkYP/zeYqZHq1izqZ5Yr9fWy3251oUnzUddQ/ceQXw7QkzxOHo8GQ9T++J6/LY5LH1iOQ/e0ErVSzUEUxzMfWQJs8daKLz3MRZ/2XrutZ7VLRDbU8HabWZmP7ScHy5bQH7Ty5RvC3M2q8vFqICfhq4Oj8geP43jXLizO2usWV9Bq2sRj39/Ofe7Q1S8Vtt57/qtj2vqz65HanlrF9xYdicFGRBtgZwb7+b+RTO58ct3ct9Xx9G2dw+BOFz71bu5Y+whtrztQ3OkRURERKGhL4l6yn/0KI8+2vnnuz+vJAiYJrmxH/GzOwyxgJ96qwtnNsT21uEdVswslxWTyYqn2AkN9QQTnYczTyyiOM+CycjBcZ2F1s/ObXD2ymLHkWMCTNhmFOM46qe+pWvdMCeeAismE9Ds4+NGBzO/5sRiAkv+bGbmH8bn626SmrFNcGBJASxOSqYZ1O/dD4kmvDvCOP9TMTmGCWN8MdPGBGjY19UYT7XhnObAktZHeZNd2A/5qQ9DpKGBxtFOpljOf1n7P67DXHgrRTkGpNkomjUdw/cx9Ylerq0XsYQVj8eK/yN/Z9BoqqPuhIvp43vs0OLl4305FM90YDGZsE0vxtHc4/6dLcm17t/lx3JDCa5sE2Tamfn1u3BZe2lOZ7txjQngbYgBEfy7AuROddN5S6x45j3A/Bk2TJjImejAGg4RTpx7mD5d6DX1EN3/CYEUJ9e7O2cqmJ138NBfzsSeCpw8wIfeI5jHjycnFWA017tG07bvE/50AeWJiIjIlUlzGvrS15yGNCfuCRvYvjfMqIP1WAtKsAKhcJhoYANPf7+ic7sERC3TicCpuQrdN9uMGS6kodjNNBQjPUqkt2H7xyO0GhZOTxkwYbEYtLaGgXPfhGNkGkSOnACguaWV7c8/SV13hIyBfUo/5wZYpuDM2YhvT4isfQFGFZRy/swQIXw8hsXe4+YOs2JpD9B6su+9zpCArKnTyKnejve4E+tHXpi6EHtq/ekn4+EwrSf9lP/zE5R3LYombGRHgGH9PA8AMcLHIxjjTl+ZKcdFUa/bWnB/yUb17npi+TH8B+y47urez4Spzc/GX9fR2BqDRCuHE54LKeSirikaboOhIxl29v/66AE2/fSnvNhSyHf+2/RTk5+HD8+Atjba4kDqhZUpIiIiVxaFhgtmIr/AzoadNdQdycJ5X+fQFsMYitk5n8cWec6Z0NzPPoXzOxmmNWLGZgBnt+kzLWRFAoRjQBpAjHA4gpHbexM+3NKKYQwFMrBkWpk5bxml48/aKHG4H0VZcBbkUumroi5owTnL2o99DCwWg3A4zKlI1RKiOS2LrH7Ouo0CJoubwglV1G7fTtbuLDz350C8/vRGmQZZlkLm/u08nBf1m27CkmkQOd4jAoabCESysdvOnb5ucbnIedeHd3eUwDg3Zd1vUmr3s3FtHZb/spiHxxvQVMHTz/c2+Ker2Hgvqy7imsyWDGhroeUk9HwtUnTXm7zaMJI7/+GbXN8jeHwWaoOMDDIUGERERK56Gp70ZzAmurHv20JtmhPXiNPL8hurqfCFiAHh/dVseK/p9Bj9vqSYMJ9oJdzex/pwA96GMCQi1G+poeFaJ47enihnO3Hl1rOlqp5wAiL7K9nSMApPQXcjPkrjnt0E2yEWrOWtuhiOiXmQkoOzwExdVTVNx4H2IHVvVODrcwiPGVNKhHD49JVZC5yMaqjFO9TZ55uIzpY31Ul022ZqD8UgFqLunW1EXdNwXFBD2MDpcRB8ayO7bdPwnH3ubCdO626qu+5JrNlP5cbariFjZkzmGK3h/kW6vAIH4Y+q8bcAkSaq162icl8fn26mG/foeireCmCf6jwdIqOttJ4AEjEi4SC+j+oJxWPnzhlIySJ7WIj6vUFisTCBD+to7O6ZSnpNyZnzJzORBj70tp25fPwtPLR0Mbfm9lgYP8CH3kNk5E+mny+cFRERkSuYehr60jWnofzUAhPOux9n8QwLGPk48w1CuU5OPVe3eJh/T4h1rz3Dk7+LYYxyU/IXxee/wXYXnrS1/OInJ1j0vXk4z45xaQbhrSt58vnDkO1mzr3Ffby600rx3fMJv1zOU98PwzA7RfMWUHyqIW0miz9S/u8b2R82yPtyGXcVdFZnv3UBpa+Vs2rFZiIpVuzT5zJ/GL0PoUqzUzg1wrqfPUXkv3b1TmS7cI7ZTMQxpd+vFTXlz2XhzHJe/tWTbGw3sLlmsfAOxwX/QpomTsdl2U14mruXYVE2Su6ZT+vL5ax4IgSWPDy3lWFNAbDg9ORT/coKfhlZwgM3n+c8k+ewIFhO+c+fYG3Cgn16GfOn9zUQy8DpslO+D9zOHj0RmYXcfnsD615YQU2KDfcNduzGCU6cMyTLRnFpEQ3rn+LvN2eRN82ONSVyal3f13Qe2dOZff0r/Kz8JXa4v8n1Xb0NLdtf5WebRvPIT+6loKtXoanyRd4KjuOOv5zcjwOLiIjIlW5Ic0trRyLR2TrsfsVqIh4nnkgQj8eJxWLE43Gi7e3E4jHa26Pc4Jk2wGVfJQ5V8vTzEcoe6eMNQv0Ww7fmSWrzl3WGns9dmNqVK2m+7WFKx16Cw8vnp7mOZ//pl+ywzOT+JXdz40gzNB9g1xEzeRNHk0EbgTd+xb+u28O1d/4tj319nL6nQURERNTTIBcpESPcsIX3jzmZr3Esl79sDw89upjnfr6an/1dHQU3FjF10niGp7dT89o77NxWy46DZgrKlvKdOQoMIiIi0kmhQS5K8N1f8MzvzRTfex85miEzONimc//yyXy5+nUqP/iYTdu30BKFjGGjyXN+je88OJMbczPOfxwRERG5amh4koiIiIiIJKVnwyIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk0iIiIiIhIUnp7Uh/C4WOEj7URTyQ4ETnn27dE5AuUkTGUtrYTA13GOYYMGUJmpkF29jCGpqcNdDkiIiKXjEJDL1pbjxE+dhzDMLBmZw10OSJXvb31ASY67ANdxjkSiQTh8HEOHDjIuHFjFBxEROSKpeFJvTh2vI2MDAUGEUkuJSWFYcMs2K610tzcMtDliIiIXDIKDb2IxeNkD1dgEJH+sVgyOX48MtBliIiIXDIKDb04eaJ9oEsQkUEkJSWFjo6OgS5DRETkklFoEBERERGRpBQaREREREQkKYUGERERERFJSqFBRERERESSUmgQEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaRMA12AiMjnquMg217ZQsOJrp9Tx1D0jZlclwZE/kDVhu0E413rjEnMurMQ2xAIfvgqVfVtXStMjJl+JzPz0774+kVERC5DCg0icsWJt0eIRLp+SI2fXpGI0x6JEOlelNpO9z/jsTYip3eiPd5jPxERkaucQoOIXBHaW4N8FgESR2k7o70f4bPDQYJpEI8co73nqngbnx0KkpoS52jbGYejvTXIwcMGqaRhsQ3HGHLJL0FEROSypdAgIleEo/9RxWZ/5NwV8SA73trIjt52igT4YHOglxVxgt4qNnuB1LHcPP9rTDI+33pFREQGE02EFpGrTCrGsOFYNF1BRESk3xQaROTqkmIh/+ZbmToydaArERERGTQUGkREREREJCnNaRCRq0KqubtnIZXUFBOkppJq7loUjxNPDFRlIiIilz+FBhG58qWOoXDOTCYZncEhNT0NSuaTlwCIc7D2FaoaeplELSIiIoBCg4hcDeIH2Vb+W7YBpAzHdfutDPv4Zd5r0ncxiIiI9IfmNIiIiIiISFIKDSIiIiIikpSGJ4nI1SXxGbvffBniGpokIiLSXwoNInJFGD7xK8wacwmCwJA0rPoiOBERucopNIjIFcEYORb7yIGuQkRE5MqkOQ0iIiIiIpKUQoOIiIiIiCSl0CAiIiIiIkkpNIiIiIiISFIKDSIiIiIikpRCg4iIiIiIJKXQICIiIiIiSSk09GKokT7QJYjIIJJIJBgyZMhAlyEiInLJKDT0IiUlhVBz60CXISKDRDh8nMxMY6DLEBERuWQUGnqRdU0GkUiE5s8UHESkb4lEgpaWMME/hcjOHjbQ5YiIiFwypoEu4HJksVxDRwccO97GseNtnIicHOiSRK5qGRlD2VsfGOgyzjFkyBAyMw3GjRvD0PS0gS5HRETkklFo6ENW1jVkZV0z0GWIiIiIiAw4DU8SEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaQUGkREREREJCmFBhERERERSUqhQUREREREklJoEBERERGRpBQaREREREQkKYUGERERERFJSqFBRERERESSUmgQEREREZGkFBpERERERCQphQYREREREUlKoUFERERERJJSaBARERERkaQUGkREREREJKn/D16Pyd5+6OqXAAAAAElFTkSuQmCC)

This configuration section is available only if your [Metric Type](#metric-type) is set to **Numeric Metric**. In that case, this configuration option is required to be set.

### Event property for Metric Value

In this section you can specify the event property whose value will populate the LaunchDarkly `metricValue` object.

> **Warning:** Since the metric value needs to be a number (e.g. `10.0`), the Tag **will fail**, if you specify an event property whose value is not a number.

In order to specify the common event property of interest, Key Path notation can be used (e.g. `x-sp-contexts_com_acme_transaction_1.0.total` to select the `total` value from an entity (at array index 0)) as your metric value.

## Context Keys

### User Options

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxAAAAB/CAYAAAB/syTvAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABnrSURBVHic7d1/dNTVnf/xJ8kkQ6AJkkKoCAYhFhJxJVChRAtYLFGLQCs//AHoRnG1Wku/WMuyVb66FrGVo+uW2gJm1ahYpP5AsAZBgVZSQ4VaCGKbuMQA1QBBgiRkMjPsHyQQEHGQahp5Ps7JOfD5zNz7/kxyJveVe+9nWu2o2rU/Go0SjUSIRKNEIhHC4TCRSIT6UIhwJEwoVE+/7D5IkiRJOrnFNXcBkiRJkloOA4QkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAUKSJElSzAwQkiRJkmIWaO4C/lnV/imf/1xQytkTZ3BF78ajYdY9fgfzN2Zwxe15ZCd9TrW8W0zhsiLWl79POLET6b0HMvTC/qS3PYFGNxfyX4+u47QxP2B01ud0IZIkSWrxnIH4JxfeXMicuQtZvTVARt8css8IsLVoIb9+uJDyUKyt1FKcP43bfvocpdGGQ4lJJLdPJTmY8BlVLkmSpC+iFhEgajcv57lXy6k9+lnKX32O5ZuPfrZlq6K4cCVbAxmM+t5NXDFyOKOuvImbLs0gYdtKCtdUffqmOw8i75brye3hJJQkSZJi1wJGj1WsX7Wa1RtXsbV+AnnDMji04KaW0qX5FLxSTiCrM9nd+pP6OVe3p2w5i5esYdN71ZB8Gr3OH86ob6QfrLFqQyGLlxVTtrOWQMde5OR+h6E9kyFazuL7ZrP6S9n0TyineFs6E2+/gl5NI93uTWwoD5OcPYj+Xz50OPXcHHotK2X9xk3sOS8HXs/nnmer6DXgNPZsXM/WUAqn9f4mY77Tn7S4Up67Zw6rdwOsZs60TQy6YSrD6xYy/eG19Lj8dib2TQLCVP7lJRa/so7ynbUEvtyL/sOGk5t14BXdUzSHe56vpf+3e1G9pphNu8KknjmIMWOGkp4EUMu2oiU8+/v1bP2gnqSvNLlWSZIkfWG0gBmIVPpfPpHcHgHKXykgf2lpw0xEk/DQI5drL//8wwN71vHs44VsCvQid+wYhqbXsn7JYzz35wMVhsueI//JlWxLyWbosKFkxpVR+OR8Vu881ER483o2RdPp/7UeH61/RyVV0QCpaWmHJ73ENNLaBwjvqOT9xiVJ0UrKtiaR/e0rGfW1VCrfWMhjheWE406j/+gryOkSgLa9yB0/mv6dPnoptW8/R/6Tq9iamMmgYUPJTCxn+eP5LC4LH3pQdCvFfygnJXswg3smU7VhOYv/WHngOt5ewmPPryV8xlDGjB5OJmUUPj6f1btO8DWWJEnSP5UWMAMBJKYz9OqJ8OhjFL5SQH50DINZydMrGsLD1UPpnNgMde2upLI2QFpmDjl90qBnJ5K7llKfHAZqWftaMZXJfckbdxEZCcAZYbb+YiXrNlaRc15DG52HMnHSUDofLcqFw9RGITXxyH0KARKCwO4wB4f3cclk544i50ygT2f2VdzH4r+sZ/PFw8no2YvOrwF7UknPyiDtI33tYf1ra6lqm83Efx1N7ySgbyo1981nzWvrGdoj+2Affb+bx6ieAahN5v2357Pp75WESaN6RxXVpNL73ByyuwXI7pJC2lvVpEaP7EuSJEktWcsIEHBEiHiMx4DkM5sxPAB0zianRzHPFc5m5ls96PXVXvTul0NG+wBEy6l8Lwy7ism/s/iwp6V+sOfgvwNtk0j+uHmgQIBAHNSH6484Eaa+vuH8UZ+YSudTk2BLJVV1wCfdZCn6PtveDxM4rUfDciSgbQYZnQNsqNxKZTS7YXYkgdbBhh6DSbQJAOF66oHUrHM5e9XTrHr4PsrPzCDjzN5k988hzRs8SZIkfaG0nAABh4WI1eR8tuEh7sBLUx8Nc+hlqocoEB+AeCAujZxrb6XbhjWs3VjKptcWsnplEUOvvYnc0+sJR4DOOVwxKvuw5UkJ7U4Dtn5yDalpdIoL8/57lYRJPfTNClVRuTNMoHManT4mfIQPLm06rqtu8gPRcK2xPr99NldMPo3sP69jw9ubKH6hmJWv9WfijaPpdSK3m5UkSdI/lRawB+IIiekMvfpWbv2MZx6SOqSSGhemsnzzoWVCoa2UvxeG9p3oFIDaslUsXrKa6vRBDL88j1snjyIjspV1b26GuE6kdQjArirq26eT3u3AV3JiEu3bx5jb2veiV5cAezauZm2TvQRV61azqTZAt6xeNN2iHK5vqDS6ja0VeyA5jU5NZwA+LgzEdaJzpwDhv7976Nawtdsofy9MIC3tKEuejhRm2+uLWfz7StIG5jL6mh9w69hsEnasZV3TPRSSJElq8VrWDESjxKRPXJVzwrr0J6fHGp57bT6/3tefXh1g11vFFFcl0es75x7YsxCspaxoORu2h/lm3zTYuoH3owE6dUgFkske1JdV/1PMcw8/RmW/biRVbWL1mmqyr5/M8NNjKSKVnEsGs+7h5Tz3qzm8+y/ptK4pZ8O6Uuq/MpTcc5vMa0T3sPbFJ0nemUGgYg3LtwVI++bZpMcBJNCmTQLsLWX10lWEzx1Er8P6Sebsr/fllceKefrRAIOzUqnasJJ1tanknHc2ScAejiVA6/A21ryyjvLaanLOCFD1l3Jq4zqR2qFl/ohJkiTp6BzdfZy4NHLG58FLhazesJLltZCUlkHO6IsY/rWGgXuXXCaMh8XLilmyoJb6tp3o9c2JDB944HxSz1Fcf2Uyi18pZnXhJhJO6cbZF49haLdAzEuDAt1yuf7aZApfWc364pWEE1M57dxRXDEsh/SmMzBxSXQ7ow2lfyhka00Spw0YxZgL0xtbodf5Q+n17nI2Fa2jU68jAwQkZY0i7/LWLH5lHS8tCZOUls6gK0dx0ZmxRbXU865gQmgxhWsKefr1epI6ZDBo7HCGdo7tOiVJktQytNpRtWt/NBolGokQiUaJRCKEw2EikQj1oRDhSJhQqJ5+2X2au1Z9jD2v53PPs1vpf+3tjDqzuauRJEnSF1nL2wMhSZIkqdkYICRJkiTFzCVMkiRJkmLmDIQkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAUKSJElSzAwQkiRJkmJmgJAkSZIUMwOEJEmSpJgZIHT86uubuwJJkiQ1k0BzF6BPo54539/D9mvb8x99Wn0O/dWw7fVlLHp1DW9u3kFNPZCQROduvfnaBZdw8YCv0OZzqEKSJEnNzxmIRv/7IWOu2cPrkSbHttRw1bhqloU+g/5CIe6btJM7/rj/8OPlexkz7gOe2fUZ9Plp1LzD7x64k3+ft4Jt7fowYvz1fP+m6/n++Es4p917LJt3J1MfWEZ5TXMXKkmSpM+DAaK5JCbwrf5xvF4Uoq7J4Xf+GGLLWa35Vvtmq+yQ+vf43YO/4Kl3T+XyqXdz+7c7UL60gP+ePYeHl77LqSN+xKypI+j27m+Z+eAytkU+uUlJkiS1bAaImO3nrUXVjJmwk37jdnLVrFreaZyZiERYNvcDRly5k4F5H3DfqoaRdCTET/N28b27P2DIuF08sqVpe604Z1ACwfUh/nxwhiPMquIoA76RSDLA3nrmz9pF7pU7GXjNLu5YHj5KXVGW3LWTyUujB9t4ZEqTmY2Pqy0G218t4Lfvnsplk2/i4tPf47cPPcWbbc4j7/rxDG6zgfxfLGJbt4v5/uQRdH53EQ8v3xFz25IkSWqZDBCx+nstdz+9nyt+lsobc5MZvK2G+5ZGgP28s3APP38nkXvnprL01gTeyv+QZ3Y2PC8aZfepbVj46Clc0+WINjOCXJhUz7KNDYP9v4f4fWUCw/vHAft588kPmU8STxR8maWTE3jr4b0sqT6eoj+htmOJVLBq1Tuccv5lXHx6AtTsJuGMwUzMu4zBA87j8jHn0XHX2/xtJyScfiFXnN+O0lUrKXcWQpIk6QvNABGreGjNfrZsibCnbSLX33UK914YB5Ewy1ZFGTQ6icy2rUjOSuLb6WFeL2kIBXFxDL4wkQ6tj7LZOT7A8K+34vU/Hrir0dY/hthydiLfaAvQiu7Dv8RDN7SmQzwkn51AZnyUrcezN+KTajuW6ncofT+FzHO6kwDQLpvLrr+cgWkANZT+6W0++NLppLcDSCDjnJ60e/8d/vbhcdQnSZKkFse7MMUqLYl7b9nPg0/v4dIHIevrrbn12iSSibKlOsqSn1WxuPGxUcj+evRYrTVoRfdBiSTfE+LNUDwbi6MMGBkk2HA2OT7K/Nkf8HL5fmA/2/e14srjKvpYtcUf+6k1NeyNb0NG8pEn6il9+gFmroSLb/kuWQeLTaFt/LvUfAi0O64iJUmS1IIYIBq1jiMxFGZPPYfG1nuj7EmE5HigOsL2Lknc9bO2sLeeR+6p5vZnAjwxLo4ObeO5+iencEvPI2YZIjGs50kP8q22e3h5TRzvvB/g6r6NbUR4ZvaH/PnsFB6/LYFgJMRPr917lAYOPD501K6OUdsn+VIb2kZ288HuejgwB3HAh8UsWv4eZ467g8t6Nbl56+5qPoi0oc2Xjq8bSZIktSwuYWqUlsDgL9fzP0/UsXXffup21TP/NyHq+gTJioe6v9Uw+c4PWVa5HxLi6Ni21YFBe3yAwefCkidreat6P+wLs+SRD1lWGWvHAb51fitefngfW/q1pk9i4/H9VH8IdUBob4S3lu/j9b1A9KPLj7qktaLszyG27tvPjjf28XLjZu0Tqa1ddzJOreXNte9w2MfGJfRkxI03MSGnQ5OD9Wxcu56aU7uT5eyDJEnSF5oBolF8AtdPa0v2/9Zw1YSd5Nywh2VtW/PADUGSgWC/ttw1aD+/nlJFv6s+4JFIa37y3QSgFedM+BK3fDnE5ElVDJy0hyWRBPqkxd71aecl0mUPDBqUcHD5EgS4bEJrkpdVMyxvN/dtimPAGVC9+8hnt+Kc77Zh0Pt7GXFlFXkLo3Ts0OTcp66tK4MG96Sm6LcsKmsSIXas5uGHnuJPTW64VF/2O54qqiVrcA6dY79sSZIktUCtdlTt2h+NRolGIkSiUSKRCOFwmEgkQn0oRDgSJhSqp192n+auVZ+3yA6WPTCDgndPZcT1/8ZlZ6VA3Q5KS6tpl9GdjkHYXbKIX835HZtPv5y7Jg+m4ydsrZAkSVLLZoDQsdVVsCx/Dk/9aTenZPVn4Nln0LldIqHdf6d0/RqKNu7mlK9dxv/LG0zn4Cc3J0mSpJbNAKEY1LP9zZW8tLKYN8ve44OaemjTgc49ejJw8EVceE6HptusJUmS9AVmgJAkSZIUMzdRS5IkSYqZAUKSJElSzAwQkiRJkmJmgJAkSZIUs0BzF/DPZGNZmF/9ppbX1oWoC330E58lCSCY2IrzshO5YVwSWT18G5UknVz8zddgY1mYCVN3GxwkfaK60H5eeb2O19aFKJjZzhAhSTqp+Fuvwa9+U0tdaD/fGdqaH17dhtR2ru6SdHRVu6Pc/2gNzy7fx69+U8uD05KbuyRJkj43jpIbvLYuBGB4kPSJUtvF8cOr2wCH3jskSTpZOFJu0Lh0yfAgKRaN7xUue5QknWwcLUuSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClmBghJkiRJMTNASJIkSYqZAeIfoHpdAdPzhjGgTyaZA4YxfloBa6ubu6rPyIqp9Ok9mcLI4Ye3P5VH5gUzWBs5+tP+YSKFTO7dh8nLD/y37sXJZHbvTvfu3eme2YcBF41n6pwiKj7rOiRJkk5SBogTVLd+Nnn/+gvKut/MA0++wAsP3ExW6Szybi6g1EHs5yMll5kvLWXp4id44JaB7HvmRsb/YBEVzV2XJEnSF1CguQto2bbz/INzqRhyLy//JJcUADKY1rmStd+ex4LiCUwb2Mwlngzi25HWI4OMeMjocRYD/6UjeSN/yuzlQ5g5NKW5q5MkSfpCcQbiRFQX8WpRkAsuHcJhw9T0XCZdm0saDeuYakpYcPt4hvXLJLPfMMbfvoCSmsY2FpHX+1Kmz5lF3kUDyOzdh2GTZlNUXkLBbaMZ3CeTzAGXMvmJEuoAqGDe2EzG3z2P6ROH0ad3JgOGT6ZgfQVFD93IpTmZZPYZzOhpiw5bxlNXXsiMScMY0DuTPoNGMzl/bWN1rJ05jMwb57Ho7oYaB1zK5Kca+/uUL836AqaOHUyfzEz6XDCeqU3bqyk99HoMGEbez1ccrHXtzGFkjp3K9ImD6dP7RhbVfFwPx9BlJOOG1PHSi0UndA2SJEn6KAPEiXivgi2k0bVr8IgTXcn94TSuG5gCbGfRbXnMeDuLKQVLWfrIFLI2ziDvtkVsb3x4pISX/gDjfv4EL8ydQlbpLG787g95teskfrlgIfnXdKTonnsoKD/Uw+qlb9A1734WLniIiWlFzJhwKTNK+jFl3gss/FkuvHgHMxY1RITqFcy45scUfeVmHlqylIV3XcCOX+fxvUcPLfKpW1HAc+0mcP/8hTw0vh0r7p7F89v5dOqKmHXLLEr73snCl5eSf1MX1t79PWb9qQ7YTuHt1zL7vSFMm7+UFx6YQMqLk/n3prWsX8vuoTN54pm7GNLm0xQQpEdGBnWbS13GJACeXBzhkhv2HfPrycWuOZQkKRYuYToR4X1AkGD8MR7z1wXMXZHGpGemkftVgK5M+89SisY+xvNlI7iuIxDflZG3TCH3bIAMJl1SQOEfRjLte7lkxAPpE7lg3mRKS4H0A82edcUUrhuSAUDGxAt4rHgLY6dfx5COQNbNjH26gHlvlwJ92f7iYyxgLPk/GUHfIJB+E/ffUsSgRwpYO34aAMFzr+Pe7+fSEaDLWAb+ehYlpXDgwHGq207F9nacdf5AMroEYfQ07g++xJYUoOx55i7vwU0vN9TKBKZcU8iwJYVU5F13oJaB1/EfVw/8VF03CrZpDXv3URcBjvX90UnhyuEHfggeX1x/1PPjhyccfIwkSTo2A8SJCLQG6g4MUj9GXekGSlP60rdHk4M9+9K37Vw2lNU1DNCDhw1yg62D0C6Fdo3HAkGCwTp2N+knGN9k1iMYpDXJpB1cRxWkdRDqwgcW8JSWlMBZ4w6EhwYds7Ppek8JpVWQ0dBGsEl7wcC+Y17XMaUMYdLVjzH55mGUDMnlgvMvYMQlY8ltA3VLN1BSU8T0CzKZfuhVoq5jFtsb+4uHI+d0jlddzT5o2/rY4U4nlY8LEYYHSZKOjwHiRHylK12opKKiDr7adMhbwYo5C6g8dxIjCR71Vf5c1+YHWn+0hghA6FO0BUHqIMzhf9mPNG0rhYE/WsiqsUUULl/NKwumcunsLKY98hAjw0D7XO58cgoDm9YUbEdaPJQcf0VHUUfZ26UEu06i6z+kPX1RHBkiDA+SJB0/90CciJSBXNC/jldfXMFhH/uwpZB5Dy6iZF8KwYwzyagqoWRzk/Nvr6Vkb1fO7HGif2ePTUbPHlDyBiVNUsv2v6yjomMWGanH2dipGXSljA1/bXqwjtJNFdC1K13jgbJCZv93IRVdBjIibwoPzH+YSakrKHixlGC3rmTsLWN7XVe6pjd+pZHWPuWEZx0OKl9AwYogQ4YN/Me1qS+MK4fHM354guFBkqRPyQBxQjoy8vsTSFt+B9+7v5C1fy2ltGgRM348lw19r2NCf+CrY5nwjQrm3jGLFX+toGJjITP+/1wqvzGJsT0+sYN/TJWXTGRkeAHTZy6iZEsFpStm8+PZG+g3YSx9j3f81GMkY7+xm4KfTGXBH0ooLSthRf6Pmf4sjLx6ZMOKrEreeOQOZvxyxYH+igp5Y0uQLl9Jg6yxTOhfwdzbprLgT6VUlK1lwW2jufTnJ3DHpMhuKjdXUFFWQtELs7nxmhn8beAUpuR6C1cd3ZXD4w0PkiR9Si5hOkHB7CnkP9SOn94/g7w5ldSldKXfsJvJ/9GEAxug6cjYe/Opmzmd6WPnURnflX6XTCP/RyNOaJPwcUkZwp3z7mLGnb/gqosqoONZDLnmIe7Ky/gUjXVk7Kx8+Pks5k0ZTeku6PjVgYz82cNMafzMhS4TuPe/qvnp/dO56peV1LXPYMj4B7hrRAqQwtgH8qm7ewaz/+1SpkfSOGvoBGbedAKzBdWFTP1WIQSCpKT3Y8jYh1iYN+TAbIgkSZL+oVrtqNq1PxqNEo1EiESjRCIRwuEwkUiE+lCIcCRMKFRPv+w+zV3rZ6r3yB0AbHi+QzNXIqml8H1DknQycgmTJEmSpJgZICRJkiTFzAAhSZIkKWYGCEmSJEkxM0BIkiRJipkBQpIkSVLMDBCSJEmSYmaAkCRJkhQzA4QkSZKkmBkgJEmSJMXMANEgmNgKgKrd0WauRFJL0Phe0fjeIUnSycIA0eC87EQA7n+0xhAh6Ziqdke5/9Ea4NB7hyRJJ4tAcxfwz+KGcUm8ti7Es8v38ezyfc1djqQWIJjYihvGJTV3GZIkfa6cgWiQ1SNAwcx2fHNA0CUJko4pmNiKbw4IUjCzHVk9/DuMJOnk0mpH1a790WiUaCRCJBolEokQDoeJRCLUh0KEI2FCoXr6Zfdp7lolSZIkNTNnICRJkiTFzAAhSZIkKWYGCEmSJEkxM0BIkiRJipkBQpIkSVLMDBCSJEmSYmaAkCRJkhQzA4QkSZKkmBkgJEmSJMXMACFJkiQpZgYISZIkSTEzQEiSJEmKmQFCkiRJUswMEJIkSZJiZoCQJEmSFDMDhCRJkqSYGSAkSZIkxcwAIUmSJClm/we1kgrkjy0gngAAAABJRU5ErkJggg==)

This section allows you to configure how to populate the value of the user property that uniquely identifies the context that the LaunchDarkly metric is about.

#### User Value

With this drop-down option you can select how to derive the value for the `user` in your `contextKeys`. The available options are:

1. **Common User ID** (default): Using this option the Tag will populate the `user` from the `user_id` property of the common event.
2. **Custom**: This option allows you to specify an alternative property of the event to be used. Selecting this option reveals the following text-box to specify which event property to use.
3. **Do not populate**: Selecting this option will not populate `user` as a context key.

#### Event property for user context key

![](/assets/images/05-user-value-custom-ffce8f0a70afdfd5304fbbf846857140.png)

This option is revealed if you have previously selected **Custom** as the option for the [User Value](#user-value). In this text box you can specify the Property Key from the GTM Event to use. You can use Key Path notation here if you want to denote a nested key (e.g. `x-sp-contexts_com_snowplowanalytics_snowplow_client_session_1.0.userId` to use the `userId` from the Snowplow client session entity (in array index 0)).

### Other Context Keys

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw0AAACoCAYAAACi2AIgAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d15XJV13v/xN5zD4bBv4gIqCBhu4EIuWZZmi+aSs2lp3U273TX31Ey3U/O457ZZ77znnt/M1My0aDXlWOqUMZmKImLiCqJOouCGZCoqyr4fruv8/mA4RcCFlkbU6/l48ACu73W+388BH57rfb7f74VXRVW1W5LcbrdatHxtmqbns9vtlmEYMgxDpmHIMA25XE1yNTbKMAzVNzRodMooAQAAAPh68e7qAgAAAAB8tREaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEaAAAAAFiyd3UB+OL27t2rzZs36/z58woJCdHYsWN1ww03WD5m6dKlOnDggJ599tkrWlthYaHWr1+vU6dOyd/fX4mJiZo6dar8/f2v6LhXSlFRkf7yl7/o1ltv1eTJkyVJlZWV+v3vfy/DMPT4448rPDy8i6sEAAC4vJhp6Oa2b9+ut956Sy6XSykpKXI6nVqzZo1SU1M95xQVFWnBggXKysr6UmsrLCzU4sWLVVZWprFjxyo+Pl45OTl69dVXZZrmZRkjIyNDCxYs0OnTp7ukP9M0tWzZMtXU1GjOnDkEBgAA8LXETMNlUlRUpCNHjujmm29utz09PV0DBw5UbGzsZRuzvr5e69atU69evfSDH/xADodDpmnqr3/9q3bs2KFx48apd+/el228jpimKW/vtvlz9erVstvteuyxxxQcHCxJ6tWrl9auXav8/HwNHTr0itd2paWlpen48eO6/vrrvxbPBwAAoD22p3/602c6anS73W0+f/rDNE2ZhiG3260mw1B0VJ8vpeivonfeeUfZ2dkyDEMJCQmt2tatW6eMjAxVVVVp5MiRl23MgwcPau/evbrllls8YcTLy0tBQUHKzc1VYGCgTp8+rWXLlkmSDh8+rOPHjyslJUUffvihSkpKFBAQoKVLl2rTpk1yuVyKj4/39J+Zmak333xTGRkZKioq0oABA+R0OnXu3Dn9/Oc/1/nz57V+/Xrl5+dr1KhRrWorLy/X2rVrNWLECKWkpHiOh4SEqKysTBEREerTp4/cbrc2btyo5cuXKy0tTceOHVPfvn0VGBgoqXkZ1T/+8Q9J0htvvKHMzEzV19crISFBr7/+unbs2CFJ2rlzp7y9vRUXF2dZ+/bt2/X8888rJCRE0dHR2rt3r37/+9/Lbrdry5YtHfb36eeVk5OjhIQENTY2KjU1VTExMbrzzjtbBaeDBw9q6dKlWrNmjQ4cOKCePXsqLCxMGzZs0EsvvaTY2FhFRETINE0tXLhQH374ocaNG6fy8nKtWLFCq1atUlZWlkpLS5WQkCCbzfaF/70AAAB8XixPukzuvvtuxcbGatOmTUpLS/McX7dunTIzMxUbG6u77rrrso5ZUlIiSW1mE3r27ClJOn/+vIYMGaKpU6dKkkaPHq1bb73Vc55pmsrOztaYMWMUFBSk9PR0nTx5UpK0ceNGrVu3ToMHD9bEiRN14sQJvfHGG63G2bt3ryIiIjR48OA2tZWWlkqSIiIiWh0PCwvTPffc4wkZ69evV3p6ugYMGKDJkyeruLhYS5YsUWNjo+cx1dXV2rt3r8aPH6/g4GBt2rRJH3/8sSZNmuQJYTNmzFBycnKntY8bN069evXSxo0bVV9fr7S0NIWHh2vChAkd9teeiooKLV++XE6nU3fddVeri/rCwkK9/vrrCgoK0i233CLTNPXKK6+osrLS02d+fr4k6aOPPlJ9fb2SkpIkScuXL1dRUZGmT5+usWPHaseOHa3+PQEAAHQFliddJg6HQw888ICWLFmiTZs2SWq+KN+8ebNiY2P1wAMPyOFwXNYx6+vrJUl+fn6tjrd839DQoIiICA0YMEBSc7iIiYnxnOfl5aWHH35YTqdT/fr10yuvvKKTJ08qKipKWVlZGjJkiKZPny5JstvtWr16tc6cOeN5R/3qq6/W7Nmz262toaFBkuTr69th/U1NTdq2bZvi4+M1d+5cSVJ0dLReffVV7d69W+PHj29Vp5+fn/r3768lS5bo1KlTGjdunI4cOSJJio+PV2RkpEzTtKy9d+/emjFjhpYsWaIXX3xRZWVluvvuu2W329W/f/82/XVk165dcrvdstlsqqurU0hIiKftgw8+UEBAgObNmye73a64uDg999xz2r9/v6699lr16tVLBQUFmjlzpg4dOiRJntBw+vRp9evXT2PGjJEkBQcHW/4MAQAAvgyEhsuoveBwpQKD9MkFeV1dXavjLd87nU7Lx3t5eXnO+XTQKCsrU11dnQ4ePKj/+q//avWY0tJS9ejRQ5JaXSh/VsvzbQkP7SktLVVDQ4Mn1EjyLI86depUqzpb6mupt6N+O6u9d+/euuqqqzRo0CAVFBQoLi7Oc8F+Kdxut1JSUpSbm6u3335bjz76qLy8vCQ1X/hXV1frmWeeaVObJCUnJys9PV0lJSU6dOiQevfu7Qko1157rTIyMrRo0SINHjxYw4cPbxX0AAAAugKh4TL7dHCQdMUCgyTPxfvZs2dbXXi3LFtqaf+8kpKSNGHChFbHevbsqerq6k4fGxYWJumTC+UWFRUVSk9PV2JioqKjoyXJc7H9aV/07kod1d6iJVjV1dXJ7Xa3W4OVlJQUzZkzR97e3srJydG2bdt03XXXedojIiI0Z86cVo9p2QzeEhp2796t06dP66abbvKcc+uttyoxMVH79u3TgQMHtHXrVk2bNq3TW+gCAABcSexpuAJagsOVDAySlJiYKIfDoe3bt7faA7B161Z5eXlp2LBhrc5v2dDembCwMDkcDp07d04xMTGKjY1VbGys7Hb7Rf99hfDwcPXq1Ut5eXmqqanxHN+3b5+ys7MlSaGhofL19dXx48c97S1f9+lzaZvqW57bxdS+b98+ffTRRxo0aJCKi4s99bTXX0daAtn06dMVGBio9evXq7y8XFLzHaLKy8sVGhrqGd9ms3lux9qrVy/16tVLWVlZcrvdnpmOiooKpaamyjAMzZo1Sz/5yU8UGRmp3bt3X9LPAgAA4HIjNFwhDofjigYGqXlJ0ZQpU3TmzBk9//zzWr16tV544QUdOHBAY8eO9WyQbnmHe9++fRd1Aert7a2JEyfq7NmzWrx4sbZs2aLFixfr5ZdfbhVOOjN9+nQ1NDTo+eef15o1a7RixQqlpaWpX79+GjJkiLy9vXX99dfr6NGjevPNN7Vp0yatWLFC/v7+re64ZKXluWVmZuro0aOd1u5yubR27VpFRkbqnnvuUXR0tNLS0jz7Qz7bX2f8/Pw0c+ZMNTQ0aNWqVZKkyZMnyzAMvfzyy9q8ebPeeecd/elPf2oVjpKTk9XU1KTIyEjP7ykgIEB5eXlauXKlcnJytHPnTpWXlysqKuqif+YAAABXAqGhm7vuuus0Z84ceXl5aceOHaqoqNCUKVP0rW99y3NOeHi4xo8fr3PnzikvL++i+r3xxht18803q6SkROvWrVNtba3uuuuuSwpCiYmJuvfeexUQEKDt27fr2LFjGj16tO6//37P3YYmT56sSZMm6ejRo8rIyFDPnj01f/58BQQEXNQYycnJiouLU35+vufOT1a1b968WeXl5ZoyZYpsNptuu+021dTUKD09vcP+OjNixAjPHom9e/dqwIABuvvuu2Wz2bR+/XodPnxY06ZNa7WErGV24dP7Kex2ux588EFFREQoNTVVGzZsUFJSkm6//faLqgMAAOBK8aqoqnZLrZdjtHzdsq7cNE253W4ZhiHDMGQahgzTkMvVJFdjowzDUH1Dg0anjGpnCACf9c9//lPLli3TD3/4Q8/eDgAAgK8qNkIDXyKXy6WMjAzl5OQoOjqawAAAALoFlicBX6KmpiZt3bpVgYGBuuOOO7q6HAAAgIvC8iQAAAAAlphpAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGDJ3tUFALBWWVmpQ4ePqqKiQk4/Pw2I7a/oqKiuLgsAAHyDEBqAr6jKqiq9teJt7dyVI5vNpuDgINXW1qm+vl4DYmN097w7FDcgtqvLBAAA3wCEBuAyWbp0qQYOHKhx48ZdUlt7zpWUaNFv/yCbzaZ/n/+ARg5Plt1ul9vtVuHxIr37j9X6zaLf6ZGH7lPKqJGX+6kAAAC0wp6GbsYwDL3//vv6xS9+oYULF+qNN95QRUXFF+738OHDOnz48BXr48iRI1q0aJHne8Mw9NJLL2nVqlVfaMzOGIah9PR0GYbxuftYv369fve7313Gqqw1NTXpD8+/oODgIC382VMaOniQFr/yVz3ygyf0zC//R5L048d/oIk3TNBLS17TyVOnv7TaAADANxOhoZt59913dfLkST366KN6+umnFRoaqqVLl37hfq90aPisd955R97e3po1a9YXGrMzlyM07N27V5WVlTp16tRlrKxjmR9k6cKFC/qPR+crwN9ff3tzhQ4fOabZ3/mWQkKC9Yfn/6L6+gbNu+N76te3r1a+/e6XUhcAAPjmYnlSN1JbW6vc3Fz953/+p8LDwyVJ06ZN069+9StduHBBERER+uijj/Tuu+/qwoULiomJ0Xe/+12FhoZKkp555hndeOON2rZtmxobGzVz5kyNHDlSy5cv1/79+yU1b7qdO3euTNPUmjVrlJOTo8DAQM2YMUODBw9Wdna2tm7dqscff1yGYei3v/2tZs6cqby8vDZ9dGTTpk06ceKEHnvsMXl7N+fWjsZbsmSJEhISNHHiREnSn//8Z40bN04pKSnatWuX0tPT5XK5lJSUpG9/+9ue/iSpuLhYixcvliQ9++yzuv/++xUdHa3s7Gxt3LjR87gZM2bIx8en3VqLiork7e2ta665Rnv27FF0dLSnzeVy6e9//7sKCgoUHR2tpqami2rrzM5dORp/zTiFhTX/3k6dLtadc76rMaNTNPrqFD32+JP6+ORJXTUwQdOm3qo/vfCyampqFBAQcNFjAAAAXApmGrqRM2fOKCAgwBMYJMlms2nhwoWKiIhQXV2dXnvtNU2aNEkLFy5UTEyM3njjjVZ9nDp1Sj/60Y90++2367333pMk3XHHHbrmmmt0zTXXeC72t2zZolOnTunpp5/WnDlztGLFCtXV1WnMmDHy8fHR7t27tWXLFvXs2VPDhg1rt4/2fPjhh8rKytJ9990np9PpOd7ReElJSTpw4ICk5tB08uRJDRo0SBUVFVqzZo0effRRPfXUUzp9+rR2797daqw+ffroqaeekiQ99dRTio6O1vHjx5WWlqZ7771XP/nJT1ReXq7169d3WO+ePXs0bNgwJScna+/evTJN09OWmZmpyspKLViwQFOmTFFJSclFtXXm5KlTGpgQ5/n+5//9U40ZnSJJytq6XUGBgeob3Xz3pISEOJmmqeLisxfdPwAAwKUiNHQjDQ0N8vPz67C9oKBAkZGRGj58uOx2uyZPnqyysjKdPfvJBeXkyZPl6+urIUOGqKamRo2Nje32lZubq0mTJsnPz08xMTGKjo5WYWGhJGnWrFnasGGDsrKydPvtt190/dXV1Vq1apXcbrdsNttFjTd06FCdPHlS1dXVOnz4sGJiYhQQECCbzSa3261z587J19dXDz30kJKTkzutYc+ePRozZoz69Okjp9OpqVOnKjc3t91zDcPQhx9+qKSkJEVFRcnhcOjIkSOe9ry8PE2aNEmBgYGKiYlRfHz8RbV1psnVJIePo83x99emadU/Vus/Hpsvf39/SZLD0Xxeo6v93yMAAMDlwPKkbiQgIED19fUdtldXVyssLMzzvbe3t0JCQlReXq5evXq1OrdlGU9Ha/0rKyv1t7/9TV5eXp7zWi7K+/Xrp7CwMAUEBCgyMvKi63e5XLrrrruUn5+v5cuX66GHHvL039F4gYGB6t+/v/Lz83Xs2DENGTJEkhQYGKh77rlHGzdu1FtvvaXhw4dr+vTpndZQVVXVaolRWFiYampq5HK52ixRys/Pl8PhUL9+/SRJSUlJys3NVWJioqTmn3fL0q/PsmrrTGhYqM6cbTtzsGbdBt0+4zYNTPgkgJw903xeeHhYm/MBAAAuF0JDN9KjRw/V1NSorKzMEw4Mw9CiRYs0f/58hYaGevYVSM37BCoqKhQYGHjJYwUFBel73/ueYmJi2rQdO3ZMVVVVKisrU3Fxsfr06XNRfYaFhSkxMVHx8fF67rnnlJmZqRtvvLHT8YYNG6YDBw7oxIkTuuWWWyQ1X5SHh4dr/vz5qq2t1dKlS7Vjxw5df/31ljWEhoa2uttUWVmZnE5nu3sa9uzZo/Lyci1YsMBzzMfHRw0NDfL19VVQUJBqa2vbHceqrTPJw4Zqx85sTZt6qydEud1uff/uuUq8amCrc7ft2KUePSLU+zOhEAAA4HJieVI34u/vr6uvvlpvv/22ysvLVVdXp3Xr1ik4OFjh4eG66qqrdP78eeXl5ckwDH3wwQcKDAxs9c56RxwOh8rKyjwbdocPH64NGzaotrZWtbW1Wr16tSoqKmQYhlJTUzV16lTddNNNrW6Z+tk+OmK32zV37lxlZmbqxIkTluNJze/w5+fnKzAw0LOf4/Tp03rxxRdVUVEhX19fOZ3Odse12+3y9vbWhQsXZJqmRo0apezsbJ09e1aNjY3asGGDRo5s+3cO6urqVFBQoB/96Ef63//9X89HWFiYJ5gNHTpUW7dulWmaKi8v9zyXzto6M+WWm3Tm7DmtXpPmOWaabr3w8isq+uiTfo4eLdSmzVs0/bYpF903AADA52F7+qc/faajRrfb3ebzpz9M05RpGHK73WoyDEVHXdw7zvj8EhISdOLECaWmpmrbtm3y9/fX7Nmz5XQ6ZbfbFR8fr7Vr12rt2rVqbGzUvHnzPOvfN2/erNGjRysgIECmaSojI0OTJk2Sj4+PAgMDtWnTJhUVFWnEiBGKiYlRcXGxVq1apa1bt6pv375KSkrSli1bVFpaqpkzZyoqKkpbt26Vj4+PoqOj2/TxaaWlpcrPz9d1110nqXl5kdPp1Pvvv6/Ro0crLi6u3fEkyel0Kj8/X4MHD1ZCQoIkKSIiQm63WytWrNDmzZsVGRmpqVOnttkr4eXlpaqqKqWmpmrgwIGKiYlRYGCgVq1apYyMDEVFRWnmzJltHrdnzx5VV1d7ZkJaGIah/fv3KyUlxbNs6r333tORI0fk5+enPn36qG/fvpZtnQkMDFBISLCWr3xHTUaTrhqYILvdplkzp6t37+YZhdw9e/XnFxcrOWmovvedWZ4ZCQAAgCvBq6Kq2i19Egw+/XXLnWJM05Tb7ZZhGDIMQ6ZhyDANuVxNcjU2yjAM1Tc0aHTKqC54CvgmeOmllzRt2rSLuuj+uti6bYeWvrlcfn5+Gjk8WeHh4aqtq9XBgwU68fFJTbz+Ot01d06bwAMAAHC5sacBX2mGYaiwsFCVlZXfqMAgSddde42Shg3V5i1Zyi84rEOHj8jp56eE+Dh9/9/maUBs2/0fAAAAVwIzDfhKy8zMVFZWlubNm3dJty0FAADA5UNoAAAAAGCJuycBAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAwNdIamqq0tPTu7oMAF8zhIZuxDRNLViwQBUVFZ5jaWlp+uMf/yiXy9WFlQEALqdly5bp/fffb3P8l7/8pQ4ePNgFFQH4piM0dGO5ubnKzc3VvffeKx8fn64uBwBwmYwcOVJ5eXmtjhUVFampqUmJiYldVBWAbzJ7VxeAz6ewsFDvvfeeHn74YQUHB3uO7969W+vXr5dpmho/frwmT56szMxMHT9+XPfdd5+k5tmJyspKzZ49u6vKBwBYSExMVH19vYqLi9WnTx9JUl5enpKTk2Wz2SRJWVlZ2rx5s0zT1OjRo3Xbbbe16eeDDz7QmTNnNGfOHEnyzF5Mnz5dUvuvGQDQHmYauqGSkhItXbpUs2fPVlRUlOf4iRMntHbtWj344IP64Q9/qNzcXB09elRJSUk6evSoGhsbJUkFBQUaOnRoV5UPAOiEzWZTUlJSq9mGvLw8jRgxQpJ04MAB7dy5Uz/+8Y/1xBNPKCcnR4WFhZc0RkevGQDQHkJDN/Tmm2+qsbHR825Tiz179ujqq69Wz549FRwcrKuvvlr5+fnq0aOHIiIidOjQIVVWVqqkpERXXXVVF1UPALgYI0aM8ISG06dPyzAMxcXFSZJiYmL04IMPyt/fX8HBwerfv7/OnTt3Sf139JoBAO1heVI3NHbsWEVFRWnlypV64oknFBQUJEmqqKjQ4cOHtXPnTkmS2+32zCgkJSXpwIEDqq+v18CBA9kDAQBfcXFxcaqtrVVpaan279+vESNGyMvLS5Lk7++v9PR0FRQUyDRNlZWVafDgwZfUv9VrBgB8FqGhGxo3bpxCQkJ06NAhrVixQg888IAkKTg4WDfddJMmTZrU5jHDhg3Tyy+/rIaGBl4UAKAb8PLy0vDhw7V//37l5eXpzjvv9LRt27ZNxcXFeuSRR+RwOPT666+324fNZpPb7W63zeo1AwA+i+VJ3djMmTNVVlamLVu2SJKSk5O1fft2nT17Vi6XS1u3btXhw4clSVFRUXI6ncrPz7/kd6MAAF1jxIgR2r59u9xud6s9bFVVVZIkwzB07NgxFRUVyTTNNo+PiIhQUVGRqqurVVxcrP3793varF4zAOCzmGnoxhwOh+bOnasXX3xRCQkJio+P10033aTXXntNNTU1SkhI0KhRozznDxs2TCdOnFBgYGAXVg0AuFh9+/aV3W73bIBuMWHCBC1btky//vWvFRsbq8TERDU0NLR5/KBBg7Rv3z4tWrRIERERrYJHZ68ZAPBpXhVV1W5JraYvW75uedfCNE253W4ZhiHDMGQahgzTkMvVJFdjowzDUH1Dg0an8J/NV1l6erocDoduuOGGri4FAAAA3QjLk74BTNPUhQsXtHv3biUnJ3d1OQAAAOhmWJ70DXDixAktWbJEt9xyi8LCwrq6HAAAAHQzLE8CAAAAYInlSQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBk7+oCcGkaGhp1obRcNbV1XV0KLAT4+ykiPFS+vo6uLgUAAOALIzR0Iw0NjTpxsliRPcIV1adnV5cDC+UVVTpxslj9+/YhOAAAgG6P5UndyIXSckX2CFdoSFBXl4JOhIYEKbJHuC6Ulnd1KQAAAF8YoaEbqamtIzB0I6EhQSwjAwAAXwuEBgAAAACWCA0AAAAALBEaAAAAAFgiNAAAAACwRGgAAAAAYInQAAAAAMASoQEAAACAJUIDAAAAAEuEBgAAAACWCA0AAAAALBEa0KF9H7sU/eRZjfx5iZrMzs9vMqXoJ8/qjxtrOjwn60ijop88q+zjrnbbx//PeUU/eVbRT55V3yfP6prfnNerW2s/71MAAADAZWDv6gLw1bUyp16+di+dqzKVWdCgm4f4finjDu/no++P91OTKW082KCfpVbJ18dL88b6fSnjAwAAoDVmGtCuxia3UvfW63ujnYqLtGlFdt2XNnZ0mLdmj/bT3LF+WvL9UMVH2vTWri9vfAAAALTGTAPalZbXoIo6U7NGONUzyFvPZdToQrWpiMBPcmZOkUu/fr9K+081qU+ItxZMCZT9MzH0rew6vZBZo1PlpoZF2zVrpPOS6vD2ksIDvNVotK7t+YwaFZxpUrDTS7clO/XU1EAFOb30/zZU63cbarTjpz3UP9ymynq3kheeU1ykXZuejJAkrf5nveYvrdDf54dpfILjc/+MAAAAvimYaUC7VuTUKTrUpnFxDn1nlJ+aDGnVnnpP+6EzTZrzYpnKat36zbeDdP8Ef/0stUrGp/Y+vJVdpydXViomwq7fzQ7W+HiHfrG6utOxy2rcKq0xda7K1Bvb65T7kUtThjYvjXpvX73u/2u5Qvy89H+zg3XfBH8tz67TvMVlMt3StOTmULIpv0FS8/KmJqO53sKS5uSRkd+oiEBvjY0jMAAAAFwMZhrQxpkKU5sPNWruWD+V1ZoK9vNSTIRNb2XX6cHr/SVJr26rlcMuvfPvYerxr9mHkf19NO2PpZ5+Xsis0ehYH/31vlDZ/hVPg/289Kv3Ow4ODU1u7TjWqKSFJZ5jd4zx0yOTAiRJf9hYozEDfLT0gTBPn4m97Lr3tXJtPtSgGwf5KqGnXRn5jfr+tf5at79etyU7tbOwUWv31+vRSQHKLGjQ1GG+nscDAADAGpdNaOPvuc37B97cVaekhSVKWliijy4YOnSmSR+ebL7r0clSQwN72T2BQZJG9PORw+7l+f7jMlNj4xytLs7HDrB+d99h81KQ00sv/1uIJCk61KZnvxMsH1tz+/HzhsYMaN3nuH/NGLTMJNyW7KttRxtVVmsqs6BRM4f7aspQX63b36B9H7t0vtr0zEgAAACgc4QGtLEypzk0LH84zPPxl7uaL+JX5DQvUeoXbtORs006X/3JeqR9H7vU2OT2fN8/3KZdhY2tliztOt5oObaXlzThKoemJTv17qPhOlNpaNG6T2YmBvSwKft46z53Fjb3GRfZoNcrtQAAAvVJREFUnCymJTnV0OTWr9+vlumWbhzsq2nDndr3sUt/21mnUH9v9jIAAABcApYnoZWcIpcKSwwtnBGkCQNbX1iv2lOvd/fUa+GMQN17rb9W5tTrW38u1fyJAaptcOv5TTXy+tT5j0z0149XVuruJWX6ToqfCs40aUnWxf/NhTEDfPSDyQH6Y3qNbkh0aMJAh350c4AeXlqheYvLNGOEU6fKDL38Qa1SYnw0MbF538OwaLv6/2s51ZRhvvJ3eOnaBIdC/b21PLtOd4zxa7NhGwAAAB3j0gmtrMiuk81bur2duxx9N8WpijpTaXkNSuxt18r5YQoP8NZ/p1bp9e21+tWsIAU6P4kNd4zx0//NDtbJMkML3q7UrsJGLfpO0CXV88TNgRoZ46P/eKtCpTWmpg936tV7Q1VZ79bP3q3Sm7vqNGeMn5Y9GCbvTyWWaUnN9U/912e7t3TrvzZTT0v+cv7eBAAAwNeFV0VVtVuS3O5PlpW0fG2apuez2+2WYRgyDEOmYcgwDblcTXI1NsowDNU3NGh0yqgueArfHEeOfaSB8TFdXQYuAb8zAADwdcBMAwAAAABLhAYAAAAAlggNAAAAACwRGgAAAABYIjQAAAAAsERoAAAAAGCJ0AAAAADAEqEBAAAAgCVCQzcS4O+n8oqqri4DF6m8okoB/n5dXQYAAMAXRmjoRiLCQ1VyvpTg0A2UV1Sp5HypIsJDu7oUAACAL8ze1QXg4vn6OtS/bx9dKC1XyfnSri4HFgL8/dS/bx/5+jq6uhQAAIAvjNDQzfj6OhTVp2dXlwEAAIBvEJYnAQAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS4QGAAAAAJYIDQAAAAAsERoAAAAAWCI0AAAAALBEaAAAAABgidAAAAAAwBKhAQAAAIAlQgMAAAAAS/8fg6zMCYhZkJsAAAAASUVORK5CYII=)

#### Context Keys to Add

Using this table you can specify context keys depending on your experiment's [randomization units](https://docs.launchdarkly.com/home/creating-experiments/allocation#randomization-units).

## Advanced Event Settings

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAvgAAAB3CAYAAAB2WzTHAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7d1xXJX13f/xl3jweGQH46RHA+ngOOoBD00wMaiBxVKbOG1Tlpluo1ZrtuWye5m/2zlrt9lmq7Zcd8250pRNrciwJMOUFRSmlCAd9XAnqczQDsqZHoHD8fcHiKioWBl5fD8fDx95zve6ru/nXNc5Pt7X9/peV10O1nmP0eLYsea/BgKB1teBQIBAUxNNgQBNTU34/X6amppobGjA3+SnoaGRoYlDEBERERGRzhfS2QWIiIiIiMiXRwFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIobOLuBiV712AU+s9+K8dSZTh5jOuqzv/cU8vGovybfPZvyAr6jAL8FZ6w6UkzN3CaW+U9cyYBtzP9PSLV9RlWfio+qdNeQXf8Quj4/Qy6JwJKUzaoQTy5f47a/517M8U9idUT+fSnLEl7ddERERkfOlgP9FBKopq6gBoHKrC9+QRM4e8YOY1UnG0Kg2X6hQzP0v8N5oObkoGzCJh25LbPfLXP3WYp7J30t4bDLpQ8007i1n05tLqKrN5t6JjnMfr/b62JXH/P8twjL2fu68tvkExmAyY4kw0UO/KBEREelkF30c8e0qIP9jO6Out7UT1nxUvZWPu/8oMmIuQNjcs4WyGgNWazg1lWW4DieSGPbld3MxMFgcpF2f/PU6wQlUU7a5Cn+vNCbdnoktBCCVqCXzydlaSNloB8nmL6cry9WTmHb1l7MtERERkS/iIg/4HsoKiyiqKGRv4xSyR9rbBEwf7jcWs3R9FYb4SBJjkvmyJ4tUlbmoMTqY8F0LhUuKKP3IS+LVbRKjr4rCl3PZ+NGn+CPsJEaemMfieedZFrzqIfmnMxkfCwSqyX/iCTYaRjH9lxlYvW4K166npGIXHr+JPoNSGTMuA7sZ2FfAk08UYPp2JpHVRWz62IPBmkBG1kRSI1sO6SEX+a/mU7L9U3wGC/akDDJvSsTa0uwpzyfvzRIqP/Nh6O0gddTNZAwyn7Pu8+Oj9IX55Ox0MGnmJBJNQG0Rz/4hF0/KNGaOtYGviqLX8iks30Ud4cRcdQOZY5KJ7Abe4md55BUfyWMc1G0qwVXrxzIgjYkTM7DV5vPknwrYGwC25jBrjoupcybhPPWukhDA58XrA8IATCSMycZ8ECzGE8ep3RoOnN7HuO94WfOqGz/geWU+s7ZPYHZ2MjVrFrDwHROZM6aRFuEm99FnKYkaRWaYi8Kte6nrHkXyTZMYP6TlW9hQTdFLL1OwbS9+cwyJV5lxbSjDdts8JjmBWhf5r+RTUtl8/GKuSmvdLyIiIiJnc5HfZGsh+ZapjIo1ULV+KYvfcNMcRduE+9hR3H7Llx/uCVRRtq0GQ38HCQMTcFj8uMs/wtu6gA9X3hLytnqwXJVBmjOcvdur8B+vPN5JTIgHt6uq+Y0DbtwHoE98AlY8FP1jKXkVjcTcMJ7xN9hprMgn59VyTkRtP+53i/BEJpJ2rZ3QfaXkrd3S3H+gmoLnl1BQaSDuhtFkXBXO3ndWsiS/uX9/ZS6Ll2+kOjyRjJEZxIVUkr88h6LPzl33GTUepbbWg6flj9frB0w4nLGY6t1UftK8BV+li10BC454G+ChKGcxueU+7NeOZvTVVjybVrFkjftEf4G9lLxdRXhiOumDzHjKC8h7twYiEhlzWwZ2Ixhi0ph6a3rLCH0bIZEkDrNjOlTK8j89S+7GcqoO+TFcbsMea8PSjbPX0E4f8c5R3HqTE3MIRA2fwNTvxJ3xqoW/opBNjXZSRyQT46+i6JV8yhta9vGrS8j9oIbwwelkJFmpKS3FEzi+pofCFUso2GsmedxExqdY8Ly3iiWvu899HEREROSSd5GP4APdbGT8aCo8v4T89UtZHJhIOhtZuaEl3P8o48KMen5SRrnHgD09DpPBjGOQhcL3y/nIm9w87cPnoqTci8ExgakTkzEDyWEeHnn10+b1Ixw4+hlYW+mmBhumShd7A1Gkx1sBLzFpk8g2ReG40gzEwcdlrKpyszfgxN5SgvXaW5k6OhLwYNqzgNx/V/NpAMwfl1BUDfZxU5mQYgaGYfE9Sf4eNzUBK3veKaHGnET2D0djDwX6+9n71EZKKzykXl119rrPwF+ZxxOP5LW+NvRruRIxIIFYYznuHXthUBSVO6rw90zEaQM+KaJoRyP2myYxPsUCJGI+tICc8g9wj7UTBRBiJun72YwfZACfmU+35+D6dw1+kxN7fBSbQoDwKByDItv9Mlu/PYW7uq3h5Te3ULTGTdHrZqKGZjAxM5VI07lqmICjnT4i+lvIw4Cpr73l+LTPcGUGU25JwwLY6908saGGTw+As1clpVs9GGLHk31LKmYgtVcdv1nual4x4KGmxo8h2knq1YmYsdPHUkpVSCiNBMOPVkRERC6k4MgKJ4X8JSwBzAMuYLjHj/vDcjwhUST2acTzmYfwqEhMxS7KK7wkDzdDrYfaegN9bDEcj4AGY9vdbcE5OIq1+ZW4a1Mx7dyL35pKQl8AM5FW+Cg/h7xln+I96qOx3g8nZUkDptbtmTAYgYAfP+CtqcFHOJFR5tb2xFtnkggQqGLLPj/UlrB4bslJn8py0NuButtn6JfKxBscmI6PohutzVdNwhw4+hsor3RR3eDD9bEPs8NJjAG8/67BE/BTs2YBs9a03VgddfU0B3xC6X68f2PLTaz+xvMIuiYih09g2tBRVLvK2FJSSNGmXJ7xGrg3O5nQc9TwhW4q6N69dfXuJhMGfPgDgOdTahqhT/82+9gQemK9kBiGDbdRtn4VC/74AY4BsTi+lUzaWU4mRERERI4LjoAPJ4X8IlIvYLgH/Lsor/CA30PBX+ZT0KbJvbUM7/BUzC1B13CWSVCWuASi8tfiqijD9HEj1uEOIkNono///BIKGhyM//7t2HuB66WnyKv5MopvxN8ERKYyaXziSVOXQntGQb37nHW36xuROJztPZXGRJzTjuElNx9tbcTtNRPnjGnzxTPhGDOFDFvbgGvC8mXcrXu4mvKdn2Lo68DR10ykM5VIZyJRL8wnp+IDXLXJJFzoGtoTAmCArmf6+RmwjbyL+x1llH7owrW9kJx3Ctl00zTuTLdeoKJEREQkWARPwIeWkH8/qZgwXcCbEf0fl+M6ZMB27XjSY3u0vv/p+y+TX1lO2aFUUntaiDD6qfx3DX6sGKA5WLdldeDou5aCwvWE1lsYFm9rfv+Qm6oaP5aUdFIHRQJe3OdRn9lqJZxdVO/1wpVmwI/7zeUUeRPIvNmOtZcB9nhojLBh69m8jqe6BlOEAXwdqPs8mQc5sXddxaZ1HjxhcTj7N3/tzL0shIe48NRBVIyt+cvoq6H6qBVzCG3uZziTc3x9j7rZuCKPmvhJzLzt+CNMQzGZDEAj/sC5ayBwAX4il/XBavRT9vEuvNdbMQO+hsYT7YdcFBS6MSVkkDY2kbQx1eQ/9QQFpWXUfDsD60V+54yIiIhcWMEV8AG6mS7woxr97CovwxMSRep1yTgvP9HioIwiVxnl5R5Sr3WQ7LRQXrqWlVc0kmD2sGWjG/9J1VlJiO9D/ht78fdKI6Ffy9thVixh4P4gn9xwB6Z/l1JU6QeTv2M3WfZPZlhkCflvLiH3aCKW/5RTVLwLw7XphGMmMS2Jwr+XkPu3JdQMjcHkcVG0qY7EO6eTGdORutvZKx4XhRuPnrRUd9swkmNMYI7DcaUB104v5iQH9uMnX/1TSe1fSt47OTzTkEpCLz+7NhdRedkY7s9O7sAHNWEOA3/VJta+FUpyuvPk8Hv5MDISi1iyOZeFS/aSaAvH73FRutmLoV86jggg4uw1mNvro4cJc4ifvR+up8A0jNQkW0eOygndHCReZaH0vTUsWVlLQoQP13vl+I//HE0GvB8VUrDdg29EEpZ6N+7PwDTA0nplSERERORMFBfOl9/NBxXe5mkfl5/cZIhNwG7ys6u8HA8mHJmTyHSG4n5zJSvfrCQ8Nua0mGyNdxIVApZBjhNPgenmYMwto3CG7aUkv4Cyow6GxZuh3oPncAdqDIkk40dTybD5KVufx9oPfFjTbiV7VPMItWnQeO68NQNHSBVF+Wsp3AlxN00kI8YAHaz7NDXlFKzJI6/Nn+IdtS2NZhKcMRhCTNjjY0+cVYZYSbstm/FD++AtL2Dt+hJqLcOYOC6JDs02D7GRfEMiVv8uit514Tnt7MeE4wfTyL4pgfADpRTmr2VjeR2WxEyyp6Q1nwycq4b2+rAmkz4sCsO/t1BU8enneLKNAceYqYwfYsWzdSMFW2qwDrSd2C/d7GT+eBKplhqKXlnOyjdcEJfJlHGX8P9ITURERDqsy8E677HjL44da/5rIBBofR0IBAg0NdEUCNDU1ITf76epqYnGhgb8TX4aGhoZmjikc6oXuVg1ePE2mTG3JHbPxoUseN1L6s9mkhnTqZWJiIjIRS74puiIfO35qcp/hsU7rQwbZsdSX8Wmf1VB3wyS+p17bREREZGzUcAX+coZsF2XSdrhAjatz6MuEE6fARlMHTOKSP0iRURE5AvSFB0RERERkSCim2xFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IfHFNjTQ2dXYRIiIiAmDo7ALkPOwr4MkXfIy/LxPbhTw1C7jJXZCPeeo0Mvqe0laew5xiG/f/NBXzBSxBLgK123lzzZsUfrCdqtpGAEIjookfkszoMenER4R2coEiIiKXJgX8jvCXs+ThjcT8chppl3d2MWfgL2fJ3CWU1wMGE5a+sTjTRjN6iPWSOcjenUWUkUDqgM449fBR9f4mfP3TcFwOBGoo+MsSvDfdz/jYC9THoRIWP1NJ8i8m4TR9WX10zP73X+DPi9+h2jyIlOt+wE39wgnFz8E929lcvJpHi0r4TvadTLm611dbmIiIiFwy2e/SEGJn/IN3ktrTj/eTEvJWLWZ5YBpTky6NsXbPjiKKiOq8gF+ykRpzS/gOsZL8vYk0nnoF5Mvso2cco38QQ8RXHO6PfPgCjz5TQuh1d/LIrbFU5y7mb4u2c4hw4m+cws9+N5qdyxfyv888SUPog9z+rR5fbYEiIiKXuK4zH5z121PfPHbs2El/b/snEAg0/7epicCxAE1NASKv+FJTzNdPoIYPC6u4bHgyttOyig/3WzksXf4Sr7/1HjsP92LgwF4Y8VG65Hesa7qGb10RCngo/MsCNoWnE9/LT/nyueTuCrDj9WXkrCnE5b2cOIcVI4C/hpKXn2PpylzWFbv4zBTDwMgwQv7zMe99UEPjZ+/xUs6LrNu8l65XDsYWHtJS424s1wwlukcIxsuiiet9gPw39hE93M5lXcBbkc+ypTnkvlHI5l1+rAO+iaUbzVN//vwmO3au46WCg3xzeC/2FVdi/FYy3+zhoWTZk+RU9iIxrhehNeVs2HMZqd86zKuPPMf/xaQysCfgK2XJvJc4lHD6PvLtKWLVc8+xKq+QrdVduMJh47IuVeQ99jwfX3kN9nCapwX9YSlV/a/BbvZTs2U1y57L4cX1m6lqsOKItRDKmfabhZ0rf8+yzQc4tKeczTWXMdR5BaGnHKeqf63i+eWrWL3+PSq9l9F/gBVTl+Zt5pTu473Vq6gIu4bEyC5n6B/8NaWsXr6MlavzKSzbR2h0HNFh/0fe44t5e5+Xva7NVBnjSYz28vbSpey0XofDAgQ8lL+Ww5KcleS/U86+Y1dgj7mMUMBb/Cy/f+sAR0tzWfpiPu9VNtA33o6l7QcIuE/v47IdrHxmMz3SErAec5M7fymlB3ewfsU/WV1YTq25F4Etq1i87EXWvVvJYetABvYyNm/vsJuCfyxl2arXKaz4DNOVA4n6RgfmfdWXs+TxF6m+6g5+c3siXd9ZxMOr9xM//ha+56jn/dfW4b48gx98L4XIfRtY+ZaHb6ZfRV8NJYiIiHxldJPtF+SvWMOqDy2MmT6XufdlYt6WS8FOfwfWbGRvlY9hP3mQh6ZnYNpaQFF1c0v1mzms9w0j+8F5zP5JIkfWrWTjvpbVDlbj7ZvJvbNnM2VQLQUbyvCdoQeDzU6kt4q9XsBbwsqXq7BlPcjc2dNIo4g171SfWPhIHeHJdzJ7etv5/T7ca5ayviGNqeMcnDRQbLDjjPVR6Wrehv/jSqoudzSPLLflc7FmeRGhN0xj9pxppBlKWLmuCkJsJAxqxOWqaV5ujwt3NwcJfcH/yVpy3qxjyO2zeei+TKwVq1hTcXyftrffDDgnziT7GivWa7KZeUsipw5q+7fnk7MplIyfzWbe/ZOIrX6Z3E3e1m3W+aOYeN9sspNMZ+m/hqJV+dQ5p/Lg3NncnuAh/9USPCF2Mu+bRkY/M0m3zCQ7xXLasajZmMPL1TYm3j+P2T/PIHRzDrlbTxw5X9VeTCPuYvasuxjWWMT6LZ6TN9CBPgh48BjTyX7wIe4fY8W9ailFxlHcO/shpqUZKM0voXlveyl5cRUu6ximz5nNXUPrWP9SUUvb2R3ZspHiI4P4wcREegIHG01cPeY2fjZ6GCmjb2Osw8/Oik+AcK6e+F3ij5RQsOVIB7YsIiIiXxYF/C/IYLuB7J+Mxh4GmB04+vrweM4UudsKxTYsHXtPA4bLHdgvr8N7EAhUU+ZqxJmWjLUbGPqmkjkukajjR+oyJ6nDIjEZTNgHRmGo8+I9UxcGAz0MjfjqAVMcmT+bRHo/A4RYsA/og8fTJkT2dJAYb8HQOtLqp/a9HFZV2pjww1Ssp31TDNiddupcH1ETgF073YQ7ErCespR/ZyllPVO5wWnBYLCQmOqASjc1AbDFO2jc4aIGqHZVEhqfQGQI7CothyHfIdlqwGB2kJpoxr1z19n32znsqnBhHpqGM8IAYTbSx96M09LYus2owYlEhhngrP1bSJxwJxOHWzFgIHKAHYvXgzdwjs4D1ZSWekm4IQNbGBginIy61orrA1fryVnogGRSY8wYTJHYv2mm7uAZj+qZhVhxXGXDbDBgiXMQGWpjyHAbJoMBq8OOxVtLXQA4VMYHH0eSmm7HbDBgHZaKvdaF+1AH9uOOT+BKJ0Miml/bMrL52dhBzVdLakrZ+glExrRc0YtIZOiVjezasfv8P4uIiIh8brpw/kUZoeZfK8ndUYMP8NU2Yh30ObbT1U9z3DyCz2fCEnaiyRKfigVgHyefkoUAgUbOyO/jiD+USCNgMOH/eC3L/1lJrR8aD3ug/1nqadpLyfsGTOYk/Gc4DTTEJmB/eSOuA3bqKk04sk6N91Dn9dJYlceTc/Ob3whAo3lYc7Dt78B+ZD3uzxz43eD4XiTgx+ut49P3n2FOcctGAo0Qd5R2r4u07rez8eM97MMUfWJuviHSSXJL22nLnrF/A4YjLtY8V8reOj8E6vg0kHjO3qEOr8+M9bIT75gvC8fwHy8+aJ1KdPzHGEoonOuk4Vy6AiEQesov3A/g9VJX7yL393PIbXm/MWAlwgf0PNtGGzlyxAfmnpw2U+3AO/x5/j/ZGftDZmccv7HWRJg5lCNHNIIvIiLyVVLA/4Jq3skhzzOMO38+CYvBT+kLD1MKQCiEgP+8g1oPTCYfdYeBlukuvn1VeMNsp42On4vPXUl1WCTpZuCTtSwv9JP503txmMHzzrM8edaBVQvJk7JxbF1Mbp6Leyc6Tpv2gikWR/+XKfrXJvzEMj7y9K2YTN0JdUxk5tTTp81ADM4Bdax/twQaHIyKBDBgMpmIGjmNad8+dRpKR6Y+tceAOcyE77APjlfhrabKF4HNeuqjHM/Sf4OL3JxSzJOzufdKE1Tn8+QL5z69gHDMYV7qDtJ6TL0H6/B/IxbTF/hUn1uYiXBzEpn/NQHHef0LEEoPswmqDnGEk88F3AVred+YygN3XEvvrsffPcTBg430OP3GFREREbmANEXnC/L9xwcB8Df68OwqoWx3Y0uoN2DpaaZmhwuP34+3chNln3YgyoVE4nCEUv52CR4/+D8rJfeFPMoOn0dRAR+enYWszPsI67dTsYWA/z9H8QXA7/fh+8xNacWnNAbOEi67momwWHGOzcT+cS5rytubdmTCEW+jZlMJvgGOdp/NbxqQQOzeQvLLPfgB765C8t6pbunXQIzTjufdIurarG93xuF9L5/SfX4I+Kh+L5+Cneee9hQaGor/iBdfOydVMfF2vJsLcR0CfNUUrlxCwcftf/oz9t9YR91RIODH562hfLMbT9PxKwihGEL91HnbmVoTEkmC00T5xkKqfMAhFwXFNTiGtHPSdPZPeOY+zkeEA4flIwrXu/EGwF/romBNCTUdOBmNGRBL6O5SNh04+f3ew3/IfXd9l3hjmzcPbGPrbhMDBkR/sXpFRETkvGgEv6MCVeQ9+mvyjr8OiWLU9HvJuC4T54o8Fs7Px9w/kYT+fahq8AJmbNeNxrHsZRb8JheLPQFrRMd2t+07k7jhlZU8+z9r8JmiSPzORNL70jxF56w1usl95NfkGkxY+tpw3nA7o69uHoU2ONLI3L6SvCfm0xjhIHmAjT61R/DB2f+HVWFOxmS6eDIvl/L+k3Ce0mwalECssZpwR0z765sTmZjlYeWrC3n4n35MfRJI+35q6xfPYHPiMLsxJNhaVzEMGsPUA7m8/PeHWXnUQJ8BqYy/+dxj3ZHORCx/X878FyZy/9TEkz6XYdAYJtXkkvvUHHICZmzDxjNxmLndbZ6x/7AkRo+uZOWyBRSFWEkYasNmOsrResBkxpEYS+ErC3jWN407v31KbSOmMGrNKnJ+vxZftz44Uicx/qrzfb7lKX0MOc/VW1lJy5pI3cu5LJjjAXMMiSPHY+nA6X6PpHRSX3qC1f/cxLBpw1pG8RvZ+fqz/C8/YdHdx6cs1VH8z9eo6JnMA0kawRcREfkqdTlY5219Jubxx2MGAoHW14FAgEBTE02BAE1NTfj9fpqammhsaMDf5KehoZGhiZ87acjFrqGcnKdcJP7yfKd7yMXqyLZ/8PCfiyDph/ziR9cSaYQje7ezi2jio3pA/T42Pr+QpVsMpP/iv5gyWAFfRETkq6SAL5+f30vVv3JYefAGpt9s1+WgS8ihshf507Nv4g6N5urhiVx1ZS+6cYSDn2yn+L1Sqhq/yU133sUtCeGdXaqIiMglRwFfPicvJc8tIP8/idw8ZTzOsz59RYLSf3az8fV1vPvhdnYdqOMIJnr2upL4bw3jppuuxfaNzi5QRETk0qSALyIiIiISRPQUHRERERGRIKKALyIiIiISRBTwRURERESCiAK+iIiIiEgQ0ZMNO+hofQO1tYc4fNjXejOyiMipunTpQliYiYiInnQ3duvsckRE5BKkgN8BR+sb2LNnH717RdDHejkhIbrwISLtCwQCeL2H2bNnH/369VXIFxGRr5ySagfU1h6id68IevY0K9yLyFmFhITQs6eZ3r0iqK091NnliIjIJUhptQMOH/ZhNod1dhkichExm8M4fNjX2WWIiMglSAG/A44dO6aRexE5LyEhIbpfR0REOoVSq4iIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgogCvoiIiIhIEFHAv5DqtrFi9m2MHB5HnHM4IyfPZNHb+zu7quBUsZAJqdksrersQkREREQ6lwL+BbOfFTOymVcRyx1PrOLVl55m1nWHWPqzbOa9X3/u1Zu2sTAzjtuWneGE4FztF7n616YTlzaH4qYOtvdLIevH40nq/ZWVKCIiIvK1ZOjsAoLW/rWsLu7OuL/OJSul+S37wEeZsS2N3+RsYMbVozB2boXBJTyJrDuTOrsKERERkU6nEfwLxWCkG4eo2dd2hD2cUTOXsfiupJZwX8eW56czIW0Icc7hjL1nIcW1QMVjjI0by2MV9RTNHk5c1iJ2t932Gdvr2LJ4OhOub97eyOx55Fed+WpBXdlSZk4eyRBnHEOuv405L7upB6hdQbZzLI9VnFh22+NjiZu6lP0AG2YyZGg2M2dNYLhzODPfhi3zRxKXNZM5U9MZ4ryb1UeAI+7mKUpD44gbPpLsP2xgd8uI+5b5I4m7exGrf3e8fSzT/7GNeupZfU8ccfespn7PUiYPGMLMDW2rPkN7xWOMdWazYj/AbhZlxXHb7xYxZ2rz5xueOZ2lZbspfvpuxqbGETcknQmzVrfWA7D/7YXcPW44Q5xDSM+aydKyDlxpEREREfma0Qj+hRIxmqk3L+Lu2ZO5230HU78/mpTYcIz9BnN8nHn3suncnRPOPY+u4vHL97B6/gNMnx1N3lMzWPXB9TyVNZnS77/B3yZHnzzaH99+u/v5n5O9qBs/nbeKx/vXUbxwJg/cUU/4i3NJCT+lvv2reeCux/De/CjLHo2HTYuYOffnzOv7KnMdHfh83mK2dZ3LEyuSsMfA7rehvmwLh2bOZ9l/24nusZ/8GbezsG4Kc3Meod9nG3hq1nQevPxVXsiOBqB+w1Jyfz6Lx3MeZH/+I0z/3WO8krGYrMc+YETGdIb/wcrTa2cx4qTajXyvvfaK00ssemMzIx56nFUz95O/4AHmTRmL/bp7mLFoBr33rGDOr3/DvGEjePrmcKhYyN0z1jJ45tO8mmSkImcOD/xyHtFr5jKiR4eOuFyk1q1bx7p16866zI033siNN9748A1LHwAABoBJREFUFVUkIiLyxWgE/4IJZ8TDq1j2YAre1+Yx+cbhDM+ayaLilhH9pi0sfW4bKb98iCkpdqIHjmDar7Lo+fbrFNeB0WikO4CxO8Z25vKc1t5UzIrnykn65aNMG2En2pZE1sNzGed/hb++cfo8/d2vrWCDZQoP3jeKwf2iGXzzLGZNtlKzY/dpy7arxyju+e8sUuLt9G4JwMaUO/h/P0ph8MDehFe+wl8LYpk27w5GDIzGnjKFGT92snltfuvVCOOwO3j0F6MYPHAwI27PIoVKtrkBoxGj0QgGMIa1++HP3t5i8KQZ3DFiMPb4EUybej3hfidZc+5gRLydwSPvIWtYPZXb3UA9G55bwv7vzmLWzUlE2wYz6lczGFe/lrUlGsUPducK7wr3IiJysdEI/gUVTtLkubwweRa7389n9XNP8VT2ZHY/vYq5g9249+yn+NfDifv1iTXqmwZTsx8IO8+u9rup2B9N4lVt7jI1JjF0cD0bXLuBk+8+3ePeidE+BXvX1oVJ+a8XSAGo3XLu/rp2x3jqt6crrVca6ivL2XakmDnXxzGndYF66nvHs//4tBij8cSVCaMRo+Eo9We4qfbzMHZtE/6NRrpjxtp6NcBIdyPU++uB3WzbXsfuHdkMWdlmAw2QcqAedLdE0Dse4E8dyVe4FxGRi5EC/oWy382W/UYGx0djxEj01d9j2tUpRN99Iw/8cwMz5tTT0NVO1h//wh2D2gTIrt3pecXn77b7qUe0CWjqhFFoPxAxirnLZ5DStiZjT6xdYdtXX9GZtZxUJN25mMe/3++kpu69T53bJMHq1JCvcC8iIhcrBfwLpO7dx8j+rZFH33qCUa0ZsTe9exlh71Hqe9uJDd9N5X4j0SOjW9errwdjV1pDZ4f1thPfezebt+6HgS2j9fVb2OyC2HT7aYv3sw+gfmU57qZRDG4ZxXe//Bj5YVlMu6Y7RuqoP9JmBf/5nSQYY6KxH97A/vpoomNbPx319cav33h4VyvR0UbcVfvpaUvh+OGqr69vd3qUBK+2gV7hXkRELlaag3+BhGdMJeuKfH4z/TFWF2/DXbmN4lVzeOy1elJGptC7awpZUwaz+Y+/Yt6r29i9x03xs3cz9taFuJuAruEYe8LuD4vZVlXHafH61PauKWRNsrPlT79h0dtudldtYfXcebzCOKZ+9/SHw0d/N4sRnhU88scNuPfsZttr85j5u3xqzFYIi8UZU0N+zgq27HCz5bXHeGSV+/x2QHwWU5J389dfz2TF+252V25hxa8nMPYPxad/lnYYw7sT7qlkS6mb/XXn335+whk1OQvrm//D9Kc3sK1qN9veeIzbR9/N0j1fdNtysdHIvYiIXOwU8C+UHinM+vsy7onaxl9nTGZs5gSmL6pk8H2LeeKW5hH7wXc/zeI7+lH8h8mMHD2BORut3PHwHS3z4qMZN3kc4QUPMHnWCk6/9fX0dvudT/P0JCOvzBrLyDF389T+FB5aNPeUp9C06P09Hn3mHvptmsOEG0cy+fEKkn77F2alGKHrYKb89h5it81j8rgJ/Oq5/SReN/g8d0A0WU8sZka8m4V3jWXkD37FiqZxzJ+W0rER/GFTmJq0m6d+PJnHNrVzSnCu9vNkTJnF3/6YBWseYMLokWT/cRvx/zWXrH7nXldERETk66TLwTrvseMvjh1r/msgEGh9HQgECDQ10RQI0NTUhN/vp6mpicaGBvxNfhoaGhmaOKRzqv+K7HRXMcBu6+wyROQio387RESkM2gEX0REREQkiCjgi4iIiIgEEQV8EREREZEgooAvIiIiIhJEFPBFRERERIKIAr6IiIiISBBRwBcRERERCSIK+CIiIiIiQUQBX0REREQkiCjgi4iIiIgEEQX8DujSpQuBQKCzyxCRi0ggEKBLly6dXYaIiFyCFPA7ICzMhNd7uLPLEJGLiNd7mLAwU2eXISIilyAF/A6IiOjJ/gO1HDrk1Ui+iJxVIBDg0CEv+w/UEhHRs7PLERGRS5Chswu4GHQ3dqNfv77U1h5i/4Fajh071tklicjXVJcuXQgLM9GvX1+6G7t1djkiInIJUsDvoO7GblzRt3dnlyEiIiIiclaaoiMiIiIiEkQU8EVEREREgogCvoiIiIhIEFHAFxEREREJIgr4IiIiIiJBRAFfRERERCSIKOCLiIiIiAQRBXwRERERkSCigC8iIiIiEkQU8EVEREREgsj/ByliDju321arAAAAAElFTkSuQmCC)

### LaunchDarkly event creation time

In this section you can specify how to derive the creation time of the event, which populates the `creationDate` in LaunchDarkly payload.

The available options are:

1. **Set to current time** (default): sets the event time to the current timestamp.
2. **Set from event**: sets the event time from a client event property.

#### Event property name

![](/assets/images/07-advanced-time-from-event-32eeb82cf4abf42f8829f122c934cf6d.png)

This text box is revealed if the [LaunchDarkly event creation time](#launchdarkly-event-creation-time) is set to "Set from event". Here you can specify the event property to use in order to set the event time (in Unix milliseconds). For example in the above image, the LaunchDarkly creation date will be set from the device created timestamp (`dvce_created_tstamp`) of the Snowplow event (prefixed with `x-sp-` in the client common event).

## Versioning

![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAuwAAAB7CAYAAAAv3NUuAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AABzPSURBVHic7d1/dNT1ne/xJ+E7hG9wAgx00Iw6oRllYCfIoIkl1MQa1uCa6riSrXBrvMZT7Gnale3l9lp7rddy+uN02V27p/S2nCvtgsfYQpeoQRNrqKQaKiihJsJQJivBJoQRB8hIRjLfmdw/EiSEYFEGM+DrcU7+yPf7nc/3/f1M4LzmM+/5zph3Dx/pB+jv7yeZTJJIJEgkEvQnE1iJBJaVIN7Xh5VI0NfXx7X+OYiIiIiIyCcjY7QLEBERERGRM1NgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksYU2EVERERE0pgCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwP5hrCAbVnyLb/2wllDy1F2xnWt55FvfYmV9V+rPGwuy4ceP8JNNHVipGG9fAz959Eds2BVLxWgiIiIi8glSYP8whgef1w5HQwT/MnSHRXuwnViGE++snNSfd6yBfbID+yU2jFSMN84cGC/TlorRREREROQTdEEF9ti+Rmp/38HI68QxOn5fS+O+VK4iG3hmerATpj04ZCW9L0Rwbwymesm/PIWnO2Gch7KvPEBVSYpeDOQUU/WPSynLS0n8FxEREZFP0AWU4CK0NjXTvKuJzvjdVN3swfxgX4zQC2tYt7kDY1YO/txCHCk6q3F1Ph6zhdbgbsI35+AErI4gu4+Bc24+7sGXPFZ3C/XPNdLyVgQry4X38+UEbnBjJjuoW7mK5kv8FNo62NblpvLhxXiTYVo21dK4cx8Ry8Qx3c/C2xfim2JArIU1K2rouLaKR+/0DozftY26TU207h8c/3NllH/Bgx2Ibl3ND5+OUXirl57t2wgetnBcVUxFRSluE9izgUce30HeXQ9TOddG25PfZW1HPmVF0PZyKwf7ssktDLD4Vi/2E/P5+/VsejnIwaQDz/VeeLWJ8NylPPhFT4pmVkRERETOxgW0wu6g8K5KyvIMOjavY80LocGV9iFhPa+M++5KXVgHYFwe+VeZWN1Bgu8ObNoXDBLFgfdv3AMbom2sf7yG5iNOCm9eSNEVMVo3raV2Z/SDYax9rQSTbgqvy8MBdL24lppXe3AWB6j4Oz/ZnU08uW4LXcnTS+BoC08+voFthxz4b1pI8XSLYMMa1v5+yKp/spNtL3eQ7S+hZIadSFsjdX8Mn/m6DrfQ/KYN3+dLmTslRugPtWzZN3g5O9ZT09BGz5RCSm/MJzu0neCxc5lEEREREfm4LqAVdmCcm9J7KuE/1tKweR1rkhWUsIX1Lw2G9XtKyRmX6pOa5Pm8mDtbCf45SvH1EYJ7IuAoJv/KgSMiO7bQGnVR8t8qKM2xwXVOYm+vYdsbIQKzB18+5JRS+ZVScjIALFoORcDMZ+7nCvGZ4HY4aAtnY4wQ2MM7mggec1L01UrKcw3Ah3l0JbXNzQRvWIQLIMPO3L+vIjDDgJidg3tqCB4IY+Ec+UnO9FF+7yL8Jlg5h9n9+A66DkQhF1pfDxKd4Kfy3gA+E5htJ/LPtURSPbUiIiIi8lddWIEdhoX2tawF7Fedr7A+wLzKS57ZQnDPbqJXHKb9EDjmewfbYSy6DoSxkjEaf/YIjUMeZ9gjRAfX+40JJvYP3s8w8BYU4tzTzJMrw3i8eXhmzKVgfg5mBhAfenaL8IGDWJn55F5+4ulykDt9GnQcJHyUgcCOjfGZg/szTbIMwIoT5wxPcoYNW+ZgNZnjMYG4FYdklMghC+NS90A7DYBhXIB/KCIiIiIXhwszhw0J7c0UndewDoDpJf8qk7a9QZpbDnMQB4WzhvRyJ+NguCm9pxxv5tA6HTjOsC5tzgiw7Js+drzeRvDPbdRvb2KLbxHVlWdu6Tn1yRqM4olzurKRjQXGXph/GiIiIiIXmwuoh32YcW5K71nO8vMd1gEw8c7KwzweZMvWTqyJXnzuE/sMnM5pkIwQOe7EnevGnevGlWXDdNpHfkWUDNNSX0d9exaFCwJUfm05VfMcRIM7CR4dfrCB87JpGMc7CXWduCt7lH0dETCn4Uxpwz6Q4cAx2cDqbKfjxA13rPdTcz94EREREfnILuxl1HHmkDvFnF/mDC95tjbajoPD6yN3yMw5C4rxba2hZeNq6PbjopPWra1Yxct4oGSEwTJMiLTQtLWd948V47VHaX2rB+xenBMY1hIDzrnz8L68gebfPAnXezC7t9PUDq6bivAYnOE2lx+XnfxrvTT+poWNa00Oek0OtzYTskjth3lFRERE5Kxc2IH9kzRhJt48g7ZgNl5f7qkTN9HPkvvi1NU30fqHelrHOcidE+CO+U6gY4TB7PjvuI/Yc3U0v7SeHcdtONxzCfxDGR6D0wI7EwupuNdi06Ymtr8QhCwX3gWVlN+Uc16eQPvcChYfhU0vb6Ox24FnthdXVwsx9MVLIiIiIp+0Me8ePtIP0N/fTzKZJJFIkEgk6E8msBIJLCtBvK8PK5Ggr6+Pa/1zRrtmOe9iRI/asE8cfDmwv46VP2vGLFtG9Reco1uaiIiIyKeMVtjlNLG2Wh7bGMZzfQHuCVFCW5sJZ+YSmK2wLiIiIvJJU2CX05jeUu4oqKfx9QZao5B9eT7ld5RTNGW0KxMRERH59FFLjIiIiIhIGrtwb+soIiIiIvIpoMAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYBcRERERSWMK7CIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYBcRERERSWMK7CIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psD+aRSPj3YFIiIiInKWjNEuQD4Zve2v8Ex9M68F/4t3eoEMGxMv+yzXXL+A2xb4+EzmaFcoIiIiIiPRCvspLNqeWsHKTR2nbk6GqPvxCjbsslJzmmSYxp+upLY9NcN9uB52/faf+R8/eoLmI1OYF6jiG9VL+cb9X2LB1TZCm1bx7f+zhi2dWnUXERERSUcK7Kcw8M6eSWxPkK7kkM37gwQTM/FdnaI3JDKcFN5WQfEVqRnuzOKE1v+Uf/1dD7PufoR/eeBGbG8+w8//72p+vq6Zo94v8b0fVnPLJW2s+ZdfsvXQ+a5HRERERD4qtcQMY1ztY+bTDbR2l5GTM7Ct480gljeAxwCwCO+oo7ahhX19djzzAiy+2YOZDFG7spZIjknnXiiprqZ4apiWp9fTsLOTHmMa+TcGCNzgxiTMtmdqiN7yIIE8IBmh7fla6l8LETWm4f18gECJGxOLtidX0Gwvxt7eTOu74LruDipv92E/m4vZ9wyPv3CIWUse4hvFWby2+t/47f4rua3qNiYGf8dT/++XTH3kf3LnP91P/AePsfbX25lVXcDE8za7IiIiIvJRaYV9OMPDHG+M4K7wwO/JLoJ7LLyzczEAa389NS/2MOe+h/neN8tx7trAphOtMokosallLPtONcVOiO2spy7spfI7P+B7Xykg1lRLc/fppwxvqWFjl5uK5T/g4a+VYnu9hto3YoN743R2xCi499t8b1kp5huNNHedzYXE+dPvt/HO5X/LXV+YCvQSz5rBbUvu5s55BSz48m1cZ3ub3Xt7IWsGtwUKoPUlmrXKLiIiIpJWFNhPY5A720s82EoYoLuVYNKHf/rAmxH7WtpgzgIKnQaG3UuR305o776Bh4514p3jwT5ucKgMIBYlEolhXFpE5bIqipzDTpfsoqUlSv5NpbgngDHZR9l8J8GdQQYiuw13QQmeiQbGFC+eKT1Ej5zNdXQTequHz/h8DLxRMJV5X67izrnZABxt3UkoPgX3lVkAZOXnMyvjbUL71MsuIiIikk7UEjMCY7ofb18dwXdLsXYFifsCuDMALKLRHg6+9gse2Tp4cDIOM9/HYvxp45hzKqg81kDjUyvZ2DeNmfPLuHX+8GaWHqIxO85JJ7fYJ2VjvBclxgjGWpxVpE7E6e21MWnS6Q0uvTueYMXqHXwm8E/clnviorPJGg+HonHAdjZnEBEREZFPgAL7SDLc5M+waGhrIx6E/IB7cIeBaZq4bq6m+gbHqY9Jhk4bJnb4MPY5AarmB7DC21j/qxoaXQ9Snjv0qGzsE6L0HAGmDGyJHunBuiQP81yuYWwWE7Pi7IocBbKH7DjElk2v0HfNUr6x8IqT0fx4D0d7IcuusC4iIiKSTtQScwbua7zEXq1je9JHfs7J7R7fTKKvNtDSbUEyRterDTTuHXEtnMPb17N64zYiFhgTsjENsBLDDsrIId9n0raliY4YcDRI49Yw3jnecwvsTMGTl01Xawsdp5wzi2v+vpp/rPCRNWRrb2sLe7mCmXkK7CIiIiLpRIH9THLy8Y7rwczPJ2fILBkzbqVyHjT/cgUPPbqS9XsNPDkjR+ucGysoNraz+vsP8dC/1hHxBSjNG+m4uymbEqTmxw/xyM8aiPkXE5h9bnEdbMz6wnzc4Zd4qnHIJ10Tb9P4y9X89k9DXmT0tlFT24LtmgUUTT7H04qIiIhISo159/CRfoD+/n6SySSJRIJEIkF/MoGVSGBZCeJ9fViJBH19fVzrnzPaNctZi9Px7GOseOYQsyru56s3f5Yseuna+zbxqTNwTwYOtbBu1TpejPr4xnequE6BXURERCStKLBf9HoJbVrNv9fuIX6Zj6ICH1c6sxh3/Chdf25h62v/xRHnfO77+peY51Q7jIiIiEi6UWD/lOjt3M6Lz7/Ca8H9dB2NgS2bSa48rvtcCbcUz2CisrqIiIhIWlJgFxERERFJY/rQqYiIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNGaNdQDqKRt8j+l4viWSS92PHR7sckU+1rKzx9Pa+P9plnGbMmDFMmGAyefJExmeOG+1yRETkIqbAPkxPz3tE3zuGaZo4JmePdjkin3p7Qx1c5XGPdhmnSSaTRKPHePvtA1xxxWUK7SIict6oJWaY9471kpWlsC4iHy4jI4OJE+04P+Pg8OGjo12OiIhcxBTYh0kkk0yepLAuImfHbp/AsWOx0S5DREQuYgrsw6hnXUQ+ioyMDPr7+0e7DBERuYgpsIuIiIiIpDEFdhERERGRNKbALiIiIiKSxhTYRURERETSmAK7iIiIiEgaU2AXEREREUljCuwiIiIiImlMgV1EREREJI0psKdYrKuNti5ryBaL8K42OqKjVpKco9hf2mjrHvpNlgPPaeexUStJREREPkWM0S7gYhLZsZbVm6L4l+ThwwBiBDeuoqbDQ8W9voGDrAhtv6ujcUc7B6NxTKcH/4IAC2c7RuXJiLyymp+86mbpsjJc5/HlW3Tran64ySDw7SoKJwzdYxF8agVruotZvqwUZ8rOGKPliZU0mBUsv9N7TnNrvbePLevraL29msWz7UCc2DvNrN24jdL7qyiamqqa5Zz1H2D701tof3/w97GXUXh7CZ8dB8T+zOa6HYQTg/vMq7nptrk4x0D4tWfYHOod3GFwWcFtlOSN++TrFxERGYFW2FMl2UHT80EcCyopyzMHtnU30bDdRvGSAL6JAFFanlzFkzvBd3sVy5bdzx1zoO2p1dTuin3Y6GfN2rmGh1ZsIJQ8u+OzpxdQPM+L4zz/Jdh9PnKTIdqGX6cVom1PDJcvP4VhHcDEPaeIor9xnfMLIbu3nMob7bQ+v4WO5ODYJXdTfkUHDS8Gsf7aAPKJSvTFiMVO/CRO7kgm6IsN2dfXx4m9Cat3yGN66UskRhxbRERkNCiwp0q0k85jDtxu+webrO6DHMx04RpcgbXaG2nYZVJyTyWlPjfOS934bqqk3Bdjx0stjEbXjJHjp3SeG/N8n8iej286hN4MckpzSaiN3TEXPl9q4zqAw1dKsdf+1w88C3a3m+yjXRz8oHgT9+UO4t2dRFJyBjkXfT1hwgfDhLvfpfeUrB3jyMGBfQfeeY++obsSvRzpDhM+eIB3e08f78DBMOGDR4j1n//6RUREPoxaYlIlObjOmjFsm2F8MMmdu4JELi3AnzP0gQbeolLmtprEADsQ3dXA+vpthMIxbFM9FP1dBWWzBoJnx9M/4heHCgg4gmx+vYMemxPvgsUsmT+N1l89RM2ugVFXP9iKv/JhFvsMIrsaqK3fRigchYlu5i6oIFDgxAAiW1ax8nUP1d8swxXbxuoVTThv9dPzahPBcBzzyiIq7i7nRO6NBhtY/1wzoXAcMyefkkCA4itNiG1j9YoGjDluIruDWHPv58Evuodcp518Xy51m1oJxvz4TQCL0Bu7iV5aRP6lA0dZh1qo+20DLR0RsLvJ/6DWMI3//hgtk+aSfWAH+yYsZPnXizH3N1H7n1to7Y6C3UX+jYuomO/CwKLlV9+lzl7Fw3d6Bmr/WPM6uEKfARDHGrqcPtaApEX84/3FSAq9++Zm6oMjvEuVCPOnFzfxp5EeFOvgj/UdI+xIEG7dTH0rMPZy5lf8LVef91e0IiIiZ6YV9lRJxLGwYQyZ0XjcggywAWAROdyD4XDgGPZQI7eYRV/04wSs/XX84okWjILFLPtfy7i7ALY98Qvq9p9MitbeZnaOL2Hx1x+gstCkfdMmth018H/5Bzx6pxdjgp+qRx+mwmdAVwNrnmjBmFfJ8m8/yP3FdnZvrGFL+AzXkQyzbWsYzxeXUv3VAJ6jTax/MTRw3r80sKamlewbqli2vJpyd5jGdbUEj594bJSOcDal91RTdaPrtKEH2mLaad0zGKysfbTtiZ5sh4kFqX28ls6ccpZ+czlVNznY9/Ra6vd/cOWE34rgvvV+qu8qwGGF2LSunvD0O6he/iD3L3DSsWkNde2nN6l8/Hkd3Jlhw4BT2l9sBpAc/JELxFjMiZOwqz1dREQuIArsKWERfiPIwQku3CfSeDJC6+5ObC43zhOznAQM24eME6P1pe1EZ5VRcYMH52QnnpLFlHujbH/5ZK+0cXkJi27x4b7UhfemItwcpCsMGAa2TAbCZebgyv7kAhZ/fSlL5rlxTHTgnlfIzMyDdHadofM6w87c8gqKrnLhyi2keLaD2IEuolgEm5rpmR0gUODGOdWF/9Yy8q0gLScCcoad/LIA/lwXTvsIb97Y85kzPU57WzsxwHprJ7uPufDNHmiHie5sYoetiMCtPlxTnbivD1DqidD2pxOroAbOebdS5nPjmmrC8QiRYyauGV5cUx24rw+weFEpntNWQ89xXgGmuHGZYYJvhD843n6Fm+wjwWF3BZK0lmEnb/4CZk8dO9qViIiInDW1xKRAx7MrWfWHGL4llXgMgDCNP11JQ9hN+TL/B/3hNgOwPqSBIhmhMxzHef3QnnITt9tJ/PVOIgzeaSZz/Mn9Y20YGRbxMw1rZmPbU8/a37TRcaiHOGD1WXjPWIQNY/zJPwubzYAkxInQ2R0jGl7Dd18/ebRlWXiOxk8+9sNej2Bn5uxcap/bQftxL+PbdhPNKSJ/sH39YNdBrHCIVf97y8nxkxaGL4o1eMW2jCEnmJDPTfO2sm7tD+n0zsQ7w0fBnCLcmXDKWngq5jXTS/ntXh57YiWPvfcAy29xwZULWXTtSlb/dAXR//4oi848qSIiIiIfmwJ7CrhuvJvFsRrWb26mc3YZrgwnRXdVEnvqSZr/EKLoDg8GBvaJ2VhvRYjAKXdEsfZvoz6YRdGCyYPtMyOn3o/TK23tq2PNhg68d1WxeJYTkxAbvr+Gno86UDIOSQPXDVUsvv7UW1Da7OZZt4XYZ80h99lNtO4OYe6K4ZrnP2UujOnlVN/pO/VDsJnZGCN+tNPE88UHeHBeiOCbQVpfXc/KF10E7q+icNitFs95XpOdNL/Yinn9YhYXTxvYdqiZhp3gv6OKMs/ZDCKjZaztxIr6WMZmGDB2LGNP/DkkEiTU1iQiImlMLTEpYNhd+Iv9TDsUon0wV5pOHyUFufT8OUjnYBhweT3Yu9to6R76aItQcwPNoQhkTMPptBF+u2PInVRidHSEsTldTPsYtUVCHfQ4fRT7nJgZQPx94h+ngyPDgXMqRN6JYp/qwDH4k23Pxp75Ecaxz2SOO06woZbWY9PwzT7Z0e9wOuBQmKj95PiOSdlkj9ReA9DdQsPzLUQdHvwl5VR+/X5KzBBNO4Y16KdiXsNBgoemUVjixzVhoJ7o3iCdWT6KCtycqURJA2MvY+6tFdy16C7uWrSQa6aa5BWf+L2Ckun6RKmIiKQ3xYxUybRhECM2JAybmQZYJ+8iYlx1E2VX/4TaJ2owy4vxTrII79pC3Rsw98sFODDILvbTsLqO9a+MZ+GsbHra6qkL2ilYenZf/mOMM7HFDtL+VphpOU7sUydj625h86tuCqa9T+ilOlpiFt7jcT7a02+SP7+QhsdrWfeCjVvn5kBXM7XPduH/2lKKxp/tOHZmzvZQ+9sgscvL8A9ZCXfMKcL70npqf+0kcLMPZ1+Izf+5md4bH6ByzghDZcboeKWBEDYCBTnYDrXQEbXhmDw8gBnkn+O8YsWIYZzS8hPvi4Pt1A8aSxpKHGB77VNsB8iYhG/hAibu3MgrXbrXuoiIXBgU2FNl6AdLh0oOXc52ULi4Gp6rZfOvV1EXA/ulHvx3VbNw1kDINHIDLF1Sx4b6dTz2bByb00PRkirKcs/yqcorovjytTQ+vorIkodZPOcOKt6uoe7Z1Wwb68BbWETxjG2E+6LwEe++blxVztIlBhueX8+qzXFsk3Px3xKgcDLwEb73yT7Lh+fpILFr/KfeMcfuZ8l9cWprN1PzWB1x04mnMEBgtgkj3aV+chGLvxxl43MbWbUlCqYTT+FiKq61w7CvMzrneU0C2E59TyphQYZxhkYbERERkdQY8+7hI/0A/f39JJNJEokEiUSC/mQCK5HAshLE+/qwEgn6+vq41j/SUufFY2+og6s87r9+4HBWkJrv1xC/42EqZw+GwP11/OjnXRR/eylFqfn+Hhklse1rWFGfTdV3FuHJALAIPrWCtX0BHq70n/8vnvoUO5t/kwe2PjXyfdiH+6gr7Gd5H/aP/f+GiIjIWdCb+alieCie56D9hVpawoOru1cWUTy9k6aNzXR+hBVoSS9WVzPrX+zANX/eB2E98kYtdbvtFBXnK6xfSJJH2P27jfyxW+0wIiJy4VBLTMoYuG5eytKpTQQPxcBpBxwU3VtN9ksthI+BS8nughQ7FGPaLdUsmXPifjZxIofHU/yVagov1z+hdDDpqs9z02XnIYSPGYdDX7IkIiKjTGkjpUxcc8s45Ts+DSe+BWWjVZCkgH12Kac+gyaekvJRqkZGYk69HPfUv36ciIjIhUgtMSIiIiIiaUyBXUREREQkjSmwi4iIiIikMQV2EREREZE0psAuIiIiIpLGFNhFRERERNKYAruIiIiISBpTYB9mvJk52iWIyAUkmUwyZsyY0S5DREQuYgrsw4zNyCByuGe0yxCRC0Q0eowJE/Q1xiIicv4osA9jvySLWCzG4SMK7SJyZslkkqNHo4TfiTB58sTRLkdERC5ixmgXkG7s9kvo74f3jvXy3rFe3o8dH+2SRD7VsrLGszfUMdplnGbMmDFMmGByxRWXMT5z3GiXIyIiFzEF9hFkZ19CdvYlo12GiIiIiIhaYkRERERE0pkCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksYU2EVERERE0pgCu4iIiIhIGlNgFxERERFJYwrsIiIiIiJpTIFdRERERCSNKbCLiIiIiKQxBXYRERERkTSmwC4iIiIiksb+P++HVVRtwz/DAAAAAElFTkSuQmCC)

The LaunchDarkly event import REST API accepts a `User-Agent` header, which helps identify the source of traffic and debug issues. One of the components to construct this header is the `Version`, which can be any format.

### Version

This configuration section allows you to define the version to be used. For example, you could use the `Container Version` as the version to use. If this is not provided, the Tag will use the string `"1"` instead.

As an example using the default version, the User-Agent header could be like:

```text
'User-Agent: MetricImport-Snowplow-int/1'
```

## Logs Settings

![](/assets/images/09-logs-settings-9d0b55515c253bf74027121c7524c6a0.png)

Through the Logs Settings you can control the logging behavior of the LaunchDarkly Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the **default** option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the LaunchDarkly GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "LaunchDarkly Metric Events", // the name of the tag
    "Type": "Message",   // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",    // the "trace-id" header if exists
    "EventName": "xxx"   // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | "Message"                                                      |
| Request     | "RequestMethod", "RequestUrl", "RequestHeaders", "RequestBody" |
| Response    | "ResponseStatusCode", "ResponseHeaders", "ResponseBody"        |

---

# Snowplow Client for GTM Server Side
> Receive Snowplow events in GTM Server Side containers with the Snowplow Client, which populates common event data and rich Snowplow properties for tags.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/

To receive events in your GTM SS container, the Snowplow Client must be installed. This works for both events direct from the tracker, or enriched events from the pipeline.

The Snowplow Client populates the common event data so many GTM SS tags will just work, however it also populates a set of additional properties to ensure the rich Snowplow event data is available to Tags which wish to take advantage of this, such as the Snowplow Authored Tags.

## Template Installation

There are two methods to install the Snowplow Client.

### Tag Manager Gallery

**Coming Soon.** The Gallery for Clients has not yet been made public.

### Manual Installation

1. Download [template.tpl](https://raw.githubusercontent.com/snowplow/snowplow-gtm-server-side-client/main/template.tpl) - Ctrl+S (Win) or Cmd+S (Mac) to save the file, or right click the link on this page and select "Save Link As..."
2. Create a new Client in the Templates section of a Google Tag Manager Server container
3. Click the More Actions menu, in the top right hand corner, and select Import
4. Import `template.tpl` downloaded in Step 1
5. Click Save

![Installing Snowplow Client](/assets/images/manualclientinstall-93c4a4f45c4decdd25c20faf1d519013.gif)

## Snowplow Client Setup

With the template installed, you can now add the Snowplow Client to your GTM SS Container.

1. From the Clients tab, select "New", then select the Snowplow Client as your Client Configuration
2. Click Save

![Adding Snowplow Client to GTM SS](/assets/images/clientsetup-a7275d01ad3c7e318d302a322af2c3f5.gif)

## Testing

You can test your Snowplow Client setup by using GTM SS [preview mode](https://developers.google.com/tag-platform/tag-manager/server-side/debug).

Once your debug container is running, obtain your preview header value:

![Opening the “send requests manually” popup](/assets/images/preview-mode-1-965b75ae77d43b5fa941eaed2e378f5e.png)

![The “send requests manually” popup](/assets/images/preview-mode-2-8913f55547ae5b9dbf3debea2c2ac704.png)

Now you can use the cURL command below. Note:

- Replace `{{your-gtm-ss-url}}` with the URL of your GTM SS container.
- Replace `{{your-preview-header}}` with the value obtained above.

```bash
curl --request POST \
  --url https://{{your-gtm-ss-url}}/com.snowplowanalytics.snowplow/enriched \
  --header 'Content-Type: application/json' \
  --header 'x-gtm-server-preview: {{your-preview-header}}' \
  --data '{
  "app_id": "example-website",
  "platform": "web",
  "etl_tstamp": "2021-11-26T00:01:25.292Z",
  "collector_tstamp": "2021-11-20T00:02:05Z",
  "dvce_created_tstamp": "2021-11-20T00:03:57.885Z",
  "event": "unstruct",
  "event_id": "c6ef3124-b53a-4b13-a233-0088f79dcbcb",
  "txn_id": null,
  "name_tracker": "sp1",
  "v_tracker": "js-3.1.6",
  "v_collector": "ssc-2.3.0-stdout$",
  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",
  "user_id": "jon.doe@email.com",
  "user_ipaddress": "92.231.54.234",
  "user_fingerprint": null,
  "domain_userid": "de81d764-990c-4fdc-a37e-adf526909ea6",
  "domain_sessionidx": 3,
  "network_userid": "ecdff4d0-9175-40ac-a8bb-325c49733607",
  "geo_country": "US",
  "geo_region": "CA",
  "geo_city": "San Francisco",
  "geo_zipcode": "94109",
  "geo_latitude": 37.443604,
  "geo_longitude": -122.4124,
  "geo_location": "37.443604,-122.4124",
  "geo_region_name": "San Francisco",
  "ip_isp": "AT&T",
  "ip_organization": "AT&T",
  "ip_domain": "att.com",
  "ip_netspeed": "Cable/DSL",
  "page_url": "https://snowplowanalytics.com/use-cases/",
  "page_title": "Snowplow Analytics",
  "page_referrer": null,
  "page_urlscheme": "https",
  "page_urlhost": "snowplowanalytics.com",
  "page_urlport": 443,
  "page_urlpath": "/use-cases/",
  "page_urlquery": "",
  "page_urlfragment": "",
  "refr_urlscheme": null,
  "refr_urlhost": null,
  "refr_urlport": null,
  "refr_urlpath": null,
  "refr_urlquery": null,
  "refr_urlfragment": null,
  "refr_medium": null,
  "refr_source": null,
  "refr_term": null,
  "mkt_medium": null,
  "mkt_source": null,
  "mkt_term": null,
  "mkt_content": null,
  "mkt_campaign": null,
  "contexts_org_w3_performance_timing_1": [
    {
      "navigationStart": 1415358089861,
      "unloadEventStart": 1415358090270,
      "unloadEventEnd": 1415358090287,
      "redirectStart": 0,
      "redirectEnd": 0,
      "fetchStart": 1415358089870,
      "domainLookupStart": 1415358090102,
      "domainLookupEnd": 1415358090102,
      "connectStart": 1415358090103,
      "connectEnd": 1415358090183,
      "requestStart": 1415358090183,
      "responseStart": 1415358090265,
      "responseEnd": 1415358090265,
      "domLoading": 1415358090270,
      "domInteractive": 1415358090886,
      "domContentLoadedEventStart": 1415358090968,
      "domContentLoadedEventEnd": 1415358091309,
      "domComplete": 0,
      "loadEventStart": 0,
      "loadEventEnd": 0
    }
  ],
  "se_category": null,
  "se_action": null,
  "se_label": null,
  "se_property": null,
  "se_value": null,
  "unstruct_event_com_snowplowanalytics_snowplow_link_click_1": {
    "targetUrl": "http://www.example.com",
    "elementClasses": [
      "foreground"
    ],
    "elementId": "exampleLink"
  },
  "tr_orderid": null,
  "tr_affiliation": null,
  "tr_total": null,
  "tr_tax": null,
  "tr_shipping": null,
  "tr_city": null,
  "tr_state": null,
  "tr_country": null,
  "ti_orderid": null,
  "ti_sku": null,
  "ti_name": null,
  "ti_category": null,
  "ti_price": null,
  "ti_quantity": null,
  "pp_xoffset_min": null,
  "pp_xoffset_max": null,
  "pp_yoffset_min": null,
  "pp_yoffset_max": null,
  "useragent": null,
  "br_name": null,
  "br_family": null,
  "br_version": null,
  "br_type": null,
  "br_renderengine": null,
  "br_lang": null,
  "br_features_pdf": true,
  "br_features_flash": false,
  "br_features_java": null,
  "br_features_director": null,
  "br_features_quicktime": null,
  "br_features_realplayer": null,
  "br_features_windowsmedia": null,
  "br_features_gears": null,
  "br_features_silverlight": null,
  "br_cookies": null,
  "br_colordepth": null,
  "br_viewwidth": null,
  "br_viewheight": null,
  "os_name": null,
  "os_family": null,
  "os_manufacturer": null,
  "os_timezone": null,
  "dvce_type": null,
  "dvce_ismobile": null,
  "dvce_screenwidth": null,
  "dvce_screenheight": null,
  "doc_charset": null,
  "doc_width": null,
  "doc_height": null,
  "tr_currency": null,
  "tr_total_base": null,
  "tr_tax_base": null,
  "tr_shipping_base": null,
  "ti_currency": null,
  "ti_price_base": null,
  "base_currency": null,
  "geo_timezone": null,
  "mkt_clickid": null,
  "mkt_network": null,
  "etl_tags": null,
  "dvce_sent_tstamp": null,
  "refr_domain_userid": null,
  "refr_dvce_tstamp": null,
  "contexts_com_snowplowanalytics_snowplow_ua_parser_context_1": [
    {
      "useragentFamily": "IE",
      "useragentMajor": "7",
      "useragentMinor": "0",
      "useragentPatch": null,
      "useragentVersion": "IE 7.0",
      "osFamily": "Windows XP",
      "osMajor": null,
      "osMinor": null,
      "osPatch": null,
      "osPatchMinor": null,
      "osVersion": "Windows XP",
      "deviceFamily": "Other"
    }
  ],
  "domain_sessionid": "2b15e5c8-d3b1-11e4-b9d6-1681e6b88ec1",
  "derived_tstamp": "2021-11-20T00:03:57.886Z",
  "event_vendor": "com.snowplowanalytics.snowplow",
  "event_name": "link_click",
  "event_format": "jsonschema",
  "event_version": "1-0-0",
  "event_fingerprint": "e3dbfa9cca0412c3d4052863cefb547f",
  "true_tstamp": "2021-11-20T00:03:57.886Z"
}'
```

---

# Configure Snowplow Client for GTM Server Side
> Configure IP forwarding, sp.js hosting, custom paths, common event mapping, and entity merging for the Snowplow Client in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-client-for-gtm-ss/snowplow-client-configuration/

> **Tip:** The [GTM Common Event](https://developers.google.com/tag-platform/tag-manager/server-side/common-event-data) has a `user_data` property. To populate this, you can attach a context Entity to your events of this schema: `iglu:com.google.tag-manager.server-side/user_data/jsonschema/1-0-0`, which can be found on [Iglu Central](https://github.com/snowplow/iglu-central/blob/853357452300b172ebc113d1d75d1997f595142a/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0).

## Forward User IP address

As the container sits between the website user and the Snowplow collector (or other downstream destinations), the users IP will be unknown to the destination. By enabling this option, the users IP address will be included in the events sent to Tags.

By disabling this, you are able to use GTM SS as a proxy which can string user IP addresses from requests. Many tags also offer this functionality at the tag level.

## Populate GAv4 Client Properties

Enabled by default, this option will populate additional properties which the GAv4 requires, that is useful if you want to forward your Snowplow events to the GAv4 Tag.

## sp.js settings

This setting allows for your GTM SS Container to serve your `sp.js` JavaScript Tracker file. This allows you to have first party hosting of your tracker without the need to set up separate hosting or use a third party CDN.

It is recommended to rename `sp.js` if enabling this setting, as many adblockers will block requests to files named `sp.js`. A random string is the best option here.

![sp.js settings](/assets/images/spjssettings-905ab9716df666e83d799fa929e3bc40.png)

You can request _any_ version of the Snowplow JavaScript Tracker with this setting enabled. e.g. `https://{{gtm-ss-url}}/3.1.6/776b5b25.js` will load v3.1.6, or `https://{{gtm-ss-url}}/2.18.2/776b5b25.js` will load v2.18.2.

## Additional Options

### Custom POST Path

As many ad blockers will block the default `/com.snowplowanalytics.snowplow/tp2` POST path, it is recommended to change this and then update your trackers initialization to use this custom POST path.

### Claim GET Requests

The default Snowplow path for GET requests is `/i`, as this is so short there is a chance it could conflict with other Clients within your GTM SS Container. If you'd only like your Snowplow Client to listen for POST requests, you can disable this GET endpoint with this setting.

### Include Original `tp2` Event

If using this Client to receive Snowplow Tracker events and then forward to a Snowplow Collector with the Snowplow Tag, you should leave this option enabled as it will allow the Snowplow Tag to forward the original tracker event with no extra processing.

### Include Original Self Describing Event

By default, the self describing event will be "shredded" into a key using the schema name as the key, this is a "lossy" transformation, as the Minor and Patch parts of the jsonschema version will be dropped. This flag populates the original, lossless, Self Describing Event as `x-sp-self_describing_event`.

Let's assume we have a self-describing event following the schema `iglu:com.acme/foobar/jsonschema/1-0-0`. By default, the option to _Include Original Self Describing Event_ is disabled. So the Snowplow client by default will include it in the common event as:

```json
"x-sp-self_describing_event_com_acme_foobar_1": {
    "foo": "bar"
}
```

In case the option to _Include Original Self Describing Event_ is enabled, then the Snowplow client, if it finds the original event (see note below), will also include it in the common event, resulting in:

```json
"x-sp-self_describing_event_com_acme_foobar_1": {
    "foo": "bar"
},
"x-sp-self_describing_event": {
    "schema": "iglu:com.acme/foobar/jsonschema/1-0-0",
    "data": {
        "foo": "bar"
    }
}
```

> **Note:** This option only makes sense when using GTM in a [**Server Side Tag Manager (pre-pipeline)**](/docs/destinations/forwarding-events/google-tag-manager-server-side/#configuration-options) architecture, because it only makes a difference when the input is a _raw_ Snowplow event.
>
> In a [**Destinations Hub (post-pipeline)**](/docs/destinations/forwarding-events/google-tag-manager-server-side/#configuration-options) architecture, this option **does not apply**. Effectively, it’s always disabled, regardless of the setting. In the example above, this would mean that the data will contain `x-sp-self_describing_event_com_acme_foobar_1`, but not`x-sp-self_describing_event`.

### Include Original Contexts Array

By default, the contexts will be "shredded" into separate keys using the context name as the key, this is a "lossy" transformation, as the Minor and Patch parts of the jsonschema version will be dropped. If you would like to keep the original "lossless" contexts array (as `x-sp-contexts`), enable this option.

## Advanced common event options

![advanced common event options](/assets/images/advanced_common_options-87a6c18646fd2ad02e1d8ca7d6e8a49d.png)

### `client_id`

#### Use default settings for client\_id mapping in common event

By default the Snowplow Client sets the `client_id` as follows: If the Snowplow event has the `client_session` context entity attached, its `userId` property is used. Else the `domain_userid` atomic property is used. Disabling this option reveals the following table that allows you to override the default behavior.

#### Specify client\_id

You can use this table to specify the rules to set the `client_id` of the common event. For consistency downstream it is suggested to specify properties that apply to all Snowplow events (atomic or through global context entities). The columns of this table are:

- **Priority**: Using this column you can set the priority (higher values mean higher priority) with which the Client will look into the Snowplow event to locate the value to set the `client_id`.
- **Property name or path**: This column refers to the common event, so you can define alternative Snowplow properties using the `x-sp-` prefix before the enriched property name or nested path (using dot notation). Example values: `x-sp-network_userid` or `x-sp-contexts_com_acme_user_1.0.anonymous_identifier`.

### `user_id`

#### Use default settings for user\_id mapping in common event

By default the Snowplow Client sets the `user_id` from the corresponding `user_id` property of the Snowplow event. Disabling this option reveals the following table that allows you to override the default behavior.

#### Specify user\_id

You can use this table to specify the rules to set the `user_id` of the common event, which will override the default Snowplow Client behavior. For consistency downstream it is suggested to specify properties that apply to all Snowplow events (atomic or through global context entities). The columns of this table are:

- **Priority**: Using this column you can set the priority (higher values mean higher priority) with which the Client will look into the Snowplow event to locate the value to set the `user_id`.
- **Property name or path**: This column refers to the common event, so you can define alternative Snowplow properties using the `x-sp-` prefix before the enriched property name or nested path (using dot notation). For example: `x-sp-contexts_com_acme_user_entity_1.0.email`.

### Snowplow Entities Mapping

> **Note:** This is an advanced feature and we recommend that you first explore the configuration options of your Tag(s) in case they are enough to cover your use case.

#### Merge selected Snowplow entities

Enable this option to allow merging of Snowplow context data to the Common Event. Checking this box reveals the following table:

![](/assets/images/context_merging_overview-583c306bceab0a6601d86760ed9ebb73.png)

#### Context entities to merge

Using this table you can specify the rules to merge Snowplow context entity data to the Common Event. The columns of this table are:

![](/assets/images/context_merging_new_row-95c55b188e68a938e0f393b772b1c3ab.png)

1. **Schema**: (Required) The schema of the context entity to merge.

> **Info:** The **Schema** can be specified in 3 ways:
>
> - Iglu URI (e.g. `iglu:com.acme/test/jsonschema/1-0-0` )
> - Enriched name (e.g. `contexts_com_acme_test_1`)
> - Common Event name (e.g. `x-sp-contexts_com_acme_test_1`)

2. **Apply to all versions**: (False/True) Whether the rule applies to all versions of the context entity schema. Default is False.

> **Info:** When you set **Apply to all versions** to `True`, it will apply the same rule to all versions of the schema, independently of the one mentioned in the **Schema** field.

3. **Prefix**: (Optional) Specify a prefix to use for property names when merging.

4. **Merge to**: (Event Properties/Custom) Specify where to merge the context entity's properties. Default is Event Properties, i.e. to the root level of the common event.

5. **Custom path**: (Optional) The path for custom merging.

> **Info:** The **Custom path** option applies only if the **Merge to** column is set to `Custom`, else the row is considered invalid.

6. **Keep original mapping**: (True/False) Whether to keep the original context mapping. Default is True.

7. **Custom transformation**: (Optional) Specify a Variable returning **a function** that represents a custom transformation of the context data to the desired object before merging. The default behaviour is the following: It first checks whether the context array contains a single entity object and if so, it merges this single object. In case the context specified contains multiples of entity objects, the rule is not applied (in those cases you will need to provide a custom transformation function through a Variable).

> **Info:** The function signature must be: `(contextArray, event) => Object`
>
> The Client guarantees that it will call this function providing as arguments the original context (Array) specified and the Common Event (Object) it has constructed. The event argument is provided in order to enable merging logic to optionally be based on event properties. Please note that it is not possible to modify the event inside the function you will define. The Client expects the function to return an Object, otherwise the value is ignored.

#### Example: Custom transformation function for context mapping

You can define and return the function in a **Variable Template**, which you can then use to create a **Variable** to reference.

The following is an example code of such a **Variable Template**:

```javascript
// Variable template code
function selectFirst(contextArray, event) {
  // the function must return an object
  return contextArray[0];
}

// The Variable must return the function
return selectFirst;
```

### Snowplow Self-describing Event Mapping

> **Note:** This is an advanced feature and we recommend that you first explore the configuration options of your Tag(s) in case they are enough to cover your use case.

#### Merge selected Snowplow self-describing event data

Enable this option to allow merging of Snowplow self-describing event data to the Common Event. Checking this box reveals the following table:

![](/assets/images/selfdesc_merging_overview-b2be0321a3de3b7d0bdf95169cf9b5b7.png)

#### Self-describing events to merge

Using this table you can specify the rules to merge Snowplow self-describing event data to the Common Event. The columns of this table are:

![](/assets/images/selfdesc_merging_new_row-f75e06e030456a7f31905c40e6b6de93.png)

**Schema**: (Required) The schema of the self-describing event.

> **Info:** The **Schema** can be specified in 3 ways:
>
> - Iglu URI (e.g. `iglu:com.acme/myevent/jsonschema/1-0-0` )
> - Enriched name (e.g. `contexts_com_acme_myevent_1`)
> - Common Event name (e.g. `x-sp-contexts_com_acme_myevent_1`)

**Apply to all versions**: (False/True) Whether the rule applies to all versions of the self-describing event schema. Default is False.

> **Info:** When you set **Apply to all versions** to `True`, it will apply the same rule to all versions of the schema, independently of the one mentioned in the **Schema** field.

**Prefix**: (Optional) Specify a prefix to use for property names when merging.

**Merge to**: (Event Properties/Custom) Specify where to merge the self-describing event’s properties. Default is Event Properties, i.e. to the root level of the common event.

**Custom path**: (Optional) The path for custom merging.

> **Info:** The **Custom path** option applies only if the **Merge to** column is set to `Custom`, else the row is considered invalid.

**Keep original mapping**: (True/False) Whether to keep the original context mapping. Default is True.

**Custom transformation**: (Optional) Specify a Variable returning **a function** that represents a custom transformation of the self-describing data to the desired object before merging. The default behaviour is to merge the self-describing data object as is.

> **Info:** The function signature must be: `(selfDescObject, event) => Object`
>
> The Client guarantees that it will call this function providing as arguments the original self-describing data (Object) specified and the Common Event (Object) it has constructed. The event argument is provided in order to enable merging logic to optionally be based on other event properties. Please note that it is not possible to modify the event inside the function you will define. The Client expects the function to return an Object, otherwise the value is ignored.

#### Example: Custom transformation function for self-describing event mapping

You can define and return the function in a **Variable Template**, which you can then use to create a **Variable** to reference.

The following is an example code of such a **Variable Template**:

```javascript
// Variable template code
const Object = require('Object');

function addKeySuffix(selfDescObject, event) {
  // the function must return an object
  return Object.keys(selfDescObject).reduce((acc, curr) => {
    acc[curr.concat('_suffix')] = selfDescObject[curr];
    return acc;
  }, {});
}

// The Variable must return the function
return addKeySuffix;
```

---

# Snowplow Tag for GTM Server Side
> Forward events to Snowplow Collector from GTM Server Side using the Snowplow Tag, supporting events from Snowplow JavaScript Tracker or other GTM SS clients like GA4.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/

The [Snowplow Tag for GTM SS](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-server-side-tag) is most useful if using GTM SS as a Server Side Tag Manager for Snowplow JavaScript Tracker events, as you will want to ensure you forward these events to your Snowplow Collector. The Snowplow Tag will automatically forward any events the Snowplow Client receives once it has been configured with your Collector URL.

The Snowplow Tag can also construct Snowplow events from other GTM SS Clients such as GAv4.

## Template Installation

> **Note:** The server Docker image must be 2.0.0 or later.

### Tag Manager Gallery

1. From the Templates tab in GTM SS, click "Search Gallery" in the Tag Templates section
2. Search for "Snowplow" and select the official "By Snowplow" tag
3. Click Add to Workspace
4. Accept the permissions dialog by clicking "Add"

## Snowplow Tag Setup

With the template installed, you can now add the Snowplow Tag to your GTM SS Container.

1. From the Tag tab, select "New", then select the Snowplow Tag as your Tag Configuration
2. Select your desired Trigger - If using the Snowplow JavaScript Tracker and Snowplow Client, you want to select "All Events"
3. Enter your Snowplow Collector Endpoint, and confirm the Cookie Name matches that of your Collector
4. Click Save

![](/assets/images/tagsetup-3a2fbd7ede526d18786087b5b2b1dfbb.gif)

---

# Event settings for Snowplow Tag in GTM SS
> Configure structured events, self-describing events, and context entities for the Snowplow Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/advanced-event-settings/

This page describes the event settings available in the Snowplow Tag for GTM Server Side.

## Structured events

This section allows you to specify which incoming events will be tracked as custom Snowplow events. It only targets events that follow the event/category/label/value Universal Analytics model.

### Send selected events as Snowplow Structured Events

Enabling this check-box reveals a multi-line text field to allow you to set custom Structured Events.

### Event name(s) selected

Add the event names (in separate lines) to be tracked as custom structured Snowplow events.

## Self-describing events

This section allows you to define [Snowplow self-describing events](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-event-self-describing).

![self describing events setup](/assets/images/self_describing_setup-5b8b1628a6630ae2a9c8d69ea3959bb0.png)

### Define events to be sent as Snowplow Self-Describing Events

Enable this to allow custom self-describing event definitions.

### Event Name to Schema

A table of the events (event names and corresponding schemas) to be tracked as custom self-describing Snowplow events.

Add events which you would like to capture and convert into Self Describing Events. The `Event Name` should equal the Clients `event_name` property, if this is found when the Snowplow Tag fires, this tag will create a Self Describing Event using the specified schema.

### Event Definitions

A table of definitions for self-describing data properties. Each row maps a single data property of a custom self-describing Snowplow event to its value.

For each Event, you can also read properties off the client event object and add them as properties to the Self Describing Event.

## Context entities

### Apply context entities settings also to Snowplow events

Enable this tick box to also apply the Context entities settings to raw Snowplow events. This may be helpful in use cases where you want to modify the entities already attached to Snowplow events **before** they are relayed to your Snowplow collector.

### Context entities settings

This section allows you to define the context entities to be attached to your events using the following subsections.

You can define both custom and global context entities: Custom Entities are attached where the event\_name matches the incoming event, whereas Global Entities will be applied to all events.

Both subsections offer two ways to set your context entities:

1. Using the default tables in the UI:

![context entities settings](/assets/images/context_entities_settings-30d068889d497c4542ba61cf9f44e4aa.png)

2. Through GTM Server-side variables (best suited for advanced use-cases):

![context entities settings through variable](/assets/images/context_entities_settings_with_var-32d664fa3c5b3d2c3172337cd0b69b6f.png)

#### Custom context entities

> **Tip:** Custom context entities can be used to augment any standard Snowplow event, including self describing events, with additional data. The entities set in this subsection are attached _where the event name matches the incoming event_.

##### Use variables to define custom context entities

Enabling this setting allows you to set custom context entities using variables that return the entities array to attach to the event.

##### Define custom context entities

This is the way to define custom context entities through table UI. The columns of this table are:

- **Event Name**: The event that this entity will be attached to.

- **Entity Schema**: The schema of the entity.

- **Entity Property Name**: The key path to the entity's data.

- **Type**: The type of the value. Available options are:

  - **Default**: This option means that the value remains to its original type.
  - **String**: This option means that the value will be interpreted as a string.
  - **Boolean**: This option means that the value will be interpreted as a boolean.
  - **Number**: This option means that the value will be interpreted as a number.

- **Reference**: What the value references. Available options are:

  - **Client Event Property** (default): This means that what is set in the **Value** column corresponds to a property path in the client event.
  - **Constant or Variable**: This means that what is set in the **Value** column is the actual value to be used.

- **Value**: The value to set the property name to.

##### Set custom context entities through variables

This table is revealed if **Use variables to define custom context entities** option is enabled. It provides an alternative way to set custom context entities for an even through a GTM Server-side variable. The variable specified must return an array of context entities.

The columns of this table are:

- **Event Name**: The event that this entity will be attached to.
- **Custom Entities Array**: The variable containing the array of entities to be attached to the event.

#### Global context entities

> **Tip:** Global context entities are custom entities that apply globally. This lets you define your own context entities once and have them sent _with **all** events_.

##### Use a variable to define global context entities

Enabling this setting allows you to set global context entities using a variable that returns the entities array to attach to the event.

##### Define global context entities

This is the way to define global context entities through a table UI. The columns of this table are:

- **Entity Schema**: The schema of the entity.

- **Entity Property Name**: The key path to the entity's data.

- **Type**: The type of the value. Available options are:

  - **Default**: This option means that the value remains to its original type.
  - **String**: This option means that the value will be interpreted as a string.
  - **Boolean**: This option means that the value will be interpreted as a boolean.
  - **Number**: This option means that the value will be interpreted as a number.

- **Reference**: What the value references. Available options are:

  - **Client Event Property** (default): This means that what is set in the **Value** column corresponds to a property path in the client event.
  - **Constant or Variable**: This means that what is set in the **Value** column is the actual value to be used.

- **Value**: The value to set the property name to.

##### Set global context entities through variable

This text box is revealed if **Use a variable to define global context entities** option is enabled.

### Examples

In the following screenshots you can find examples of context entities settings using:

#### 1. The default table UI

_**Scenario**_: On a tutorial platform, we use GTM to send a `tutorial_begin` event type to our GTM Server-side container with parameters:

```javascript
dataLayer.push({
  'event': 'tutorial_begin',
  'tutorial.id': 'math101',
  'tutorial.category_name': 'mathematics',
  'user_data.email_address': 'foo@bar.baz',
});
```

We are using the Snowplow GTM SS Tag to forward it to our Snowplow pipeline, and we want to attach context entities. For our example, let's say:

- our `tutorial` Snowplow entity corresponds to this schema

  ```json
  {
      "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
      "self": {
          "vendor": "com.acme",
          "name": "tutorial",
          "version": "1-0-0",
          "format": "jsonschema"
      },
      "description": "Tutorial information",
      "type": "object",
      "properties": {
          "id": {
              "description": "The ID of the tutorial",
              "type": "string",
              "maxLength": 4096
          },
          "category": {
              "description": "The category of the tutorial",
              "type": ["string", "null"],
              "maxLength": 4096
          }
      },
      "required": ["id"]
      "additionalProperties": false
  }
  ```

- we also want user information, so we will use the `user_data` context entity ([available in Iglu Central](https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.tag-manager.server-side/user_data/jsonschema/1-0-0)) attached to all events as well.

Then we can configure the Snowplow GTM SS Tag's Context entities settings as:

![context entities settings example A](/assets/images/context_entities_settings_example-b75aaf07d445a7e76d034b91aac9270c.png)

Here we have configured the Context entities settings in order to:

1. Attach the `tutorial` entity only to `tutorial_begin` event (using the **Custom context entities** section). Consequently we specify how to derive the data of the entity:

- We set the `id` to the value found in the `tutorial.id` key path of the client event
- We set the `category` property to the value found in the `tutorial.category_name` key path of the client event

2. Attach the `user_data` entity to all events (using the **Global context entities** section). In the row, we specify how to derive the `user_data` entity's data:

- We set the `email_address` to the value found in the `user_data.email_address` key path of the client event

> **Note:** As you can see in the images above, the schema for an entity can be written in 2 equally correct ways:
>
> 1. With the `iglu:` prefix, e.g. `iglu:com.acme/product/jsonschema/1-0-0`
> 2. Without the `iglu:` prefix, e.g. `com.acme/product/jsonschema/1-0-0`

#### 2. GTM SS Variables

As mentioned, it is also possible to reference GTM Server-side Variables in order to set both the Custom and the Global context entities. As a simple example:

![context entities settings example B](/assets/images/context_entities_settings_example_var-a10e8e255f4ca4506f15accc03749520.png)

Here:

1. We specify the Variable `product_entities` as the context entities to attach only to `purchase` events
2. We specify the Valiable `global_entities` as the context entities to attach to all events

The Variables referenced must return an array of context entities. For example, the return value of such a Variable should look like:

```javascript
[
  {
    schema: "iglu:com.example_company/page/jsonschema/1-2-1",
    data: {
      pageType: 'test',
      lastUpdated: '2022-11-18T17:59:00',
    }
  },
  {
    schema: "iglu:com.example_company/user/jsonschema/2-0-0",
    data: {
      userType: 'tester',
    }
  }
]
```

## Additional event settings

### Base64 encoding

Whether to encode the custom self-describing event data in base64.

### Platform identifier

When a platform is not specified on the event, this value will be used.

### App ID

This text box allows you to specify the `app_id` for the event.

> **Note:** In case you are specifying the `app_id` through a GTM-SS Variable, please ensure that its return value is a string, otherwise it will be ignored.

---

# Configure Snowplow Tag for GTM Server Side
> Configure collector URL, cookie settings, advanced event settings, and logging for the Snowplow Tag in GTM Server Side.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/

## Collector URL (Required)

Set this to the URL of your Snowplow Collector you wish to send events to.

## Cookies Settings

If you have configured your Snowplow collector to have different cookie details, you should ensure they match here.

### Name (Required)

This value must match the value of your Collector cookie name, this will allow the Snowplow Tag to find and return your Collector Cookie back to your users browser.

### Override Cookie Properties

If you'd like to overwrite your collectors cookie settings you can do that here.

#### Domain Override

To return a cookie with a different domain value than your collector, you can override it to another string here. "auto" will ensure the value is unchanged.

#### Path Override

This will override the path value of the cookie.

#### SameSite Override

This will override the SameSite flag on the cookie.

#### Expiration Override in Seconds

Allows the expiration time of the cookie to be altered. This value is in seconds, and defaults to 63072000 (2 years).

#### HttpOnly

Overwrites the HttpOnly flag on the cookie. Will be `true` if enabled (default), or `false` if disabled.

#### Secure

Overwrites the Secure flag on the cookie. Will be `true` if enabled (default), or `false` if disabled. Setting this to `false` and `SameSite` to `None` will prevent browsers from being able to store the cookie.

### Cookie Headers

![cookie headers](/assets/images/cookie_headers-acafd18e7933e0e939c4ff547256a52a.png)

#### Forward Cookies in the request headers

This option allows you to forward selected cookies in the request headers and it is disabled by default. Enabling it reveals the "Cookies to forward" table.

#### Cookies to forward

Using this table allows you to select the cookies to forward. Its columns are:

- **Cookie Name**: The name of the cookie to forward
- **Decode**: Whether to want to decode the cookie value(s) before forwarding (defaults to `No`)

## Advanced Event Settings

![advanced event settings overview](/assets/images/advanced_event_settings_overview-6567e2661a9a168f1271f37ac28335ae.png)

This section allows you to:

- Specify events to be tracked as Structured events
- Configure Self-describing event definitions
- Specify Context entities to be attached to events
- Customize additional event settings

You can find out more about all the available configuration options in the [Advanced Event Settings page](/docs/destinations/forwarding-events/google-tag-manager-server-side/snowplow-tag-for-gtm-ss/snowplow-tag-configuration/advanced-event-settings/).

## Logs Settings

Through the Logs Settings you can control the logging behavior of the Snowplow Tag. The available options are:

- `Do not log`: This option allows you to completely disable logging. No logs will be generated by the Tag.
- `Log to console during debug and preview`: This option enables logging only in debug and preview containers. This is the default option.
- `Always`: This option enables logging regardless of container mode.

> **Note:** Please take into consideration that the logs generated may contain event data.

The logs generated by the Snowplow GTM SS Tag are standardized JSON strings. The standard log properties are:

```json
{
    "Name": "Snowplow", // the name of the tag
    "Type": "Message",  // the type of log (one of "Message", "Request", "Response")
    "TraceId": "xxx",   // the "trace-id" header if exists
    "EventName": "xxx"  // the name of the event the tag fired at
}
```

Depending on the type of log, additional properties are logged:

| Type of log | Additional information                                         |
| ----------- | -------------------------------------------------------------- |
| Message     | "Message"                                                      |
| Request     | "RequestMethod", "RequestUrl", "RequestHeaders", "RequestBody" |
| Response    | "ResponseStatusCode", "ResponseHeaders", "ResponseBody"        |

---

# Debug and test GTM Server Side tags
> Test and debug Google Tag Manager Server Side tag configurations using Preview Mode before deploying to production environments.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/google-tag-manager-server-side/testing/

If you are working on some changes to the configuration of your Google Tag Manager tags and would like to test them before applying them in production, you can use GTM’s [Preview Mode](https://developers.google.com/tag-platform/tag-manager/server-side/debug) feature. It shows information about the events it receives, which tags get triggered, etc.

You can direct some (or all) of your Snowplow events to the Preview Mode, instead of the production tags.

> **Note:** To follow the steps below, you will need to be running [Snowbridge](/docs/api-reference/snowbridge/) 2.3+. You will also need to have the [`spGtmssPreview` transformation](/docs/api-reference/snowbridge/configuration/transformations/builtin/spGtmssPreview/) activated (this is the default for Snowplow customers using Snowbridge with GTM Server Side).

## Copy the preview header

Once you enter Preview Mode in Google Tag Manager, click on the three dots in the top right corner of the screen and click “Send requests manually”.

You will see a popup with a preview header, for example (not a real value):

> **Note:**```text
> sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo=
> ```

Copy the header value (in the example above, `sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo=`).

## Add the preview header to your events

You can add the header value to all or some of your events as an [entity](/docs/fundamentals/entities/).

For example, if you are using the [JavaScript tracker](/docs/sources/web-trackers/):

```javascript
snowplow('trackPageView', {
  context: [{
    schema: 'iglu:com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0',
    data: {
      // paste your header value here
      'x-gtm-server-preview': 'sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo='
    }
  }]
});
```

You can also add it as a [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) for all events:

```javascript
const gtmPreviewContext = {
  schema: 'iglu:com.google.tag-manager.server-side/preview_mode/jsonschema/1-0-0',
  data: {
    // paste your header value here
    'x-gtm-server-preview': 'sTjhMcUdkNldaM2RsOThwWTRvNzE3VkZtb1BwK0E9PQo='
  }
};
snowplow('addGlobalContexts', [gtmPreviewContext]);
```

For trackers other than the JavaScript tracker, the approach is the same — you need to add the preview header as an entity.

> **Tip:** Note that the header value can change over time as you make changes to your Google Tag Manager setup.

---

# Real-time event forwarding to third-party platforms
> Send Snowplow events to third-party platforms in real-time using Snowplow's managed event forwarding solution with built-in filtering, field mapping, and JavaScript transformations.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/

Event forwarders let you filter, transform, and send Snowplow events to third-party platforms in real-time. They're deployed as fully managed apps that sit alongside warehouse and lake loaders in your Snowplow cloud account. You can configure forwarders through Snowplow Console.

![Event forwarding architecture showing data flow from Snowplow pipeline through forwarders to destination APIs](/assets/images/event-forwarding-diagram-2e83ef2253fd7137b01a0f5a81e2aab4.svg)

Event forwarding uses [Snowbridge](/docs/api-reference/snowbridge/) under the hood, deployed within your existing Snowplow cloud account, to transform and deliver events reliably. For detailed setup guides and field mappings, check out the list of [available integrations](/docs/destinations/forwarding-events/integrations/). For complex requirements or unsupported destinations, [advanced alternatives](#alternative-approaches) are also available.

## Use cases

Event forwarding works best for use cases where you need low-latency event delivery and don't require complex aggregations across multiple events. For more complex transformations or batch processing, consider using [reverse ETL](/docs/destinations/reverse-etl/) instead.

Event forwarding is a good fit for use cases such as:

- **Real-time personalization**: send events to marketing automation or customer engagement platforms for immediate campaign triggers
- **Product analytics**: forward user actions to analytics tools for real-time product insights
- **A/B testing**: send experiment events to testing platforms for real-time optimization and analytics
- **Fraud detection**: forward security-relevant events to monitoring systems
- **Customer support**: stream events to support platforms for context-aware assistance

## How it works

Event forwarders are deployed as managed [Snowbridge](/docs/api-reference/snowbridge/) apps that consume events from your enriched event stream in near real-time. It uses a [JavaScript transformation function](/docs/api-reference/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/) generated from your configuration to filter and transform events.

Here's how forwarders process events:

1. **Read events**: reads enriched events from your stream (Kinesis, Pub/Sub, or EventHub) as the Snowplow pipeline produces them
2. **Apply filters**: checks each event against your configured [JavaScript filters](/docs/destinations/forwarding-events/reference/#event-filtering) to decide whether to forward it
3. **Transform data**: transforms matching events using [field mapping expressions](/docs/destinations/forwarding-events/reference/#field-mapping) and custom JavaScript to convert Snowplow event data into your destination's API format
4. **Delivery handling**: sends transformed events to the destination via HTTP API calls. Retries failures depending on the [failure type](/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/#failure-types-and-handling) and [logs non-retryable failures](/docs/destinations/forwarding-events/event-forwarding-monitoring-and-troubleshooting/#what-happens-when-events-fail) to cloud storage

The end-to-end latency from event collection to destination delivery is on the order of seconds. Latency depends on overall pipeline event volume, complexity of transformation logic, and destination rate limits.

## Getting started

To set up a new event forwarder, you must first create a **connection**, which stores the credentials and endpoint details needed to send events to your destination, and then a **event forwarder** configuration, which defines which pipeline to read events from and the transformations to apply to your events. For a step-by-step guide, see [creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

Each destination has its own requirements for API credentials, configuration, and field mappings. See the [available integrations](/docs/destinations/forwarding-events/integrations/) for destination-specific guides.

For detailed information on supported JavaScript expressions, field transformations, and mapping syntax, see the [filter and mapping reference](/docs/destinations/forwarding-events/reference/).

## Alternative approaches

Using event forwarders is the recommended starting point for most real-time delivery use cases. For more complex requirements or unsupported destinations, consider these alternatives:

- **[Snowbridge](/docs/api-reference/snowbridge/)**: flexible event routing with custom transformations and destinations (Kafka, Kinesis, HTTP APIs). Use when you need destinations not yet supported by event forwarders, complex custom transformations, non-HTTP destinations, or advanced batching and retry configurations.
- **[Google Tag Manager Server Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)**: use GTM SS to relay enriched events to destinations using rich libraries of tags. Best if your organization is heavily invested in GTM or if you need destinations not yet supported by event forwarders, but supported by GTM SS, such as Google Analytics.
- **[Custom integrations](/docs/destinations/forwarding-events/custom-integrations/)**: build your own solutions using AWS Lambda, GCP Cloud Functions, or other stream processing systems for fully bespoke requirements.

---

# Forward events to Amplitude
> Send Snowplow events to Amplitude for product analytics and behavioral insights using the HTTP API v2 with support for event tracking and user properties.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/amplitude/

Send Snowplow events to Amplitude to power product and marketing analytics or guide and survey personalization using Amplitude's [HTTP API v2](https://www.docs.developers.amplitude.com/analytics/apis/http-v2-api/).

## Prerequisites

Before setting up the forwarder in Console, you'll need an Amplitude API Key. To find your API key:

1. Log in to your Amplitude workspace
2. Click **Settings** in the top right, then click **Organization Settings**
3. From the sidebar, select **Projects**, then select your Project to view its details
4. Copy the **API Key**

> **Tip:** To avoid introducing bad data in your production Amplitude project, we recommend using a test or development Amplitude project to test your transformations first. Then, create a new Connection in Console with your production API key, and a new forwarder that imports the configuration from your development forwarder.

## Getting started

### Configure the destination

To create the connection and forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Amplitude** for the connection type, enter your API key, and select the **Server Location** where your Amplitude project is hosted.

### Validate the integration

You can confirm events are reaching Amplitude by checking the **Ingestion Debugger** page in your Amplitude account:

1. From the left navigation bar, click **Data**, then select **Sources** from the sidebar. You will see a list of sources.
2. Select the **Ingestion Debugger** tab
3. Filter the graphs to show only events from the **HTTP API** to confirm data is flowing as expected from Snowplow.

## Sending custom properties

You can send custom properties beyond the standard fields defined in the schema reference below. Amplitude supports three types of custom properties:

- **event\_properties**: custom data associated with specific events (e.g., `event_properties.plan_type`, `event_properties.feature_flag`)
- **user\_properties**: custom data tied to user profiles (e.g., `user_properties.subscription_tier`, `user_properties.account_age`)
- **group\_properties**: custom data tied to groups when `event_type` is `$groupidentify` (requires Amplitude Accounts add-on)

For property names containing spaces, use bracket notation (e.g., `event_properties["campaign source"]`).

See Amplitude's [HTTP API v2 documentation](https://amplitude.com/docs/apis/analytics/http-v2#) for details on supported data types, property operations, and object depth limits. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

This section contains information on the fields you can send to Amplitude, including field names, data types, required fields, and default Snowplow mapping expressions.

| Field                                  | Details                                                                                                                                                                                                                                                                                        |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `event_type` _string_                  | _Required._ A unique identifier for your event.Default mapping: `event?.event_name`                                                                                                                                                                                                            |
| `user_id` _unknown_                    | _Optional._ ID for the user. Required if `device_id` isn't provided.Default mapping: `event?.user_id`                                                                                                                                                                                          |
| `device_id` _unknown_                  | _Optional._ A device-specific identifier, such as the Identifier for Vendor on iOS. Required if `user_id` isn't provided.Default mapping: `event?.domain_userid ?? event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.userId`                                               |
| `event_id` _integer_                   | _Optional._ An incrementing counter to distinguish events with the same user\_id and timestamp from each other.                                                                                                                                                                                |
| `session_id` _integer_                 | _Optional._ The start time of the session in milliseconds since epoch (Unix timestamp). Necessary if you want to associate events with a particular system.Default mapping: `spTstampToEpochMillis(event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.firstEventTimestamp)` |
| `insert_id` _string_                   | _Optional._ A unique identifier for the event. Amplitude deduplicates subsequent events sent with the same device\_id and insert\_id within the past 7 days.Default mapping: `event?.event_id`                                                                                                 |
| `time` _integer_                       | _Optional._ The timestamp of the event in milliseconds since epoch. If time isn't sent with the event, then it's set to the request upload time.Default mapping: `spTstampToEpochMillis(event?.derived_tstamp)`                                                                                |
| `event_properties` _object_            | _Optional._ Arbitrary key-value pairs assigned to the event.                                                                                                                                                                                                                                   |
| `user_properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the user.                                                                                                                                                                                                                                    |
| `groups` _object_                      | _Optional._ Arbitrary key-value pairs representing groups of users.                                                                                                                                                                                                                            |
| `group_properties` _object_            | _Optional._ Arbitrary key-value pairs assigned to the groups listed in the `groups`.                                                                                                                                                                                                           |
| `$skip_user_properties_sync` _boolean_ | _Optional._ When true user properties aren't synced. Defaults to false.Default mapping: `false`                                                                                                                                                                                                |
| `app_version` _string_                 | _Optional._ The current version of your application.Default mapping: `event?.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.version`                                                                                                                                                |
| `platform` _string_                    | _Optional._ Platform of the device.Default mapping: `event?.platform`                                                                                                                                                                                                                          |
| `os_name` _string_                     | _Optional._ The name of the mobile operating system or browser that the user is using.Default mapping: `event?.os_name`                                                                                                                                                                        |
| `os_version` _string_                  | _Optional._ The version of the mobile operating system or browser the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemVersion`                                                                                                                  |
| `device_brand` _string_                | _Optional._ The device brand that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceBrand`                                                                                                                                                             |
| `device_manufacturer` _string_         | _Optional._ The device manufacturer that the user is using.                                                                                                                                                                                                                                    |
| `device_model` _string_                | _Optional._ The device model that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`                                                                                                                                                              |
| `carrier` _string_                     | _Optional._ The carrier that the user is using.Default mapping: `event?.contexts_nl_basjes_yauaa_context_1?.[0]?.carrier`                                                                                                                                                                      |
| `country` _string_                     | _Optional._ The current country of the user.Default mapping: `event?.geo_country`                                                                                                                                                                                                              |
| `region` _string_                      | _Optional._ The current region of the user.Default mapping: `event?.geo_region`                                                                                                                                                                                                                |
| `city` _string_                        | _Optional._ The current city of the user.Default mapping: `event?.geo_city`                                                                                                                                                                                                                    |
| `dma` _string_                         | _Optional._ The current Designated Market Area of the user.                                                                                                                                                                                                                                    |
| `language` _string_                    | _Optional._ The language set by the user.Default mapping: `event?.br_lang`                                                                                                                                                                                                                     |
| `price` _number_                       | _Optional._ The price of the item purchased. Required for revenue data if the revenue field isn't sent. You can use negative values for refunds                                                                                                                                                |
| `quantity` _integer_                   | _Optional._ The quantity of the item purchased.                                                                                                                                                                                                                                                |
| `revenue` _number_                     | _Optional._ Revenue = (price x quantity). If you send all 3 fields of price, quantity, and revenue, then the revenue value is (price x quantity). Use negative values for refunds.                                                                                                             |
| `productId` _string_                   | _Optional._ An identifier for the item purchased. You must send a price and quantity or revenue with this field.                                                                                                                                                                               |
| `revenueType` _string_                 | _Optional._ The type of revenue for the item purchased. You must send a price and quantity or revenue with this field.                                                                                                                                                                         |
| `location_lat` _number_                | _Optional._ The current Latitude of the user.Default mapping: `event?.geo_latitude`                                                                                                                                                                                                            |
| `location_lng` _number_                | _Optional._ The current Longitude of the user.Default mapping: `event?.geo_longitude`                                                                                                                                                                                                          |
| `ip` _string_                          | _Optional._ The IP address of the user.Default mapping: `event?.user_ipaddress`                                                                                                                                                                                                                |
| `idfa` _string_                        | _Optional._ (iOS) Identifier for Advertiser.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.appleIdfa`                                                                                                                                                 |
| `idfv` _string_                        | _Optional._ (iOS) Identifier for Vendor.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.appleIdfv`                                                                                                                                                     |
| `adid` _string_                        | _Optional._ (Android) Google Play Services advertising ID.Default mapping: `event?.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.androidIdf`                                                                                                                                  |
| `android_id` _string_                  | _Optional._ (Android) Android ID (not the advertising ID).                                                                                                                                                                                                                                     |
| `plan` _object_                        | _Optional._ Tracking plan properties.Properties:* `branch` (string, optional): The tracking plan branch name.
* `source` (string, optional): The tracking plan source.
* `version` (string, optional): The tracking plan version.                                                              |

---

# Forward events to Braze
> Send Snowplow events to Braze for real-time personalization and campaign automation using the Track Users API with support for user attributes, custom events, and purchases.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/braze/

Send Snowplow events to Braze for real-time personalization, user tracking, and campaign automation using Braze's [Track Users API](https://www.braze.com/docs/api/endpoints/user_data/post_user_track).

Snowplow supports the following Braze object types:

- **[User attributes](https://www.braze.com/docs/api/objects_filters/user_attributes_object)**: Profile data and custom user properties
- **[Custom events](https://www.braze.com/docs/api/objects_filters/event_object)**: User actions and behaviors
- **[Purchases](https://www.braze.com/docs/api/objects_filters/purchase_object)**: Transaction data with product details

## Prerequisites

Before setting up the forwarder in Console, you'll need the following from your Braze account:

- Braze REST API key with these permissions:

  - `users.track`
  - `users.alias.new`
  - `users.identify`
  - `users.export.ids`
  - `users.merge`
  - `users.external_ids.rename`
  - `users.alias.update`

- Braze REST API endpoint, found in Braze under **Settings** > **APIs and Identifiers**

## Getting started

### Configure the destination

To create the forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Braze** for the connection type and enter your API key and endpoint.

When configuring the forwarder, you can choose from the following **Braze object types** to map:

- **[Attributes](https://www.braze.com/docs/api/objects_filters/user_attributes_object)**: update user profile data
- **[Events](https://www.braze.com/docs/api/objects_filters/event_object)**: send custom user actions
- **[Purchases](https://www.braze.com/docs/api/objects_filters/purchase_object)**: send transaction events

### Validate the integration

You can confirm events are reaching Braze by checking the following pages in your Braze account:

1. Query Builder: in Braze, navigate to **Analytics** > **Query Builder**. You can write queries on the following tables to preview the data forwarded from Snowplow: `USER_BEHAVIORS_CUSTOMEVENT_SHARED`, `USERS_BEHAVIORS_PURCHASE_SHARED`.
2. API Usage Dashboard: in Braze, navigate to **Settings** > **API and Identifiers** to see a chart of API usage over time. You can filter specifically for the API key used by Snowplow and see both successes and failures.

## Sending custom properties

You can send custom properties beyond the standard fields defined in the schema reference below. The structure depends on which Braze object type you're using:

- **User attributes**: add as top-level fields (e.g., `subscription_tier`, `loyalty_points`)
- **Event properties**: nest under `properties` object (e.g., `properties.plan_type`, `properties.feature_flag`)
- **Purchase properties**: nest under `properties` object (e.g., `properties.color`, `properties.size`)

For property names containing spaces, use bracket notation (e.g., `["account type"]` or `properties["campaign source"]`).

See Braze's [Event Object documentation](https://www.braze.com/docs/api/objects_filters/event_object) for details on supported data types, property naming requirements, and payload size limits. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Limitations

**Rate limits:** Braze enforces a rate limit of 3,000 API calls every three seconds for the Track Users API. Because Snowplow does not currently support batching for event forwarders, this API rate limit also functions as the event rate limit. If your input throughput exceeds 3,000 events per three seconds, you will experience increased latency.

## Schema reference

The sections below contain information on the fields you can send to Braze, including field names, data types, required fields, and default Snowplow mapping expressions.

### User attributes

Use the user attributes object to update standard user profile fields or add your own custom attribute data to the user.

| Field                                     | Details                                                                                                                                                                                                                          |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_update_existing_only` _boolean_         | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.Default mapping: `false`                                                                                               |
| `user_alias` _object_                     | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.               |
| `external_id` _string_                    | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.Default mapping: `event?.user_id`                                                                                |
| `braze_id` _string_                       | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                          |
| `country` _string_                        | _Optional._ ISO-3166-1 alpha-2 standard standard country code. Where the value does not meet that standard, Braze attempts to map it to a country. Where it cannot, the value will be NULL.Default mapping: `event?.geo_country` |
| `current_location` _object_               | _Optional._Properties:* `longitude` (number, optional): Longitude of the user's location.
* `latitude` (number, optional): Latitude of the user's location.                                                                      |
| `date_of_first_session` _string_          | _Optional._ Date at which the user first used the app. Must be ISO 8601 format.                                                                                                                                                  |
| `date_of_last_session` _string_           | _Optional._ Date at which the user most recently used the app. Must be ISO 8601 format.                                                                                                                                          |
| `dob` _string_                            | _Optional._ The user's date of birth.                                                                                                                                                                                            |
| `email` _string_                          | _Optional._ The user's email address.                                                                                                                                                                                            |
| `email_subscribe` _string_                | _Optional._ The user's email subscription status. Must be one of: `opted_in`, `unsubscribed`, `subscribed`                                                                                                                       |
| `email_open_tracking_disabled` _boolean_  | _Optional._ Set to true to disable the open tracking pixel from being added to all future emails sent to this user.                                                                                                              |
| `email_click_tracking_disabled` _boolean_ | _Optional._ Set to true to disable the click tracking for all links within future emails sent to this user.                                                                                                                      |
| `first_name` _string_                     | _Optional._ User's first name.                                                                                                                                                                                                   |
| `gender` _string_                         | _Optional._ The user's gender. Must be one of: `M`, `F`, `O`, `N`, `P`, `null`                                                                                                                                                   |
| `home_city` _string_                      | _Optional._ The user's city.                                                                                                                                                                                                     |
| `language` _string_                       | _Optional._ The user's preferred language. Must be an ISO-639-1 standard language code.Default mapping: `event?.br_lang`                                                                                                         |
| `last_name` _string_                      | _Optional._ User's last name.                                                                                                                                                                                                    |
| `marked_email_as_spam_at` _string_        | _Optional._ Date at which the user’s email was marked as spam. Must be ISO 8601 format.                                                                                                                                          |
| `phone` _string_                          | _Optional._ The user's phone number.                                                                                                                                                                                             |
| `push_subscribe` _string_                 | _Optional._ The user's push message subscription status. Must be one of: `opted_in`, `unsubscribed`, `subscribed`                                                                                                                |
| `push_tokens` _array of object_           | _Optional._ Array of objects with app\_id and token string.                                                                                                                                                                      |
| `time_zone` _string_                      | _Optional._ The user's time zone. Must be a valid IANA Time Zone.Default mapping: `event?.geo_timezone`                                                                                                                          |

### Events

Each event object represents a single occurrence of a custom event by a particular user at the specific time. You can set and use custom event properties in messages, data collection, and personalization.

| Field                             | Details                                                                                                                                                                                                                                |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `time` _string_                   | _Required._ Time of the event, required. Must be ISO 8601 format.Default mapping: `spTstampToJSDate(event?.collector_tstamp)?.toISOString()`                                                                                           |
| `name` _string_                   | _Required._ Name of the type of event.Default mapping: `event?.event_name`                                                                                                                                                             |
| `external_id` _string_            | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.Default mapping: `event?.user_id`                                                                                      |
| `braze_id` _string_               | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                                |
| `phone` _string_                  | _Optional._ The user's phone number.                                                                                                                                                                                                   |
| `user_alias` _object_             | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.                     |
| `_update_existing_only` _boolean_ | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.Default mapping: `false`                                                                                                     |
| `email` _string_                  | _Optional._ The user's email address.                                                                                                                                                                                                  |
| `app_id` _string_                 | _Optional._ Associates activity with a specific app in your Braze workspace. If set, should match a Braze App Identifier, found in Braze console's API section. Can be omitted, but incorrect values may result in data loss in Braze. |
| `properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the event in Braze.                                                                                                                                                                  |

### Purchases

The purchase object represents a user purchasing a single item by a user at a particular time. Each purchase object is located within a purchase array, which can represent a transaction with multiple items. The purchase object has fields that allow the Braze back-end to store and use this information for messages, data collection, and personalization.

| Field                             | Details                                                                                                                                                                                                                                |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `time` _string_                   | _Required._ Time of the purchase, required. Must be ISO 8601 format.                                                                                                                                                                   |
| `product_id` _string_             | _Required._ Identifier for the product purchased.                                                                                                                                                                                      |
| `currency` _string_               | _Required._ ISO 4217 Alphabetic Currency Code.                                                                                                                                                                                         |
| `price` _number_                  | _Required._ Price per item.                                                                                                                                                                                                            |
| `external_id` _string_            | _Optional._ A unique user identifier. Required if `user alias`, `braze_id`, `email`, or `phone` is not provided.                                                                                                                       |
| `braze_id` _string_               | _Optional._ Identifier reserved for the Braze SDK. Required if `external_id`, `user alias`, `email`, or `phone` is not provided. Not recommended for use with the Snowplow integration.                                                |
| `phone` _string_                  | _Optional._ The user's phone number.                                                                                                                                                                                                   |
| `user_alias` _object_             | _Optional._ Braze user alias object.Properties:* `alias_name` (string, required): The actual value of the alias identifier.
* `alias_label` (string, required): Indicates the type of alias. E.g. `domain_userid`.                     |
| `_update_existing_only` _boolean_ | _Optional._ If `true`, API requests will only update existing user profiles in Braze. `false` recommended.                                                                                                                             |
| `quantity` _integer_              | _Optional._ Quantity of the item purchased. Braze treats this as multiple individual purchases.                                                                                                                                        |
| `email` _string_                  | _Optional._ The user's email address.                                                                                                                                                                                                  |
| `app_id` _string_                 | _Optional._ Associates activity with a specific app in your Braze workspace. If set, should match a Braze App Identifier, found in Braze console's API section. Can be omitted, but incorrect values may result in data loss in Braze. |
| `properties` _object_             | _Optional._ Arbitrary key-value pairs assigned to the purchase in Braze.                                                                                                                                                               |

---

# Pre-built event forwarding integrations
> Pre-built event forwarding integrations for third-party destinations with authentication, field mapping, and API-specific configurations included.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/

Event forwarding supports third-party destinations through pre-built integrations that handle authentication, field mapping, and API-specific requirements.

## Available destinations

Snowplow event forwarding supports the following destinations:

- [Amplitude](/docs/destinations/forwarding-events/integrations/amplitude/)
- [Braze](/docs/destinations/forwarding-events/integrations/braze/)
- [Mixpanel](/docs/destinations/forwarding-events/integrations/mixpanel/)

---

# Forward events to Mixpanel
> Send Snowplow events to Mixpanel for product analytics and user behavior insights using the Import API with support for event tracking and custom properties.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/integrations/mixpanel/

Send Snowplow events to Mixpanel to power product analytics, user behavior tracking, and funnel analysis using Mixpanel's [Import API](https://developer.mixpanel.com/reference/import-events).

## Prerequisites

Before setting up the forwarder in Console, you'll need the following from your Mixpanel account:

- **Project ID**: found in Mixpanel under **Settings** > **Project Settings**
- **Service Account Username**: create a service account in Mixpanel under **Settings** > **Organization Settings** > **Service Accounts**. The service account must have either **Admin** or **Owner** permissions.
- **Service Account Password**: generated when you create the service account

> **Tip:** To avoid introducing test data in your production Mixpanel project, we recommend using a test or development Mixpanel project to test your transformations first. Then, create a new Connection in Console with your production credentials, and a new forwarder that imports the configuration from your development forwarder.

## Getting started

### Configure the destination

To create the connection and forwarder, follow the steps in [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/).

When configuring the connection, select **Mixpanel** for the connection type, enter your project ID, service account username, and service account password. You'll also need to select the **Server Location** where your Mixpanel project is hosted:

- **United States**: `https://api.mixpanel.com/`
- **European Union**: `https://api-eu.mixpanel.com/`
- **India**: `https://api-in.mixpanel.com/`

### Validate the integration

You can confirm events are reaching Mixpanel by checking the **Events** page in your Mixpanel account:

1. From the left navigation bar, click **Events**
2. You should see your Snowplow events appearing in the live view
3. You can also navigate to **Settings** > **Project Settings** > **Data & Exports** to view import statistics

## Identity management

Mixpanel uses a combination of `distinct_id` (user identifier) and `$device_id` (device identifier) to track users across sessions. The Snowplow integration defaults to using `user_id` for `distinct_id` and a coalesce of `domain_userid` and `client_sesion.user_id` for `$device_id`, which supports Mixpanel's [Simplified ID Merge system](https://docs.mixpanel.com/docs/tracking-methods/id-management#simplified-id-merge-system).

When a user logs in and your event contains a `user_id` value, Mixpanel will automatically merge the user's anonymous activity (tracked via `$device_id`) with their identified profile (tracked via `distinct_id`).

## Sending custom properties

You can send custom event properties beyond the standard fields defined in the schema reference below. Custom properties are nested under the `properties` object. When configuring your forwarder, add field mappings formatted as `properties.your_custom_field` (e.g., `properties.plan_type`, `properties.feature_flag`).

For property names containing spaces, use bracket notation (e.g., `properties["referred by"]`).

See Mixpanel's [Import Events API documentation](https://developer.mixpanel.com/reference/import-events) for details on supported data types and property requirements. See [Creating forwarders](/docs/destinations/forwarding-events/creating-forwarders/) for details on configuring field mappings.

## Schema reference

This section contains information on the fields you can send to Mixpanel, including field names, data types, required fields, and default Snowplow mapping expressions.

| Field                 | Details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `event` _string_      | _Required._ The name of the eventDefault mapping: `event.event_name`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `properties` _object_ | _Required._Properties:* `time` (integer, required): The time at which the event occurred, in milliseconds since epoch. Default mapping: `spTstampToEpochMillis(event.derived_tstamp)`
* `distinct_id` (string, required): Identifies the user who performed the event. Default mapping: `event.user_id`
* `$insert_id` (string, required): A unique identifier for the event, used for deduplication. Default mapping: `event.event_id`
* `ip` (string, optional): The IP address of the user. Default mapping: `event.user_ipaddress`
* `$city` (string, optional): The city of the event sender. Default mapping: `event.geo_city`
* `$region` (string, optional): The region (state or province) of the event sender. Default mapping: `event.geo_region`
* `mp_country_code` (string, optional): The country of the event sender. Default mapping: `event.geo_country`
* `$browser` (string, optional): Name of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentName`
* `$browser_version` (string, optional): Version of the browser. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.agentVersion`
* `$current_url` (string, optional): The full URL of the webpage on which the event is triggered. Default mapping: `event.page_url`
* `$referrer` (string, optional): Referring URL including your own domain. Default mapping: `event.page_referrer`
* `$referring_domain` (string, optional): Referring domain including your own domain. Default mapping: `event.refr_urlhost`
* `$device` (string, optional): Name of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`
* `$device_id` (string, optional): A unique device identifier. Used in Mixpanel's Simplified ID Merge API. Default mapping: `event.domain_userid ?? event.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0]?.userId`
* `$screen_height` (integer, optional): The height of the device screen in pixels. Default mapping: `event.dvce_screenheight`
* `$screen_width` (integer, optional): The width of the device screen in pixels. Default mapping: `event.dvce_screenwidth`
* `$os` (string, optional): Operating system of the device. Default mapping: `event.os_name`
* `$os_version` (string, optional): Operating system version. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.operatingSystemVersion`
* `$manufacturer` (string, optional): Manufacturer of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceBrand`
* `$model` (string, optional): Model of the device. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.deviceName`
* `$app_version_string` (string, optional): The app version. Default mapping: `event.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.version`
* `$app_build_number` (string, optional): Build number for the mobile app. Default mapping: `event.contexts_com_snowplowanalytics_mobile_application_1?.[0]?.build`
* `$carrier` (string, optional): Wireless carrier of the device owner. Default mapping: `event.contexts_nl_basjes_yauaa_context_1?.[0]?.carrier`
* `$radio` (string, optional): The current cellular network communication standard (3G, 4G, LTE, etc). Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.networkTechnology`
* `$wifi` (boolean, optional): Set to true if the user's device is connected to wifi. Default mapping: `event.contexts_com_snowplowanalytics_snowplow_mobile_context_1?.[0]?.networkType === 'wifi'`
* `$user_id` (string, optional): The identified ID of the user. Used in Mixpanel's Simplified ID Merge API. Default mapping: `event.user_id`
* `$lib_version` (string, optional): Tracker library version. Default mapping: `event.v_tracker`
* `utm_source` (string, optional): UTM source parameter from the URL. Default mapping: `event.mkt_source`
* `utm_medium` (string, optional): UTM medium parameter from the URL. Default mapping: `event.mkt_medium`
* `utm_campaign` (string, optional): UTM campaign parameter from the URL. Default mapping: `event.mkt_campaign`
* `utm_content` (string, optional): UTM content parameter from the URL. Default mapping: `event.mkt_content`
* `utm_term` (string, optional): UTM term parameter from the URL. Default mapping: `event.mkt_term`
* `mp_lib` (string, optional): Tracker library that sent the event. Default mapping: `'Snowplow: ' + event.v_tracker`
* `mp_sent_by_lib_version` (string, optional): Mixpanel library version used to send data (not necessarily the same as the version which enqueued the data).
* `$screen_dpi` (integer, optional): Pixel density of the device screen.
* `$initial_referrer` (string, optional): Referring URL when the user first arrived on your site. Defaults to “$direct” if the user is not referred.
* `$initial_referring_domain` (string, optional): Referring domain at first arrival. Defaults to “$direct” if the user is not referred.
* `$search_engine` (string, optional): The search engine that the customer used when they arrived at your domain.
* `mp_keyword` (string, optional): Search keywords detected on the referrer from a search engine to your domain. This property is only collected when search keywords are included in a URL.
* `$watch_model` (string, optional): The model of the iOS watch.
* `$bluetooth_enabled` (boolean, optional): Set to true if Bluetooth is enabled, false if not.
* `$bluetooth_version` (string, optional): Set to “none”, “ble”, or “classic”.
* `$has_nfc` (boolean, optional): The device supports Near Field Communication(NFC). Set to true if the device hardware supports NFC, false if not.
* `$has_telephone` (boolean, optional): Set to true if this device has telephone functionality, false if not. |

---

# Manage event forwarders in Console
> Edit, clone, and delete Snowplow event forwarders in Console to update configurations, duplicate setups, or remove unused destinations.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/managing-forwarders/

This page explains how to edit, clone, and delete event forwarders. To start, go to **Destinations** > **Destination list** in [Snowplow Console](https://console.snowplowanalytics.com).

## Edit a forwarder

To edit a forwarder:

1. Click **Details** under the destination you want to change to open the destination details page.
2. On the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Edit**. You will see the forwarder configuration page.
3. Modify the forwarder configuration as needed. When you're done, select **Deploy** to re-deploy the forwarder with the updated configuration. The forwarder instances will be re-deployed on a rolling basis over the next few minutes.

## Rename a forwarder

To rename a forwarder:

1. Click **Details** under the destination you want to change to open the destination details page.
2. On the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Rename**.
3. Enter a new forwarder name and select **Rename** to save.

## Clone a forwarder

When creating a new forwarder, you can import the configuration from an existing forwarder of the same type. This is especially helpful when migrating a forwarder setup from a development pipeline to production.

To clone a forwarder:

1. Navigate to the **Available** tab and select **Configure** on the destination card from the list of available integrations to start setting up the forwarder.
2. Give the forwarder a **name**, select the **pipeline** you want the forwarder to read events from, and choose a **connection**.
3. From the **Import configuration from** dropdown, choose an existing forwarder.
4. Click **Continue**. The filters, mappings, and custom functions will be pre-populated with those of the existing forwarder you imported from.

## Delete a forwarder

To permanently delete a forwarder:

1. Click **Details** under the destination you want to change to open the destination details page.
2. On the event forwarders overview table, click the three dots next to the forwarder you want to change and select **Delete**.
3. On the confirmation modal, select **Delete**. This will start the process of destroying the underlying forwarder infrastructure.

---

# Event forwarding filter and mapping reference
> Complete reference for Snowplow event forwarding JavaScript expressions, field mapping syntax, event filtering, data transformations, and custom functions.
> Source: https://docs.snowplow.io/docs/destinations/forwarding-events/reference/

Event forwarders use JavaScript expressions for filtering events and mapping Snowplow data to destination fields. These expressions are entered during [forwarder setup](/docs/destinations/forwarding-events/#getting-started) in Console, specifically in the **Event filtering**, **Field mapping**, and **Custom functions** sections. This reference covers the syntax and available data for these operations.

## Available event fields

You can reference any field in your Snowplow events for both filters and field mappings.

### Standard atomic fields

Access [standard Snowplow fields](/docs/fundamentals/canonical-event/) in your filters and mappings using JavaScript dot notation:

```javascript
// Standard atomic fields
event.app_id
event.event_name
event.platform
event.collector_tstamp
event.event_id
event.domain_userid
event.user_id
event.page_url
event.page_title
event.useragent
event.network_userid
```

### Custom events and entities

You can also access fields in Snowplow or custom event and entity schemas.

Forwarders transform Iglu schema URIs to JavaScript-safe field names:

| Original schema                            | Transformed field name               |
| ------------------------------------------ | ------------------------------------ |
| `com.acme/signup/jsonschema/1-0-0`         | `unstruct_event_com_acme_signup_1`   |
| `com.acme/user_profile/jsonschema/2-1-0`   | `contexts_com_acme_user_profile_2`   |
| `nl.basjes/yauaa_context/jsonschema/1-0-4` | `contexts_nl_basjes_yauaa_context_1` |

Schema names follow these transformation rules:

- Self-describing events: `unstruct_event_` prefix
- Entities: `contexts_` prefix
- Dots and slashes become underscores
- Only major version number retained
- Hyphens in vendor/name become underscores

> **Info:** Always use [optional chaining](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) (`?.`) when accessing custom events and entities to handle cases where they're not present.

For a self-describing event with schema `com.acme/signup/jsonschema/1-0-0`:

```javascript
// Access event properties
event?.unstruct_event_com_acme_signup_1?.signup_method
event?.unstruct_event_com_acme_signup_1?.user_type
```

For entities with schema `com.acme/user_profile/jsonschema/1-0-0`:

```javascript
// Access entity properties (entities are arrays)
event?.contexts_com_acme_user_profile_1?.[0]?.subscription_tier
event?.contexts_com_acme_user_profile_1?.[0]?.account_created
```

## Event filtering

Event filters determine which events are forwarded to your destination. Only events matching your filter criteria (JavaScript expression evaluating to `true`) will be processed and sent.

### Basic filters

Filter events can contain standard JavaScript comparison operators:

```javascript
// Single condition
event.app_id == "website"

// Multiple conditions with AND
event.app_id == "website" && event.event == "page_view"

// Multiple conditions with OR
event.event_name == "add_to_cart" || event.event_name == "purchase"

// Check if event is in a list
["page_view", "add_to_cart", "purchase"].includes(event.event_name)

// Exclude events
event.app_id == "website" && event.event_name != "link_click"
```

### Advanced filtering patterns

Regular expressions:

```javascript
// Match multiple domains
event.page_urlhost.match(/mysite\.(com|fr|de)/)

// Match event name patterns
event.event_name.match(/^purchase_/)

// Match custom field patterns
event?.unstruct_event_com_acme_product_view_1?.category.match(/electronics|computers/)
```

Define reusable logic in the Custom Functions section:

```javascript
// Defined in the Custom Function editor panel
function isHighValueUser(event) {
  const profile = event?.contexts_com_acme_user_profile_1?.[0];
  return profile?.subscription_tier == "premium" ||
         profile?.lifetime_value > 1000;
}

// Use in filter
isHighValueUser(event) && event.event_name == "purchase"
```

## Field mapping

Field mapping defines how Snowplow event data is transformed and sent to destination APIs. Each mapping consists of:

- A destination field name (key)
- A JavaScript expression that extracts the value from your Snowplow event (value)

> **Info:** The code snippets below contain JavaScript expressions that you can include in the **Snowplow expression** mapping field in the UI.

### Basic mappings

Map standard event fields directly:

![](/assets/images/event-forwarding-basic-mapping-be50ef0162cd510af04538089519f6ee.png)

```json
// sample output
{ "event_type": "page_view" }
```

You can also apply fallback and conditional logic:

![](/assets/images/event-forwarding-conditional-mapping-b21729fb908ecd1cf784bb8f2952e6f9.png)

```json
// sample output
{
  "user_id": "a50d3dfe-ba21-432e-a165-1a1d2d633693"
  "source": "website"
}
```

You can also send static values:

![](/assets/images/event-forwarding-static-mapping-89a993963e026dc4e9fc4a6bc421757c.png)

```json
// sample output
{ "source": "snowplow" }
```

### Data transformation

Convert data types, such as strings, boolean values, and dates:

![](/assets/images/event-forwarding-type-conversions-cfc98aa6623258f0d8ec29cd32ef2576.png)

```json
// sample output
{
  "page_width": 720,
  "page_height": 600,
  "is_mobile": true,
  "timestamp": "2025-10-01T18:35:38.563Z"
}
```

Use standard [Javascript String methods](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#instance_methods) to manipuate strings:

```javascript
// Case conversion
event.event_name.toLowerCase()
event.page_title.toUpperCase()

// String operations
event.page_url.replace("http://", "https://")
event.page_title.substring(0, 100)
event.page_urlpath.split('/')
```

Map to nested objects using dot notation in field names:

![](/assets/images/event-forwarding-nested-mapping-9f700b7f85d97a5edfe10bc78f9ae348.png)

```json
// sample output
{
  "user": {
    "id": "user123"
  },
  "properties": {
    "page_title": "Home Page",
    "page_url": "https://example.com",
    "referrer": "google.com"
  }
}
```

### Custom mapping functions

Define complex transformations as functions in the custom functions section. You can then reference these functions in filters and mappings. Below are a few example transformations:

```javascript
// Event name formatting
function formatEventName(event) {
  const nameMap = {
    'page_view': 'Page Viewed',
    'add_to_cart': 'Product Added to Cart',
    'purchase': 'Purchase Completed'
  };
  return nameMap[event.event_name] || event.event_name;
}

// Extract product data
function extractProductInfo(event) {
  const product = event?.unstruct_event_com_acme_product_1;
  if (!product) return null;

  return {
    id: product.product_id,
    name: product.product_name,
    category: product.category,
    price: parseFloat(product.price),
    currency: product.currency || 'USD'
  };
}

// User profile enrichment
function buildUserProfile(event) {
  const session = event?.contexts_com_snowplowanalytics_snowplow_client_session_1?.[0];
  const geo = event?.contexts_com_snowplowanalytics_snowplow_geolocation_context_1?.[0];

  return {
    user_id: event.domain_userid,
    session_id: session?.sessionId,
    location: geo ? `${geo.latitude},${geo.longitude}` : null,
    user_agent: event.useragent,
    platform: event.platform
  };
}
```

## Other common patterns

Timestamp formatting:

```javascript
// ISO 8601 format
new Date(event.collector_tstamp).toISOString()

// Unix timestamp
Math.floor(new Date(event.collector_tstamp).getTime() / 1000)

// Readable format
new Date(event.collector_tstamp).toLocaleDateString()
```

Conditional field mapping:

```javascript
// Platform-specific mapping
event.platform == "web" ? event.page_url : event.screen_name

// Event-type specific properties
event.event_name == "purchase" ?
  event.unstruct_event_com_acme_purchase_1?.total_value :
  null
```

Array handling:

```javascript
// Get first entity
event?.contexts_com_acme_product_1?.[0]?.product_name

// Map all entities
event?.contexts_com_acme_product_1?.map(p => p.product_id)

// Filter and transform entities
event?.contexts_com_acme_product_1
  ?.filter(p => p.price > 10)
  ?.map(p => ({ id: p.product_id, name: p.product_name }))
```

---

# Data destinations overview
> Send Snowplow data to warehouses, lakes, and third-party platforms with event forwarding, native integrations, and reverse ETL for data activation.
> Source: https://docs.snowplow.io/docs/destinations/

Read more about Snowplow data destinations [here](/docs/fundamentals/destinations/).

[Data warehouses and data lakes](/docs/destinations/warehouses-lakes/) are primary destinations for Snowplow data.

Snowplow also supports many ways to [forward events](/docs/destinations/forwarding-events/) to a variety of platforms. Snowplow has native integrations as well as supporting Snowplow, vendor and community authored destinations via Google Tag Manager Server Side.

Finally, [reverse ETL](/docs/destinations/reverse-etl/) enables you to publish data from your warehouse directly to marketing platforms, where it can be activated.

---

# Reverse ETL for data activation
> Activate warehouse data by syncing it to marketing platforms with reverse ETL, enabling sophisticated audience targeting based on predictive models and behavioral insights.
> Source: https://docs.snowplow.io/docs/destinations/reverse-etl/

Snowplow and Reverse ETL represents best in class tooling for companies executing more sophisticated use cases with their behavioral data.

As one example of where this approach is beneficial, many organizations begin with marketing use cases by creating simple segments but quickly want to target their ads more effectively wanting to incorporate customer propensity to buy and predictive lifetime value. That increase in sophistication can only come from building a deep understanding of users in a place like a data warehouse with modeling tooling (AI/ML) and Reverse ETL.

This can only be done repeatedly, and with confidence with excellent governance practises which comes from Snowplow’s compliance controls (i.e. controlling which data is sent to 3rd parties), schematised workflows and UI/API for management. Targeting in a sophisticated way ensure resource is allocated best as well (e.g. don’t target users who have already purchased).

Snowplow and Reverse ETL is for organizations who want to:

- Adapt to changes in customer behavior and the business questions being asked.
- Use Rich, extensible behavioral data.
- Maintain high quality data due to validation and private cloud deployment.
- Activate very high value audiences based on propensity to convert.
- Execute well on dozens of other use cases.

![](/assets/images/reverseetl-70742060395d2f5b62046145cceaf1e2.png)

## What problem does Reverse ETL solve?

Organizations have invested in building a high quality data asset in their data warehouse to power numerous use cases, so naturally want to use this to effectively target their users.

Reverse ETL enables organizations to take the output of the intelligence they've built using all their customer data (behavioral and non-behavioral) and publish that directly to marketing platforms where it can be activated.

## Reverse ETL Platforms

Reverse ETL helps organizations operationalize the data in their warehouse by syncing it with other SaaS solutions such as Salesforce and Google Ads.

Snowplow partners with and recommends Census as Reverse ETL platforms to allow organisations to achieve the use cases described above.

- [Census](https://www.getcensus.com/)

---

# Load Snowplow data to BigQuery
> Send Snowplow data to BigQuery for analytics and data warehousing with automatic table creation, schema evolution, and cross-batch deduplication.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/bigquery/

> **Info:** The BigQuery integration is available for Snowplow pipelines running on **AWS** and **GCP**.

The Snowplow BigQuery integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) directly into your BigQuery datasets for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Google Cloud Project ID and region
- Allow-list Snowplow IP addresses
- Specify the desired dataset name
- Create a service account with the `roles/bigquery.dataEditor` permission (more permissions will be required for loading failed events and setting up [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))

## Getting started

You can add a BigQuery destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **BigQuery**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **BigQuery**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

> **Tip:** For more details on the loading flow, see the [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in BigQuery

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                |
| ------- | --- | ------------------------------------------------------- | -------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `RECORD`) | data for your custom `product` entities (as `REPEATED RECORD`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=bigquery) Snowplow data.

---

# Load Snowplow data to Databricks
> Send Snowplow data to Databricks for analytics and data processing with Delta Lake tables, schema evolution, and lakehouse architecture support.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/databricks/

> **Info:** The Databricks integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

The Snowplow Databricks integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into your Databricks environment for analytics, data modeling, and more.

Depending on the cloud provider for your Snowplow pipeline, there are different options for this integration:

| Integration                                                                                                         | AWS | Azure | GCP | Failed events support |
| ------------------------------------------------------------------------------------------------------------------- | --- | ----- | --- | --------------------- |
| Direct, batch-based ([RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/))                | ✅   | ❌     | ❌   | ❌                     |
| Via Delta Lake ([Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/))                            | ❌¹  | ✅²    | ✅²  | ✅                     |
| Streaming / Lakeflow ([Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/)) | ✅   | ✅     | ✅   | ✅                     |

_¹ Delta+Databricks combination is currently not supported for AWS pipelines. The loader uses DynamoDB tables for mutually exclusive writes to S3, a feature of Delta. Databricks, however, does not support this (as of September 2025). This means that it’s not possible to alter the data via Databricks (e.g. to run `OPTIMIZE` or to delete PII)._

_² The lake must be in the same cloud as the pipeline._

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to do a few things.

**Batch-based (AWS):**

- Provide a Databricks cluster along with its URL

- Specify the Unity catalog name and schema name

- Create an access token with the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `CAN USE` on the SQL warehouse

**Via Delta Lake (Azure, GCP):**

See [Delta Lake](/docs/destinations/warehouses-lakes/delta/).

**Streaming:**

- Create an S3 or GCS bucket or ADLS storage container, located in the same cloud and region as your Databricks instance

- Create a storage credential to allow Databricks to access the bucket or container

- Create an external location and a volume within Databricks pointing to the above

- Provide a Databricks SQL warehouse URL, Unity catalog name and schema name

- Create a service principal and grant the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `READ VOLUME` and `WRITE VOLUME` on the volume
  - `CAN USE` on the SQL warehouse (for testing the connection and monitoring, e.g. as part of the [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))

Note that Lakeflow features require a Premium Databricks account. You might also need Databricks metastore admin privileges for some of the steps.

***

## Getting started

You can add a Databricks destination through the Snowplow Console.

**Batch-based (AWS):**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Databricks**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

**Via Delta Lake (Azure, GCP):**

Follow the instructions for [Delta Lake](/docs/destinations/warehouses-lakes/delta/#getting-started).

Then create an external table in Databricks pointing to the Delta Lake location.

**Streaming:**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Databricks Streaming**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

Once the loader is up and running, click on the “...” button in the **Loaders** table and select **Databricks setup instructions**. Follow the outlined steps to create a Lakeflow pipeline within Databricks.

***

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

**Batch-based (AWS):**

For more details on the loading flow, see the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) reference page, where you will find additional information and diagrams.

**Via Delta Lake (Azure, GCP):**

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

**Streaming:**

For more details on the loading flow, see the [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) reference page, where you will find additional information and diagrams.

***

## Snowplow data format in Databricks

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data.

---

# Load Snowplow data to Delta Lake
> Send Snowplow data to Delta Lake for analytics and data processing with ACID transactions, schema evolution, and time travel capabilities.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/delta/

> **Info:** The Delta Lake integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

Delta Lake is an open table format for data lake architectures. The Snowplow Delta integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into Delta tables in your data lake for analytics, data modeling, and more.

Data in Delta Lake can be consumed using various tools and products, for example:

- Amazon Athena
- Apache Spark or Amazon EMR
- Databricks¹
- Microsoft Synapse Analytics
- Microsoft Fabric

_¹ Delta+Databricks combination is currently not supported for AWS pipelines. The loader uses DynamoDB tables for mutually exclusive writes to S3, a feature of Delta. Databricks, however, does not support this (as of September 2025). This means that it’s not possible to alter the data via Databricks (e.g. to run `OPTIMIZE` or to delete PII)._

> **Note:** Currently, we only support loading to a lake in the same cloud as your Snowplow pipeline.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

**AWS:**

- Provide an S3 bucket

- Create a DynamoDB table (required for file locking)

- Create an IAM role with the following permissions:

  - For the S3 bucket:

    - `s3:ListBucket`
    - `s3:GetObject`
    - `s3:PutObject`
    - `s3:DeleteObject`
    - `s3:ListBucketMultipartUploads`
    - `s3:AbortMultipartUpload`

  - For the DynamoDB table:

    - `dynamodb:DescribeTable`
    - `dynamodb:Query`
    - `dynamodb:Scan`
    - `dynamodb:GetItem`
    - `dynamodb:PutItem`
    - `dynamodb:UpdateItem`
    - `dynamodb:DeleteItem`

- Schedule a regular job to optimize the lake

**GCP:**

- Provide a GCS bucket
- Create a service account with the `roles/storage.objectUser` role on the bucket
- Create and provide a service account key

**Azure:**

- Provide an ADLS storage container
- Create a new App Registration with the `Storage Blob Data Contributor` permission
- Provide the registration tenant ID, client ID and client secret

***

## Getting started

You can add a Delta Lake destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/lake-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Delta**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Delta**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

We recommend scheduling regular [lake maintenance jobs](/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/?lake-format=delta) to ensure the best long-term performance.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Delta Lake

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data. (You will need a query engine such as Spark SQL or Databricks to query Delta tables.)

---

# Load Snowplow data to Apache Iceberg
> Send Snowplow data to Apache Iceberg data lakes for analytics and data processing with open table format, schema evolution, and cross-engine compatibility.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/iceberg/

> **Info:** The Iceberg integration is available for Snowplow pipelines running on **AWS** and **GCP** only.

Apache Iceberg is an open table format for data lake architectures. The Snowplow Iceberg integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into Iceberg tables in your data lake for analytics, data modeling, and more.

Iceberg data can be consumed using various tools and products, for example:

- Amazon Athena
- Amazon Redshift Spectrum
- Apache Spark or Amazon EMR
- Snowflake
- ClickHouse

We currently support the following catalogs:

| Catalog | AWS | GCP |
| ------- | --- | --- |
| Glue    | ✅   | ❌   |
| REST¹   | ✅   | ✅   |

_¹The REST catalog has only been tested with the Snowflake Open Catalog implementation._

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

**REST:**

- Specify your Snowflake Open Catalog account id and region, as well as namespace
- Create a service connection to the catalog and provide the client id and client secret

**AWS Glue:**

- Specify your AWS account ID

- Provide an S3 bucket and an AWS Glue database

- Create an IAM role with the following permissions:

  - For the S3 bucket:

    - `s3:ListBucket`
    - `s3:GetObject`
    - `s3:PutObject`
    - `s3:DeleteObject`

  - For the Glue database:

    - `glue:CreateTable`
    - `glue:GetTable`
    - `glue:UpdateTable`

- Schedule a regular job to optimize the lake

***

## Getting started

You can add an Iceberg destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/lake-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Iceberg**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Iceberg**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

For AWS Glue, we recommend scheduling regular [lake maintenance jobs](/docs/api-reference/loaders-storage-targets/lake-loader/maintenance/?lake-format=iceberg) to ensure the best long-term performance.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Iceberg

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data. (You will need a query engine such as Spark SQL or Snowflake to query Iceberg tables.)

---

# Supported warehouse and data lake destinations
> An overview of the available options for storing Snowplow data in data warehouses and lakes
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/

Data warehouses and data lakes are primary destinations for Snowplow data. For other options, see the [destinations overview](/docs/fundamentals/destinations/) page.

### Data warehouses

- [Snowflake](/docs/destinations/warehouses-lakes/snowflake/)
- [Databricks](/docs/destinations/warehouses-lakes/databricks/)
- [BigQuery](/docs/destinations/warehouses-lakes/bigquery/)
- [Redshift](/docs/destinations/warehouses-lakes/redshift/)

### Data lakes

- [Iceberg](/docs/destinations/warehouses-lakes/iceberg/)
- [Delta Lake](/docs/destinations/warehouses-lakes/delta/)

---

# How to query Snowplow data in the warehouse
> Introduction to querying Snowplow data in warehouses including self-describing events, entities, and handling duplicate events with SQL techniques.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/querying-data/

You will typically find most of your Snowplow data in the `events` table. If you are using Redshift, there will be extra tables for [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) — see [below](#self-describing-events).

Please refer to [the structure of Snowplow data](/docs/fundamentals/canonical-event/) for the principles behind our approach, as well as the descriptions of the various standard columns.

> **Tip:** Querying the `events` table directly can be useful for exploring your events or building custom analytics. However, for many common use cases it’s much easier to use our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/), which provide a pre-aggregated view of your data.

The simplest query could look like this:

```sql
SELECT * FROM <events>
WHERE event_name = 'page_view'
```

You will need to replace `<events>` with the appropriate location — the database, schema and table name will depend on your configuration.

> **Warning:** With large data volumes (read: any production system), you should always include a filter on the partition key (normally, `collector_tstamp`), for example:
>
> ```sql
> WHERE ... AND collector_tstamp between timestamp '2023-10-23' and timestamp '2023-11-23'
> ```
>
> This ensures that you read from the minimum number of (micro-)partitions necessary, making the query run much faster and reducing compute cost (where applicable).

## Self-describing events

[Self-describing events](/docs/fundamentals/events/#self-describing-events) can contain their own set of fields, defined by their [schema](/docs/fundamentals/schemas/).

**Redshift:**

For Redshift users, self-describing events are not part of the standard `events` table. Instead, each type of event is in its own table. The table name and the fields in the table will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query just the table for that particular self-describing event, if that's all that's required for your analysis, or join that table back to the `events` table:

```sql
SELECT
    ...
FROM
    <schema>.<events> ev
LEFT JOIN
    <schema>.my_example_event_table sde
    ON sde.root_id = ev.event_id AND sde.root_tstamp = ev.collector_tstamp
```

> **Warning:** You may need to take care of [duplicate events](#dealing-with-duplicates).

**BigQuery:**

Each type of self-describing event is in a dedicated `RECORD`-type column. The column name and the fields in the record will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT
    ...
    unstruct_event_my_example_event_1.my_field,
    ...
FROM
    <events>
```

> **Note:** The column name produced by previous versions of the BigQuery Loader (<2.0.0) would contain full schema version, e.g. `unstruct_event_my_example_event_1_0_0`. The [BigQuery Loader upgrade guide](/docs/api-reference/loaders-storage-targets/bigquery-loader/upgrade-guides/2-0-0-upgrade-guide/) describes how to enable the legacy column names in the 2.0.0 loader.

**Snowflake:**

Each type of self-describing event is in a dedicated `OBJECT`-type column. The column name will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT
    ...
    unstruct_event_my_example_event_1:myField::varchar, -- field will be variant type so important to cast
    ...
FROM
    <events>
```

**Databricks, Spark SQL:**

Each type of self-describing event is in a dedicated `STRUCT`-type column. The column name and the fields in the `STRUCT` will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event by extracting them like so:

```sql
SELECT
    ...
    unstruct_event_my_example_event_1.my_field,
    ...
FROM
    <events>
```

**Synapse Analytics:**

Each type of self-describing event is in a dedicated column in JSON format. The column name will be determined by the event’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query fields in the self-describing event like so:

```sql
SELECT
    ...
    JSON_VALUE(unstruct_event_my_example_event_1, '$.my_field')
    ...
FROM
    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events
```

***

## Entities

[Entities](/docs/fundamentals/entities/) (also known as contexts) provide extra information about the event, such as data describing a product or a user.

**Redshift:**

For Redshift users, entities are not part of the standard `events` table. Instead, each type of entity is in its own table. The table name and the fields in the table will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

The entities can be joined back to the core `events` table by the following, which is a one-to-one join (for a single record entity) or a one-to-many join (for a multi-record entity), assuming no duplicates.

```sql
SELECT
    ...
FROM
    <schema>.<events> ev
LEFT JOIN -- assumes no duplicates, and will return all events regardless of if they have this entity
    <schema>.my_entity ent
    ON ent.root_id = ev.event_id AND ent.root_tstamp = ev.collector_tstamp
```

> **Warning:** You may need to take care of [duplicate events](#dealing-with-duplicates).

**BigQuery:**

Each type of entity is in a dedicated `REPEATED RECORD`-type column. The column name and the fields in the record will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT
    ...
    contexts_my_entity_1[SAFE_OFFSET(0)].my_field AS my_field,
    ...
FROM
    <events>
```

Alternatively, you can use the [`unnest`](https://cloud.google.com/bigquery/docs/reference/standard-sql/arrays#flattening_arrays) function to explode out the array into one row per entity value.

```sql
SELECT
    ...
    my_ent.my_field AS my_field,
    ...
FROM
    <events>
LEFT JOIN
    unnest(contexts_my_entity_1) AS my_ent -- left join to avoid discarding events without values in this entity
```

> **Note:** Column name produced by previous versions of the BigQuery Loader (<2.0.0) would contain full schema version, e.g. `contexts_my_entity_1_0_0`.

**Snowflake:**

Each type of entity is in a dedicated `ARRAY`-type column. The column name will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT
    ...
    contexts_my_entity_1[0]:myField::varchar,  -- field will be variant type so important to cast
    ...
FROM
    <events>
```

Alternatively, you can use the [`lateral flatten`](https://docs.snowflake.com/en/sql-reference/functions/flatten) function to explode out the array into one row per entity value.

```sql
SELECT
    ...
    r.value:myField::varchar,  -- field will be variant type so important to cast
    ...
FROM
    <events> AS t,
    LATERAL FLATTEN(input => t.contexts_my_entity_1) r
```

**Databricks, Spark SQL:**

Each type of entity is in a dedicated `ARRAY<STRUCT>`-type column. The column name and the fields in the `STRUCT` will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT
    ...
    contexts_my_entity_1[0].my_field,
    ...
FROM
    <events>
```

Alternatively, you can use the [`LATERAL VIEW`](https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-lateral-view.html) clause combined with [`EXPLODE`](https://docs.databricks.com/sql/language-manual/functions/explode.html) to explode out the array into one row per entity value.

```sql
SELECT
    ...
    my_ent.my_field,
    ...
FROM
    <events>
    LATERAL VIEW EXPLODE(contexts_my_entity_1) AS my_ent
```

**Synapse Analytics:**

Each type of entity is in a dedicated column in JSON format. The column name will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

You can query a single entity’s fields by extracting them like so:

```sql
SELECT
    ...
    JSON_VALUE(contexts_my_entity_1, '$[0].my_field')
    ...
FROM
    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events
```

Alternatively, you can use the [`CROSS APPLY` clause combined with `OPENJSON`](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-parquet-nested-types#project-values-from-repeated-columns) to explode out the array into one row per entity value.

```sql
SELECT
    ...
    JSON_VALUE(my_ent.[value], '$.my_field')
    ...
FROM
    OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') as events
    CROSS APPLY OPENJSON(contexts_my_entity_1) AS my_ent
```

***

## Failed events

See [Exploring failed events](/docs/monitoring/exploring-failed-events/).

## Dealing with duplicates

In some cases, your data might contain duplicate events (full deduplication _before_ the data lands in the warehouse is optionally available for [Redshift, Snowflake and Databricks on AWS](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/deduplication/)).

While our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) deal with duplicates for you, there may be cases where you need to de-duplicate the events table yourself.

**Redshift:**

In Redshift, you must first generate a `ROW_NUMBER()` on your events and use this to de-duplicate.

```sql
WITH unique_events AS (
    SELECT
        ...
        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index
    FROM
        <events> a
)

SELECT
    ...
FROM
    unique_events
WHERE
    event_id_dedupe_index = 1
```

Things get a little more complicated if you want to join your event data with a table containing [entities](#entities).

Suppose your entity is called `my_entity`. If you know that each of your events has at most 1 such entity attached, the de-duplication requires the use of a row number over `event_id` to get each unique event:

```sql
WITH unique_events AS (
    SELECT
        ev.*,
        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index
    FROM
        <schema>.<events> ev
),

unique_my_entity AS (
    SELECT
        ent.*,
        ROW_NUMBER() OVER (PARTITION BY a.root_id ORDER BY a.root_tstamp) AS my_entity_index
    FROM
        <schema>.my_entity_1 ent
)

SELECT
    ...
FROM
    unique_events u_ev
LEFT JOIN
    unique_my_entity u_ent
    ON u_ent.root_id = u_ev.event_id AND u_ent.root_tstamp = u_ev.collector_tstamp AND u_ent.my_entity_index = 1
WHERE
    u_ev.event_id_dedupe_index = 1
```

If your events might have more than one `my_entity` attached, the logic is slightly more complex.

**Details**

First, de-duplicate the events table in the same way as above, but also keep track of the number of duplicates (see `event_id_dedupe_count` below). In the entity table, generate a row number per unique combination of _all_ fields in the record. Then join on `root_id` and `root_tstamp` as before, but with an _additional_ clause that the row number is a multiple of the number of duplicates, to support the 1-to-many join. This ensures all duplicates are removed while retaining all original records of the entity. This may look like a weird join condition, but it works.

Unfortunately, listing all fields manually can be quite tedious, but we have added support for this in the [de-duplication logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/) of our dbt packages.

```sql
WITH unique_events AS (
    SELECT
        ev.*,
        ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) AS event_id_dedupe_index,
        COUNT(*) OVER (PARTITION BY a.event_id) AS event_id_dedupe_count
    FROM
        <schema>.<events> ev
),

unique_my_entity AS (
    SELECT
        ent.*,
        ROW_NUMBER() OVER (PARTITION BY a.root_id, a.root_tstamp, ... /*all columns listed here for your entity */ ORDER BY a.root_tstamp) AS my_entity_index
    FROM
        <schema>.my_entity_1 ent
)

SELECT
    ...
FROM
    unique_events u_ev
LEFT JOIN
    unique_my_entity u_ent
    ON u_ent.root_id = u_ev.event_id AND u_ent.root_tstamp = u_ev.collector_tstamp AND mod(u_ent.my_entity_index, u_ev.event_id_dedupe_count) = 0
WHERE
    u_ev.event_id_dedupe_index = 1
```

**BigQuery:**

In BigQuery it is as simple as using a `QUALIFY` statement over your initial query:

```sql
SELECT
    ...
FROM <events> a
QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Snowflake:**

In Snowflake it is as simple as using a `qualify` statement over your initial query:

```sql
SELECT
    ...
FROM <events> a
QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Databricks, Spark SQL:**

In Databricks it is as simple as using a `qualify` statement over your initial query:

```sql
SELECT
    ...
FROM <events> a
QUALIFY ROW_NUMBER() OVER (PARTITION BY a.event_id ORDER BY a.collector_tstamp) = 1
```

**Synapse Analytics:**

In Synapse you must first generate a `ROW_NUMBER()` on your events and use this to de-duplicate.

```sql
WITH unique_events AS (
    SELECT
        ...
        ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY collector_tstamp) AS event_id_dedupe_index
    FROM
        OPENROWSET(BULK 'events', DATA_SOURCE = '<events>', FORMAT = 'DELTA') AS events
)

SELECT
    ...
FROM
    unique_events
WHERE
    event_id_dedupe_index = 1
```

***

---

# Load Snowplow data to Amazon Redshift
> Send Snowplow data to Amazon Redshift for analytics and data warehousing with automatic table creation, schema evolution, and optimized batch loading from S3.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/redshift/

> **Info:** The Redshift integration is available for Snowplow pipelines running on **AWS** only.

The Snowplow Redshift integration allows you to load enriched event data directly into your Redshift cluster (including Redshift serverless) for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Redshift cluster endpoint and connection details

- Allow-list Snowplow IP addresses

- Specify the desired database and schema names

- Create a user and a role with the following permissions:

  - Schema ownership (`CREATE SCHEMA ... AUTHORIZATION`)
  - `SELECT` on system tables (`svv_table_info`, `svv_interleaved_columns`, `stv_interleaved_counts`) — this is required for maintenance jobs

## Getting started

You can add a Redshift destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Redshift**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Redshift**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Redshift

The event data is split across multiple tables.

The main table (`events`) contains the [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For each type of self-describing event and entity, there are additional tables that can be joined with the main table:

**unstruct\_event\_com\_acme\_button\_press\_1**

| root\_id                             | root\_tstamp            | button\_name | button\_color | ... |
| ------------------------------------ | ----------------------- | ------------ | ------------- | --- |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Cancel       | red           | ... |

**contexts\_com\_acme\_product\_1**

| root\_id                             | root\_tstamp            | name   | price | ... |
| ------------------------------------ | ----------------------- | ------ | ----- | --- |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Salt   | 2.60  | ... |
| c6ef3124-b53a-4b13-a233-0088f79dcbcb | 2025-05-06 12:30:05.123 | Pepper | 3.10  | ... |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=redshift) Snowplow data.

---

# Load Snowplow data to Snowflake
> Send Snowplow data to Snowflake for analytics and data warehousing with automatic table management, schema evolution, and efficient data loading via Snowpipe or batch.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/snowflake/

> **Info:** The Snowflake integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

The Snowplow Snowflake integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) directly into your Snowflake warehouse for analytics, data modeling, and more.

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to:

- Provide your Snowflake account locator URL, cloud provider and region

- Allow-list Snowplow IP addresses

- Generate a key pair for key-based authentication

- Specify the desired database and schema names, as well as a warehouse name

- Create a role with the following permissions:

  - `USAGE`, `OPERATE` on warehouse (for testing the connection and monitoring, e.g. as part of the [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))
  - `USAGE` on database
  - `ALL` privileges on the target schema

## Getting started

You can add a Snowflake destination through the Snowplow Console. (For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Destinations** > **Connections**
2. Select **Set up connection**
3. Choose **Loader connection**, then **Snowflake**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Destination list**. Switch to the **Available** tab and select **Snowflake**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Destination list**.

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

For more details on the loading flow, see the [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) reference page, where you will find additional information and diagrams.

## Snowplow data format in Snowflake

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1                      | contexts\_com\_acme\_product\_1                                |
| ------- | --- | ----------------------------------------------------------------- | -------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as a `VARIANT` object) | data for your custom `product` entities (as a `VARIANT` array) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=snowflake) Snowplow data.

---

# Data structure management overview
> Create, manage, and update data structures (schemas) using Snowplow Console UI, Data Structures API, Snowplow CLI, or Iglu for community users.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/

This section explains how to create, manage, and update [data structures](/docs/fundamentals/schemas/) (schemas). Snowplow provides different options for data structure management:

- Snowplow Console UI
- [Data structures API](/docs/event-studio/programmatic-management/data-structures-api/)
- [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/)
- For Community users: [Iglu](/docs/api-reference/iglu/iglu-repositories/iglu-server/)

## Create a data structure

To create a data structure within Console, go to **Data collection** > **Data structures** and click the **Create data structure** button. You can use the builder to create simple data structures with basic types and validation rules, or the JSON editor for more complex data structures.

> **Info:** The data structure builder supports the following types:
>
> - String
> - Enumerated list
> - Integer
> - Decimal
> - Boolean
>
> For more complex data structures that require nesting or more advanced data types, use the [JSON editor](/docs/event-studio/data-structures/json-editor/). To understand all available JSON Schema validation options, see the [JSON Schema reference](/docs/api-reference/json-schema-reference/).

Populate the general information, such as Name, and a Description and Vendor. Vendor allows you to organize your data structures, for example, by teams. Snowplow will automatically generate the Tracking Url to be referenced in your tracking code.

![](/assets/images/data-structures-1-5b794bc4236fadba42000ce5d714b43f.png)

When creating a new [data structure](/docs/fundamentals/schemas/), you can add one or multiple properties. For each property, you can set a name, description, its type and a possible enumeration of allowed values (for type `string`). You can also set additional constraints to define if this property should be optional or mandatory, and if `null` values are allowed.

![](/assets/images/data-structures-2-b23aa8f4043bced0dfd2ac52eb0267d5.png)

Click **Save** on the Property dialog box to save your property changes.

Clicking **Save** on the data structure page will save your data structure as a draft. At this point, your data structure is not yet deployed to your development environment and cannot be used for event validation. When you're ready to test your data structure, you'll need to deploy it from the draft state to your development environment.

## Working with drafts

When you create a new data structure your changes are initially saved as a **draft**. Drafts allow you to:

- Make multiple changes without worrying about version numbers
- Experiment freely before committing to a final version
- Review and refine your data structure before deployment

**Important**: Draft data structures are not deployed to your development environment and will not be available for event validation. You must deploy your draft to the development environment when you're ready to test it.

This workflow gives you the flexibility to iterate on your data structure design without the overhead of managing version increments for every small change.

## Edit a data structure

To edit an existing data structure, navigate to **Data Structures** and locate the data structure you wish to edit. You can more easily find your data structure by:

- Using the search facility to search by name or vendor
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located, click on the name to view the data structure. You can then select from two options to edit the data structure: **Edit with builder** or **Edit with JSON editor**.

> **Note:** The **Edit with builder** option will be unavailable if the data structure you're viewing is not supported. More complex data structures must be edited with the **JSON Editor**.

On the edit page, under the General Information panel, you can update the data structure type or its description. To add a new property, cick the "Add Property" button. To edit or delete an existing property, click the three dots next to the property name to open the action menu, and then select the appropriate option.

![](/assets/images/edit-data-structure-a751c564691dd2386c85aae6a9702fb0.png)

When you modify the data structure, the builder will mark your changes in yellow, and automatically determine the new version of your data structure based on these modifications. You can reset the data structure and erase all changes at any moment by clicking the **Clear Changes** button found in the alert beneath the properties.

If you are satisfied with your changes, click **Save** and make sure to note the newly updated tracking URL.

![](/assets/images/data-structure-version-e2f76da4a575ceedae49575b78954288.png)

## Promote a data structure

When you're ready to use your data structure, you need to publish it from draft status to your development environment for testing.

Once you are happy with your changes in the development environment, you will want to migrate these changes to your production environment.

> **Note:** The action of migrating data structures to production is only available to Admin users.

Navigate to **Data structures** and locate the data structure you wish to migrate. You can more easily find your data structure by:

- Using the search facility to search by name or vendor
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located, either click on the name to view the data structure and then click the "Migrate to production" button, or click the three dots to bring up the action menu where you can select "Migrate to production".

![](/assets/images/image-8-887f63361cd518905ef3ba0aa4294c88.png)

At this stage you will see the publish dialog, and depending on how you versioned your edits you will see one of two messages:

If you are **publishing a new schema**, or **have incremented** the version whilst editing then you will see a confirmation of the action. Click **Migrate to Production** to migrate the data structure.

If you **have patched** the version whilst editing then you will see a warning that you must increment before publishing. Patching the version on Production is not a permitted action. [Increment the version number according to the changes you have made](/docs/event-studio/data-structures/versioning/) and click **Migrate to production** to migrate the latest version of your data structure to your production environment.

Your data structure will now be available in your production environment to send events against.

## Hide a data structure

Sometimes you will make errors when creating a data structure, or simply be creating new data structures as part of a quick experiment. On these occasions you may wish to hide the schema to clean up the listing in Console.

Navigate to **Data structures** and locate the data structure you wish to hide. You can more easily find your data structure by:

- Using the search facility to search by name
- Ordering the Name column alphabetically
- Filtering the listing by Type and / or Vendor

Once located either click on the name to view the data structure and then click the **Hide** button, or click the three dots to bring up the action menu where you can select **Hide data structure**.

Follow the dialog instructions to confirm the action.

> **Note:** Hiding a data structure will not remove it from the registry, it simply hides it from the console listing. This means:
>
> (1) events can still be sent against this structure (2) you cannot create a new structure of the same name

### Restore a hidden data structure

If you have hidden a data structure and wish to restore it, navigate to the bottom of the list of data structures and locate the 'View hidden data structures' link.

![](/assets/images/image-9-aaac318a1695105606bcf8e362e3314a.png)

This will take you to a list of hidden data structures, locate the one you wish to restore and click **Restore data structure** to show it in the main listing.

## Externally managed data structures

Data structures can be managed from an external repository using [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/).

When a data structure is managed this way it becomes locked in the UI, disabling all editing. You will see a banner explaining the situation and giving people with the 'publish to production' (default for admin users) capability the ability to unlock.

![](/assets/images/locked-ds-5318a0b425e253f84213ba78ba9743de.png)

> **Warning:** Having a single source of truth for a data structure is a good idea. If your source of truth is an external repository then unlocking and editing will cause conflicts.

---

# Create complex data structures with the JSON editor
> Define complex data structures with heavy nesting and advanced data types using the JSON Editor for full JSON Schema support.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/json-editor/

> **Info:** The JSON editor is ideal for more complex data structures that require nesting or more advanced data types. For simple data structures, use the [Data Structures Builder](/docs/event-studio/data-structures/).

## Creating a new data structure

Select whether you'd like to create an [Event](/docs/fundamentals/events/) or an [Entity](/docs/fundamentals/entities/). You can always change this selection at a later date.

![Choice between builder and JSON editor options](/assets/images/image-2-bad6698ed68cad8d2bd005ce7ff2a082.png)

You can now write the first version of your JSON schema for this data structure. Some template JSON is provided in the code window to start you off. For comprehensive guidance on all supported JSON Schema features and validation options, see the [JSON Schema reference](/docs/api-reference/json-schema-reference/).

![](/assets/images/json-template-406dba60e0fcee3f5c03ed6e579372d4.png)

Once you are done, click the **Validate** button and we'll validate that your schema is valid JSON markup. Assuming it passes validation, you can save your data structure as a draft.

See the [Working with drafts](/docs/event-studio/data-structures/#working-with-drafts) section for more information about the draft workflow.

Click **Save as draft** to save your data structure as a draft. As this is the first version of your data structure, it will be created as version `1-0-0` when you later deploy it to your development environment.

## Editing a data structure

Make the required edits to the JSON schema. You can use the 'Difference' toggle above the editor to see a 'diff' view against the latest Production version of your data structure.

In the example below we have changed the `maxLength` of `example_field_1`.

![](/assets/images/image-5-64bfbf5b5861f6d058d055bf896f5192.png)

Once you are happy with your changes, click **Validate** to ensure you have valid JSON markup. Then click **Publish to development environment** to save your changes to your development environment.

![](/assets/images/image-7-9423f8ff7c3adcfd4e7d36858e60a934.png)

The versioning dialog will appear, at this point you have three options:

- Increment a minor version to indicate a non-breaking change to the schema. In our example, this would increment the schema to from `1-0-1` to `1-0-2`.
- Increment a major version to indicate a breaking change to the schema. In our example, this would increment the schema from `1-0-1` to `2-0-0`.
- [Patch the current version](/docs/event-studio/data-structures/versioning/#patch-a-schema), this will overwrite the existing schema without increasing the version. In our example, this would leave the schema at 1-0-1.

For more information see [Versioning your data structures](/docs/event-studio/data-structures/versioning/).

Once you have selected the appropriate version, click **Deploy to development environment** and your data structure will be deployed to your development environment ready [for you to test](/docs/testing/).

You can identify data structures where the Development version is ahead of the Production version by the yellow background on the version number. In this example both `user` and `alert` have been edited on development.

***

---

# Version and amend data structures
> Evolve your tracking design safely with backwards-compatible data structure versioning using JSON schema version numbers to control warehouse loader behavior.
> Source: https://docs.snowplow.io/docs/event-studio/data-structures/versioning/

Every data structure is based on a [versioned schema](/docs/fundamentals/schemas/versioning/).

There are two kinds of schema changes:

- **Non-breaking** - a non-breaking change is backward compatible with historical data and increments the `patch` number i.e. `1-0-0` -> `1-0-1`, or the middle digit i.e. `1-0-0` -> `1-1-0`.
- **Breaking** - a breaking change is not backwards compatible with historical data and increments the `model` number i.e. `1-0-0` -> `2-0-0`.

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Warning:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

## Automatic versioning with the data structure builder

Versioning is automated when using the data structure builder to create or edit your custom data structures.

It will automatically select how to version up your data structure depending on the changes you have just made.

In this example, a new required property has been added to the data structure. This is a breaking change, so the builder will increment the first digit:

![](/assets/images/data-structures-2-1133e290be8b44d1ed1d0f74dcef1c3a.png)

In this example, an additional enum option has been added to `category`. This is a non-breaking change, so the builder is incrementing the middle digit:

![](/assets/images/data-structures-1-984215ae8aba92b496bfca0378c9e0be.png)

## Versioning with the JSON editor

When using the JSON editor, at the point of publishing a data structure you'll be asked to select which version you'd like to create.

![](/assets/images/json_editor_version_options-bf9596ad2af60a2639de1ee5107f87cd.png)

## Patch a schema

To [patch a schema](/docs/fundamentals/schemas/versioning/#patch-a-schema), i.e. apply changes to it without updating the version, select the **Patch** option when saving the schema.

Note that various pipeline components, most importantly Enrich (including Enrich embedded in Snowplow Mini and Snowplow Micro), cache schemas to improve performance. The default caching time is 10 minutes (it's controlled by the [Iglu Resolver configuration](/docs/api-reference/iglu/iglu-resolver/)). This means that the effect of patching a schema will not be immediate.

> **Note:** If you are using Snowplow Self-Hosted, to patch a schema, don't increment the schema version when [uploading it with `igluctl`](/docs/api-reference/iglu/manage-schemas/).
>
> You'll need to explicitly enable patching in the [Iglu Server configuration](/docs/api-reference/iglu/iglu-repositories/iglu-server/reference/) (`patchesAllowed`) at your own risk.

## Mark a schema as superseded

To [mark a schema as superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded), use the JSON editor and add a `$supersedes` field.

---

# Discover and manage events with the Event Catalog
> Browse, search, and manage all event specifications across tracking plans from a single location to improve event discoverability and governance.
> Source: https://docs.snowplow.io/docs/event-studio/event-catalog/

The Event Catalog provides a centralized location to discover and manage all event specifications across your [tracking plans](/docs/event-studio/tracking-plans/). Instead of navigating through individual tracking plans to find specific events, you can browse, search, and filter all event specifications from a single view.

When your organization has multiple tracking plans across different teams and domains, finding specific event specifications can become challenging. The Event Catalog addresses this by:

- **Centralizing discovery**: browse all event specifications in one place rather than searching through individual tracking plans
- **Improving governance**: maintain oversight of all event specifications and their status across your organization
- **Streamlining onboarding**: help new team members understand what events are available and how they're organized
- **Enabling cross-team collaboration**: see how different teams have defined similar events and share best practices

## Access the Event Catalog

Navigate to **Event Catalog** in the main navigation.

![Event Catalog overview](/assets/images/event-catalog-overview-0239dfa65e17f58839465c6c9b30533a.png)

## Browse event specifications

The Event Catalog provides a comprehensive list of all event specifications defined across your tracking plans. Each row displays:

| Column                   | Description                                                        |
| ------------------------ | ------------------------------------------------------------------ |
| Event specification name | The name and schema identifier of the event specification          |
| Entities                 | The [entities](/docs/fundamentals/entities/) attached to the event |
| Tracking plan            | The tracking plan containing the event specification               |
| Volume                   | The number of events collected                                     |
| Last seen                | When the event was last received                                   |
| Status                   | The status of the event specification                              |

### Filter and search

You can filter and search the list to find specific event specifications. Use the controls at the top of the list:

- **Search**: enter text to filter by event specification name
- **Status filter**: show all specifications or filter by Draft or Published status
- **Entity filter**: filter specifications by attached entities

## Create event specifications

From the Event Catalog, you can create new event specifications without first navigating to a tracking plan.

To create an event specification:

1. Click **Create event specification**
2. Enter a name that describes the event
3. Optionally add a description
4. Select the tracking plan this event specification belongs to
5. Select the source applications where this event is tracked
6. Click **Confirm**

![Create event specification dialog](/assets/images/create-event-specification-7ba523f45dd227ae96a8434ee0d0ee4d.png)

After creation, you can add event and entity data structures, triggers, and property instructions.

> **Tip:** Use descriptive names that reflect the business action being tracked. For example, "Add to cart" is clearer than "cart\_event\_1".

---

# Implement tracking in your applications
> Generate and implement tracking code from your event specifications using Snowtype, Console code snippets, or manual SDK integration.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/

Once you've defined your [tracking plans](/docs/event-studio/tracking-plans/) and [event specifications](/docs/event-studio/tracking-plans/event-specifications/), the next step is implementing the tracking code in your applications.

Snowplow provides several approaches to generate and implement tracking code:

- Ready-to-use code snippets in [Console](https://console.snowplowanalytics.com) (web only)
- [Snowtype](/docs/event-studio/implement-tracking/snowtype/) code generation tool
- Manual integration with [Snowplow trackers](/docs/sources/)

## Console code snippets

When viewing an [event specification](/docs/event-studio/tracking-plans/event-specifications/) in Console, the **Working with this event** section provides ready-to-use code snippets.

The snippets in the **Implementation** tab show the exact tracking calls needed for each event, including all required properties and entities.

> **Note:** Code snippets are available for the JavaScript tracker only, for event specifications with custom event data structures.

Here's an example snippet for the JavaScript tracker. It provides a `trackSelfDescribingEvent` call for the event specification, with the correct schema references and properties.

The example event specification has an event data structure named `article_click`, and one entity data structure, `article`. The snippet also includes an autogenerated `event_specification` entity. This helps with analysis as it's a direct link between the tracked event and the tracking plan.

To use your snippet, paste it into your application code, and provide the appropriate property values.

```javascript
window.snowplow("trackSelfDescribingEvent", {
    "event": {
        "schema": "iglu:com.example/article_click/jsonschema/1-0-1",
        "data": {
            "name": "", // string - Required - maxLength: 1000
            "location": "", // string - Nullable - maxLength: 1000
        }
    },
    "context": [
        // Entity: article (min: 0)
        {
            "schema": "iglu:com.example/article/jsonschema/3-0-0",
            "data": {
                "publish_date": "", // string - Nullable
                "content_id": "", // string - Nullable
            }
        },
        // System entity. Please do not edit it.
        {
            "schema": "iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-3",
            "data": {
                "id": "0a0ef8bb-314c-4973-8988-f192e8714d68",
                "name": "Article Click",
                "data_product_id": "28a6316a-47fd-473b-b5a1-00c555ba25e4",
                "data_product_name": "Article Performance",
                "data_product_domain": "Marketing"
            }
        }
    ]
});
```

Use the **Show Snowtype code** toggle to display the specific Snowtype function name to call for tracking implementation.

![Show snowtype code](/assets/images/show-snowtype-code-f0c0e475e699e14327af5e306eee2121.png)

---

# Client-side schema validation with Snowtype
> Enable real-time schema validation in the browser for JavaScript and TypeScript trackers to catch tracking errors before events are sent.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/client-side-validation/

> **Info:** This feature is available since version 0.2.8 of Snowtype for the [Browser Tracker](/docs/sources/web-trackers/quick-start-guide/?platform=browser) in both JavaScript and TypeScript.

## Schema validation right on your browser

Using Snowtype you can get notified, at runtime, about schema validation errors and fix them before slipping to production.

To opt-in to client-side validation you should include the `--validations` flag when you are generating your code.

```sh
npx @snowplow/snowtype@latest generate --validations
```

For validations to work, you will also need to install `ajv@8`, `ajv-formats@2` and `ajv-draft-04@1`.

```sh
# Example using npm
npm install ajv@8 ajv-formats@2 ajv-draft-04@1
```

This command will generate your code as expected but behind the scenes run all the validations required when an event is being sent from the generated code.

## Schema validation example

Below there is an example of how validations will show up in your environment.

Suppose we are tracking against a custom schema for button clicks:

```json
{
    type: 'object',
    description: 'Data structure for custom button clicks ',
    properties: {
        label: {
            type: 'string',
            description: 'The text on the button, or a user-provided override'
        },
        id: {
            type: 'string',
            description: 'The identifier of the button'
        },
    },
    /* Other attributes... */
}
```

When the respective method, from Snowtype, that handles tracking of this event is fired validation will happen at runtime for all schema attributes. Following we can see an example of how the schema validation will show up in the browser console when the event responsible for tracking against the custom button click schema fires.

![validation example](/assets/images/validation-0a7cf374318205444a962c9a69fbc40d.png)

As we can observe, the value passed as the `id` attribute is violating the schema rules. The erroneous value can be found under `errors[n].data` which in this case is the number `1`.

Currently the validation information will include attributes that can help point out the issue happening at the schema level, plus the stack trace revealing the caller of the function.

## Entity cardinality rules validation example

> **Info:** This feature is available since version 0.3.1 of Snowtype for the [Browser Tracker](/docs/sources/web-trackers/quick-start-guide/?platform=browser) in both TypeScript and JavaScript.

Cardinality rules allow you to specify the expected number of an entity taking part in an Event Specification. You would use this capability to ensure the correct number of entities are getting sent alongside your event. E.g.

- `Exactly 1`
- `At least 1`
- `Between 1 and 2`

By using Snowtype client-side validations you will get notified right in your browser that there is a violation of cardinality rules for an Event Specification. This can prove to be really important during development and/or testing.

Below there is an example of how entity cardinality rules validations will show up in your environment. In this example there is a `product` entity that is expected to have cardinality of `Exactly 1`.

![cardinality validation example](/assets/images/cardinality-validation-7387c2d3b0d509881ac5973741cce55a.png)

The code generated for the Event Specification can be used as such without violating the cardinality rules:

```ts
trackButtonClickSpec({
    label: "Product click",
    context: [createProduct({ name: "Product", price: 1, quantity: 1 })],
});
```

In the case that a rule is violated, for example in the case of adding more than one product contexts to the event, you will get notified with a validation warning on your browser as such:

```ts
trackButtonClickSpec({
    label: "Product click",
    context: [
        createProduct({ name: "Product", price: 1, quantity: 1 }),
        // This violates the cardinality rule of Exactly 1
        createProduct({ name: "Product 2", price: 1, quantity: 1 })
    ],
});
```

![cardinality validation browser example](/assets/images/cardinality-browser-6e2ea31e9b212020037906f5da74e21d.png)

The warning will include information about the `minCardinality` and `maxCardinality` expected alongside the `currentCardinality` which is the number of currently included contexts in the event. All these together with stack trace information you can use to trace back the violating function.

A similar warning will occur when there is a cardinality rule set for an entity, and this entity does not exist as context in the event:

![empty cardinality validation browser example](/assets/images/cardinality-empty-fed6ca67310fc3b06dd24ea53a735b26.png)

## Property rules validation example

> **Info:** This feature is available since version 0.10.0 of Snowtype for the [Browser Tracker](/docs/sources/web-trackers/quick-start-guide/?platform=browser) in both TypeScript and JavaScript.

Property rules are [specific instructions](/docs/event-studio/tracking-plans/) you can add in every schema that takes part in an Event Specification. This capability will allow you to adjust the expected values specifically to this event. E.g.

- The `category` attribute of the `product` entity is expected to take the values of "related" or "cross-sell" for this Event Specification

![property rules browser example](/assets/images/property-rules-browser-0895b2997955f456740a1bf4db235428.png)

The code generated for the Event Specification can be used as such without violating the property rules:

```ts
trackRelatedSpec({
    label: "Related product",
    context: [
        /* This is a method to create this specific product entity for the `Related` Event Specification */
        createProductRelated({
            /* Category can only be `cross-sell` or `related` based on the type generated */
            category: "cross-sell",
            name: "product",
            quantity: 1,
            price: 10,
        })
    ],
});
```

In the case that a rule is violated, for example adding an unintended `category` value, you will get notified with a validation warning on your browser as such:

```ts
trackRelatedSpec({
    label: "Related product",
    context: [
        createProductRelated({
            /* `cross-sells` is not a valid category based on the set instructions. */
            category: "cross-sells",
            name: "product",
            quantity: 1,
            price: 10,
        })
    ],
});
```

![property rules enum error](/assets/images/property-rules-enum-error-1efab496e8e3d763cad18c269380dc1b.png)

## Custom Violation Handler

By default, when a JSON Schema or Tracking Plan rule is violated, Snowtype will print a warning using `console.log`, displaying this information in the browser's developer tool panel. While this is useful for debugging events in the browser, it can be adjusted for different environments to better suit your needs. For example:

- When unit testing with a library such as Jest, you might prefer each violation to throw a new `Error` so that relevant tests automatically fail.
- In staging or production environments, you might want to report the violation to a third-party error monitoring solution such as Sentry.

To accommodate custom violation handling use cases, Snowtype provides an option to set the `violationsHandler`. Using the `snowtype.setOptions` API, you can configure the `violationsHandler` to be called whenever a violation is detected.

```ts
import { snowtype } from "{{outpath}}/snowplow";

function myViolationsHandler(error){
    // Custom violation handling logic
}

snowtype.setOptions({ violationsHandler: mockViolationsHandler });
```

The `error` attribute is typed as follows:

```ts
type ErrorType = {
  /* Specific error code number e.g. 100, 200, 201 ... */
  code: number;
  /* Error message */
  message: string;
  /* Description of the violation */
  description: string;
  /* Violations occurred */
  errors: (ErrorObject | Record<string, unknown>)[];
};
```

> **Info:** When Snowtype detects the `NODE_ENV` environment variable being set to `test`, as is done by many testing libraries, it will automatically default to throwing an `Error` when a violation is detected.

## Caveats

### Bundle size consideration

Since the validation capability depends on a set of additional libraries which can increase the application bundle size, it is advised that this feature is used in development and test environments. Moving forwards there is consideration for creating a validation capability with minimal overhead both at runtime performance and bundle size.

### Divergence with pipeline validation

Due to the differences between environments, there could be a few cases where validation result might diverge between the client and the pipeline. These differences can be found in cases where regular expressions are included in the schema. For JSON Schema, these kinds of formats are mostly included in the [pattern](https://json-schema.org/understanding-json-schema/reference/string#regexp) attribute.

For that reason, when Snowtype detects a `pattern` key in string type attributes will warn accordingly during generation.

---

# Snowtype CLI command reference
> Summary of Snowtype CLI commands and options for code generation, with details on usage scenarios and configuration.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/commands/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

> **Info:** This page only summarizes the CLI commands and the options for each command. For details on which scenarios they can be used, you can go to the [Working with the CLI page](/docs/event-studio/implement-tracking/snowtype/using-the-cli/).

## Usage

`snowtype [COMMAND] [OPTIONS] [CONTEXT-SPECIFIC-OPTIONS]`

## Available CLI commands

### `snowtype init`

Initialize the setup of Snowtype code generation in a project. Creates the configuration file.

**Options**

- `-i, --organizationId` Organization ID.
- `-t, --tracker` Tracker to use. [See available](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#available-trackerslanguages)
- `-l, --language` Language to use. [See available](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#available-trackerslanguages)
- `-o, --outpath` Output path.

### `snowtype generate`

Generates tracking code based on configuration on the configuration file. Can generate/modify the `.snowtype-lock.json` file.

**Options**

- `-c, --config` Config file path.
- `--instructions` Generate event specification instructions.
- `--no-instructions` Generate without instructions.
- `--validations` Add runtime validation on events. _Currently available for the Browser tracker_.
- `--no-validations` Do not add runtime validation on events.
- `--disallowDevSchemas` Disallow generation of code using schemas deployed on DEV environment. _Sending events using schemas deployed on DEV, will result in failed events in production pipelines._ (default: false)
- `--deprecateOnlyOnProdAvailableUpdates` Show deprecation warnings only when there are PROD available schema updates. (default: false)

### `snowtype update`

Checks for latest version updates in Data Structures and Event Specifications.

**Options**

- `-c, --config` Config file path.
- `-y, --yes` Updates all to latest version without prompting. (default: false)
- `-m, --maximumBump` The maximum SchemaVer update to show an available update notification for. Possible values are 'patch', 'minor', 'major' and will work as expected regular SemVer bumps. (default: 'major')

### `snowtype patch`

Adds new Data Structures and Event Specifications in the `snowtype.config.json` file without needing to modify the file by hand.

**Options**

- `-c, --config` Config file path.
- `-e, --eventSpecificationIds` Event Specification ID/s.
- `-p, --dataProductIds` Tracking Plan ID/s.
- `-d, --dataStructures` Data structure schema URI/s.
- `-i, --igluCentralSchemas` Iglu central schema URI/s.
- `-r, --repositories` Local Data Structure repositories generated from the [snowplow-cli](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/).

### `snowtype help`

Shows a helpful message and brief instructions for the Snowtype CLI usage.

### Global options

- `-h, --help` Shows helpful instructions for the command.
- `-V, --version` Output the package version number.
- `-k, --apiKey` Provide the Snowplow Console API key as a CLI option.
- `-v, --verbose` Enable verbose logging.

---

# Generate tracking code with Snowtype
> Automatically generate type-safe tracking code from data structures and event specifications with compile-time validation, reducing implementation time and maintenance overhead.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/

**Snowtype** is a code generation tool that automates the creation of type-safe tracking code for Snowplow SDKs. Snowtype connects directly to your data structures and event specifications. This eliminates manual instrumentation work and ensures that your tracking code is compliant with the schemas and produces high quality data.

Snowtype streamlines the development workflow by providing several key advantages:

- **Type safety enforcement:** Generates strongly-typed code that validates events and entities at compile time, preventing schema violations before data reaches your pipeline.
- **Automated code generation:** Converts event specifications into production-ready SDK code, reducing implementation time from weeks to days.
- **Integrated documentation:** Syncs inline code documentation with your data structures and products, maintaining consistency between design and implementation.
- **Development workflow integration:** Fits seamlessly into CI/CD processes, enabling GitOps-style tracking plan management and automated updates when schemas evolve.
- **Reduced maintenance overhead:** Automatically updates tracking code when data structures change, eliminating the need for manual synchronization across multiple codebases.

## Supported trackers

| **Tracker**                      | **Language/s**         |
| -------------------------------- | ---------------------- |
| `@snowplow/browser-tracker`      | javascript, typescript |
| `@snowplow/node-tracker`         | javascript, typescript |
| `@snowplow/react-native-tracker` | typescript             |
| `@snowplow/javascript-tracker`   | javascript             |
| `snowplow-golang-tracker`        | go                     |
| `snowplow-ios-tracker`           | swift                  |
| `snowplow-android-tracker`       | kotlin                 |
| `snowplow-flutter-tracker`       | dart                   |
| `snowplow-java-tracker`          | java                   |

## Prerequsites

To use Snowtype, you must have [Node.js](https://nodejs.org/en/) (>=@18) installed.

## Installation

Navigate to your project and install Snowtype using your favorite package manager:

**npm:**

```bash
npm install --save-dev @snowplow/snowtype@latest
```

**Yarn:**

```bash
yarn add --dev @snowplow/snowtype@latest
```

**pnpm:**

```bash
pnpm add --save-dev @snowplow/snowtype@latest
```

***

## Executing commands

Installing Snowtype will also create a local executable `snowtype` which you can use with `npx`, `yarn` or `pnpm` directly when on your project's directory.

**npm:**

```bash
npx @snowplow/snowtype@latest init
# Same as
npx snowtype init
```

**Yarn:**

```bash
yarn @snowplow/snowtype@latest init
# Same as
yarn snowtype init
```

**pnpm:**

```bash
pnpm @snowplow/snowtype@latest init
# Same as
pnpm snowtype init
```

***

_We will show example commands using `npm/npx` but it should work the same with any other package manager._

---

# Snowtype configuration options
> Configure Snowtype code generation with options for output paths, tracker selection, language settings, and custom templates.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/snowtype-config/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

The Snowtype CLI configuration can be saved in a `.json`, `.js`, or `.ts` file after initialization. For example: `snowtype.config.json`, `snowtype.config.js`, or `snowtype.config.ts`. **We highly recommend you keep this file in the root of your project folder.**

## Attributes in your configuration file

### `igluCentralSchemas`

The schema tracking URLs for schemas available in [Iglu Central](https://iglucentral.com/).

### `repositories`

Local Data Structure repositories generated from the [snowplow-cli](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/).

### `dataStructures`

The schema tracking URLs for Data Structures published in the Console.

### `eventSpecificationIds`

The Event Specification IDs you wish to generate tracking code for. The Event Specification ID is a UUID that can be retrieved as the final part of the URL when visiting an event specification main page.

### `dataProductIds`

The Tracking Plan IDs you wish to generate tracking code for. By providing the Tracking Plan Id, Snowtype will fetch all the event specifications for the Tracking Plan and generate code for all of them. The Tracking Plan ID is a UUID that can be retrieved as the final part of the URL when visiting a tracking plan main page.

### `organizationId`

The Organization ID for your Snowplow account. The Organization ID is a UUID that can be retrieved from the URL immediately following the .com when visiting console.

### `tracker`

The target tracker to generate the required code for. [See list of available trackers](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#available-trackerslanguages).

### `language`

The target language to generate the required code for. [See list of available languages](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#available-trackerslanguages).

### `outpath`

The outpath relative to the current working directory when running the script.

### `options`

Options related to Snowtype behavior and are described by the following TypeScript type:

```ts
options?: {
  /* Command related options. */
  commands: {
    generate?: {
      /* Generate implementation instructions. */
      instructions?: boolean;
      /* Add runtime validations. */
      validations?: boolean;
      /* Disallow generation of code using schemas only deployed on DEV environment. */
      disallowDevSchemas?: boolean;
      /* Show deprecation warnings only when there are PROD available schema updates. */
      deprecateOnlyOnProdAvailableUpdates?: boolean;
    }
    update?: {
      /* Update your configuration file automatically and regenerate the code of the latest available update. */
      regenerateOnUpdate?: boolean;
      /* The maximum SchemaVer update to show an available update notification for. */
      maximumBump?: "major" | "minor" | "patch";
      /* The `update` command will only display updates for Data Structures that have been deployed to production environment. */
      showOnlyProdUpdates?: boolean;
    }
    patch?: {
      /* Automatically regenerate the code after a successful patch operation. */
      regenerateOnPatch?: boolean;
    }
  }
}
```

### `namespace`

> **Info:** This option only applies when generating Swift code.

The namespace for the generated code. All classes generated will be included in this namespace, which can be used to avoid naming conflicts.

For example, setting `namespace` to `Snowtype` will result in classes being accessed with the `Snowtype` prefix:

```swift
let data = Snowtype.AccountConfirmed(companyCountry: "", companyName:
"", ...)
```

_Keep in mind that CLI flags take precedence over configuration file options._

## Example configuration file

**JSON:**

```json
{
  "igluCentralSchemas": ["iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"],
  "repositories": ["../data-structures"],
  "dataStructures": ["iglu:com.myorg/custom_web_page/jsonschema/1-1-0"],
  "eventSpecificationIds": [
    "a123456b-c222-11d1-e123-1f123456789g"
  ],
  "dataProductIds": [
    "a123456b-c222-11d1-e123-1f12345678dp"
  ],
  "organizationId": "a654321b-c111-33d3-e321-1f123456789g",
  "tracker": "@snowplow/browser-tracker",
  "language": "typescript",
  "outpath": "./src/snowtype"
}
```

**JavaScript:**

```javascript
const config = {
  "igluCentralSchemas": ["iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"],
  "repositories": ["../data-structures"],
  "dataStructures": ["iglu:com.myorg/custom_web_page/jsonschema/1-1-0"],
  "eventSpecificationIds": [
    "a123456b-c222-11d1-e123-1f123456789g"
  ],
  "dataProductIds": [
    "a123456b-c222-11d1-e123-1f12345678dp"
  ],
  "organizationId": "a654321b-c111-33d3-e321-1f123456789g",
  "tracker": "@snowplow/browser-tracker",
  "language": "typescript",
  "outpath": "./src/snowtype"
}

module.exports = config;
```

**TypeScript:**

```typescript
type SnowtypeConfig = {
  tracker:
    | "@snowplow/browser-tracker"
    | "@snowplow/javascript-tracker"
    | "snowplow-android-tracker"
    | "snowplow-ios-tracker"
    | "@snowplow/node-tracker"
    | "snowplow-golang-tracker"
    | "@snowplow/react-native-tracker"
    | "snowplow-flutter-tracker";
  language: "typescript" | "javascript" | "kotlin" | "swift" | "go" | "dart";
  outpath: string;
  organizationId?: string;
  igluCentralSchemas?: string[];
  repositories?: string[];
  dataStructures?: string[];
  eventSpecificationIds?: string[];
  dataProductIds?: string[];
  options?: {
    commands: {
      generate?: {
        instructions?: boolean;
        validations?: boolean;
        disallowDevSchemas?: boolean;
        deprecateOnlyOnProdAvailableUpdates?: boolean;
      }
      update?: {
        regenerateOnUpdate?: boolean;
        maximumBump?: "major" | "minor" | "patch";
        showOnlyProdUpdates?: boolean;
      }
      patch?: {
        regenerateOnPatch?: boolean
      }
    }
  }
};

const config: SnowtypeConfig = {
  "igluCentralSchemas": ["iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0"],
  "repositories": ["../data-structures"],
  "dataStructures": ["iglu:com.myorg/custom_web_page/jsonschema/1-1-0"],
  "eventSpecificationIds": [
    "a123456b-c222-11d1-e123-1f123456789g"
  ],
  "dataProductIds": [
    "a123456b-c222-11d1-e123-1f12345678dp"
  ],
  "organizationId": "a654321b-c111-33d3-e321-1f123456789g",
  "tracker": "@snowplow/browser-tracker",
  "language": "typescript",
  "outpath": "./src/snowtype"
};

export default config;
```

***

---

# Work with the Snowtype CLI
> Use the Snowtype CLI to initialize projects, generate tracking code, and configure code generation for Snowplow tracking SDKs.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/using-the-cli/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

The Snowtype CLI is a tool which aims to speed up tracking implementations, provide type safety and inline documentation for developers and ultimately reduce the number of erroneous events. By integrating this tool in the development workflow we introduce a way to connect the additions and updates done in a Snowplow implementation with the corresponding tracking code of the project.

## Authenticating with the Console

A Console API key is required for the Snowtype CLI to authenticate with your account. You can find your own or create one in the Console [API key management](https://console.snowplowanalytics.com/credentials).

> **Note:** Both API key and API key ID variables are required for versions > `0.9.0`.

The ways for the CLI to read the credentials are either through the global `-k, --apiKey` and `-s, apiKeyId` options or the `SNOWPLOW_CONSOLE_API_KEY` and `SNOWPLOW_CONSOLE_API_KEY_ID` environment variables. Additionally, the Snowtype CLI automatically reads from a `.env` file at the root of your project.

**.env file:**

```bash
SNOWPLOW_CONSOLE_API_KEY=MY-API-KEY
SNOWPLOW_CONSOLE_API_KEY_ID=MY-API-KEY-ID
```

**Shell variable:**

```bash
# The required command will depend on your shell
export SNOWPLOW_CONSOLE_API_KEY=MY-API-KEY
export SNOWPLOW_CONSOLE_API_KEY_ID=MY-API-KEY-ID
```

**CLI parameter:**

```bash
npx @snowplow/snowtype@latest generate --apiKey MY-API-KEY --apiKeyId MY-API-KEY-ID
```

***

**Recommended:** We recommend that you use the `SNOWPLOW_CONSOLE_API_KEY` and `SNOWPLOW_CONSOLE_API_KEY_ID` environment variables.

## Initializing Snowtype for your project

For the Snowtype CLI to work properly, it requires a [configuration file](/docs/event-studio/implement-tracking/snowtype/snowtype-config/) to be initialized and present on your project's root folder. This file will be automatically generated using the `snowtype init` command, after adding the required input.

```bash
# Start prompting for configuration inputs
npx @snowplow/snowtype@latest init
```

The input required for the initialization to work, it the following:

- The organization ID from Snowplow Console.
- The [tracker](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#available-trackerslanguages) you wish to generate code for.
- _If applicable,_ the language for that tracker.
- The output path you wish the CLI to generate the code to.

These will all be prompted to you by default, but if needed you can call the `snowtype init` command with any or all the attributes passed as [optional flags](/docs/event-studio/implement-tracking/snowtype/commands/#snowtype-init) so prompting is not required.

## Generate tracking code

The CLI will generate tracking code using a valid Snowtype configuration file with the `snowtype generate command`. The code that will be generated, depending on the language, will have all the required types used in schemas and Event Specifications together with methods/classes that allows for tracking these.

```bash
# Code will be generated to the outpath configuration
npx @snowplow/snowtype@latest generate
```

The code generated by the CLI is not minified and contains inline documentation for methods, classes and types. If needed you can modify it in any way it suits your project.

### Contents

The contents of a generated file from the Snowtype CLI will be:

- Types/Interfaces/Classes for each schema that relates to the Data Structures, Iglu Central Schemas and Event Specifications selected.
  - For each Event Specification [instruction set](/docs/event-studio/tracking-plans/), a type for the adjusted schema is generated as well. The type/class will contain the Event Specification name as suffix to avoid conflicts.

- For each schema:

  - A method/class to instantiate the structure as a Self Describing JSON. _This is particularly useful to add entities as extra context on events._
  - A method that sends a Self Describing Event with the schema as the main event entity.

- For each Event Specification, a method/class to track the event specification with the set event and context entity schemas.

> **Warning:** The Snowtype CLI does not automatically install the required Snowplow tracking libraries. For now it generates code that use the tracking libraries which are expected to be installed on the project.

### Available Trackers/Languages

Following is the set of available trackers and languages the Snowtype CLI currently can work with. This list is also the source of truth for valid keys in the `tracker` and `language` attributes of the Snowtype configuration file.

| **Tracker**                      | **Language/s**         |
| -------------------------------- | ---------------------- |
| `@snowplow/browser-tracker`      | javascript, typescript |
| `@snowplow/node-tracker`         | javascript, typescript |
| `@snowplow/react-native-tracker` | typescript             |
| `@snowplow/javascript-tracker`   | javascript             |
| `snowplow-golang-tracker`        | go                     |
| `snowplow-ios-tracker`           | swift                  |
| `snowplow-android-tracker`       | kotlin                 |
| `snowplow-flutter-tracker`       | dart                   |
| `snowplow-java-tracker`          | java                   |

### Example Usage

Below we show example usage of the generated code. For demonstration, we assume the code was generated for the [web\_page](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0) and [product](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.ecommerce/product/jsonschema/1-0-0) schemas.

**@snowplow/browser-tracker TypeScript:**

```tsx
import {
  trackWebPage,
  createProduct,
  WebPage,
  Product,
  createWebPage,
} from "./{outpath}/snowplow";
/* Track a WebPage event */
trackWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
/* Track a WebPage event with a Product context entity */
const product = createProduct({
  id: "Product id",
  name: "Snowplow product",
  currency: "EUR",
  price: 10,
  category: "Snowplow/Shoes",
});
trackWebPage({
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product],
});
/* You can enforce specific context entities on any `track` function using type arguments */
const webPage = createWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
trackWebPage<Product | WebPage>({
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product, webPage],
});
```

**@snowplow/node-tracker TypeScript:**

```tsx
import {
  trackWebPage,
  createProduct,
  WebPage,
  Product,
  createWebPage,
} from "./{outpath}/snowplow";

/* `t` is the tracker instance created by the `tracker` function of the @snowplow/node-tracker package. */

/* Track a WebPage event */
trackWebPage(t, { id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
/* Track a WebPage event with a Product context entity */
const product = createProduct({
  id: "Product id",
  name: "Snowplow product",
  currency: "EUR",
  price: 10,
  category: "Snowplow/Shoes",
});
trackWebPage(t, {
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product],
});
/* You can enforce specific context entities on any `track` function using type arguments */
const webPage = createWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
trackWebPage<Product | WebPage>(t, {
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product, webPage],
});
```

**snowplow-android-tracker:**

```kotlin
import {{ specified package }}.Product
import {{ specified package }}.WebPage

/* Track a WebPage event */
tracker.track(WebPage(id = "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent())

/* Track a WebPage event with a Product context entity */
val product = Product(
    id = "Product id",
    name = "Snowplow product",
    currency = "EUR",
    price = 10.0,
    category = "Snowplow/Shoes",
)
val event = WebPage(id = "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent()
event.entities.add(product.toEntity())
tracker.track(event)
```

**snowplow-ios-tracker:**

```swift
import SnowplowTracker

/* Track a WebPage event */
_ = tracker.track(WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent())

/* Track a WebPage event with a Product context entity */
let product = Product(
    category: "Snowplow/Shoes",
    currency: "EUR",
    id: "Product id",
    name: "Snowplow product",
    price: 10
)
let event = WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43").toEvent()
event.entities.append(product.toEntity())
_ = tracker.track(event)
```

**snowplow-golang-tracker:**

```go
// Track a WebPage event
TrackWebPage(tracker, WebPage{ID: "212a9b63-1af7-4e96-9f35-e2fca110ff43"})

// Track a WebPage event with a Product context entity
productName := "Snowplow product"
product := Product{
	ID:       "Product_id",
	Currency: "EUR",
	Price:    10,
	Category: "Snowplow/Shoes",
	Name:     &productName,
}

TrackWebPage(
	tracker,
	WebPage{ID: "212a9b63-1af7-4e96-9f35-e2fca110ff43"},
	WithContexts(product),
)
```

**@snowplow/react-native-tracker TypeScript:**

```tsx
import {
  trackWebPage,
  createProduct,
  WebPage,
  Product,
  createWebPage,
} from "./{outpath}/snowplow";

/* `t` is the tracker instance created by the `createTracker` function of the @snowplow/react-native-tracker package. */

/* Track a WebPage event */
trackWebPage(t, { id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
/* Track a WebPage event with a Product context entity */
const product = createProduct({
  id: "Product id",
  name: "Snowplow product",
  currency: "EUR",
  price: 10,
  category: "Snowplow/Shoes",
});
trackWebPage(t, {
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product],
});
/* You can enforce specific context entities on any `track` function using type arguments */
const webPage = createWebPage({ id: "212a9b63-1af7-4e96-9f35-e2fca110ff43" });
trackWebPage<Product | WebPage>(t, {
  id: "212a9b63-1af7-4e96-9f35-e2fca110ff43",
  context: [product, webPage],
});
```

**snowplow-flutter-tracker:**

```dart
import './{outpath}/snowplow.dart';

/* Track a WebPage event */
await tracker.track(const WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43"));

/* Track a WebPage event with a Product context entity */
const product = Product(
  category: "Snowplow/Shoes",
  currency: "EUR",
  id: "Product id",
  price: 10.0,
  name: "Snowplow product"
);
const event = WebPage(id: "212a9b63-1af7-4e96-9f35-e2fca110ff43");
await tracker.track(event, contexts: [product]);
```

**snowplow-java-tracker:**

```java
package test;

import com.snowplowanalytics.snowplow.tracker.*;
import com.snowplowanalytics.snowplow.tracker.Tracker;
import com.snowplowanalytics.snowplow.snowtype.*;
import com.snowplowanalytics.snowplow.tracker.events.SelfDescribing;

import java.util.Collections;


public class SnowtypeTest {
    public static void main(String[] args) {
        Tracker tracker = Snowplow.createTracker("asdf", "asdf", "asdf");

        /* Track a WebPage event */
        tracker.track(SelfDescribing.builder().eventData(new WebPage.Builder().setId("212a9b63-1af7-4e96-9f35-e2fca110ff43").build().toSelfDescribingJson()).build());

        /* Track a WebPage event with a Product context entity */
        Product product = new Product.Builder().setId("Product id").setName("Snowplow product").setCurrency("EUR").setPrice(10.0).setCategory("Snowplow/Shoes").build();
        WebPage webPage = new WebPage.Builder().setId("212a9b63-1af7-4e96-9f35-e2fca110ff43").build();
        SelfDescribing event = SelfDescribing.builder().eventData(product.toSelfDescribingJson()).customContext(Collections.singletonList(webPage.toSelfDescribingJson())).build();
        tracker.track(event);
    }
}
```

***

### Tracking Plans

To generate code for the whole set of Event Specifications of a Tracking Plan, either manually or through the `snowtype patch` command, you will need the ID of the Tracking Plan. You can do this either by clicking on the `Implement tracking` button on the Tracking Plans main page to get the command directly:

![tracking plan track](/assets/images/dp-track-544c276d1535553cf5f930f697af85ad.png)

Or retrieve the ID from the URL bar and then add it on the `dataProductIds` array:

![tracking plan id](/assets/images/dp-id-6c8bd5e400425daa6c524a48efdc129e.png)

### Event Specifications

To add an Event Specification to the code generation, either manually or through the `snowtype patch` command, you would need the ID of the Event Specification. You can find the Event Specification ID in the main page of the Event Specification as shown below:

![event specification id](/assets/images/es-id-0a3367afcd71c89c002445c58795d483.png)

Then you should add this ID to your configuration file `eventSpecificationIds` array.

### Data Structures

To add a Data Structure to the code generation, either manually or through the `snowtype patch` command, you would need the Data Structure `Schema tracking URL`. You can find the Data Structure tracking URL on the Data Structure page in the Console, under the **Overview** tab as shown below:

![data structure url](/assets/images/ds-url-c93e327627928eba0a66750da09b833a.png)

Then you should add this Data Structure tracking URL to your configuration file `dataStructures` array.

### Iglu Central Schemas

To add a Data Structure to the code generation, either manually or through the `snowtype patch` command, you would need the `Schema tracking URL`. You can find the Schema tracking URL on [Iglu Central](http://iglucentral.com/) by searching for the schema and under the **General Information** tab you can find the URL as shown below:

![iglu central tracking url](/assets/images/iglu-url-99a7c2aae4d2cd5f818c9dfa1605804a.png)

Then you should add this Schema tracking URL to your configuration file `igluCentralSchemas` array.

### Local Data Structure Repositories

To add a local Data Structure repository to the code generation, either manually or through the `snowtype patch` command, you would only need the path/s to the repository/ies you have generated schemas using the [snowplow-cli](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/).

Then you should add the path/s to your configuration file `repositories` array.

## Generating event specification instructions

When generating code for event specifications, you have the option of delivering the implementation instructions and triggers for each specification right on the developer's environment.

By using the `--instructions` option on the `snowtype generate` command, you can generate a markdown file with all the required information about tracking an event specification.

This includes:

- Trigger description.
- Implementation rules.
- Images uploaded on your Event Specification triggers.
- App identifiers and URLs this event should be triggered on.
- Direct links to the code for this Event Specification.

## Keeping up with latest updates

It is important that the tracking code is up-to-date with latest versions of Data Structures and Event Specifications we are tracking on a project. The Snowtype CLI gives the engineers the ability to check if there are available updates for Data Structures and Event Specifications that are used in the project.

This works with the `snowtype update` command.

```bash
npx @snowplow/snowtype@latest update
```

The above command will output a _diff_ showing the available version updates, similar to what you can see below:

![patch command version diff](/assets/images/patch-diff-bf39da50d673e493a76a07d201ed72ef.png)

In that case, you can select to update to latest versions and regenerate the tracking code. To automatically update and regenerate the tracking code reflecting the latest updates, you can use the `--yes` flag.

### Adjust the level of update notifications

For possible Data Structure updates, you can set the maximum level of update you want to be notified about using the `--maximumBump` flag. This value provides the maximum bump to be notified about and update to if available. This value defaults to 'major' meaning the update command will notify you for all updates up to and including major type of updates to the schema model.

An example showcasing the flag's behavior:

```js
// Data Structure version added to the snowtype config is 1-0-0.
{
  // Other options...
  dataStructures: ["iglu:com.acme_company/page_unload/jsonschema/1-0-0"]
}
```

This Data Structure has other deployed versions such as `1-0-1`, `1-1-0` and `2-0-0`. The `update` command will show available updates as follows:

```bash
npx @snowplow/snowtype@latest update --maximumBump=major
# Will prompt an update to 2-0-0 or any other available update.

npx @snowplow/snowtype@latest update --maximumBump=minor
# Will prompt an update to 1-1-0 or 1-0-1 if the only available update.

npx @snowplow/snowtype@latest update --maximumBump=patch
# Will prompt an update to 1-0-1 if available or any other patch type update.
```

## Disallow generating code using schemas not deployed on production environment.

While developing or testing, it might be useful to use [Snowplow Micro](/docs/testing/snowplow-micro/) to validate against your new schemas in your development environment. In this and any other case you are developing a schema and eventually publishing the tracking to production, you need to make sure all the schemas you are using are deployed to the production environment for the pipeline to use. Failing to do that will result in failed events.

Snowtype by default will print a warning when code is generated using schemas only published to development environment. To make sure that there are no schemas not yet deployed to production, you can use the `--disallowDevSchemas` flag or option when using the `generate` command. Using this flag will make sure each generation attempt will fail, indicating the schemas that are not yet deployed to the production environment.

---

# Use Snowtype with Google Tag Manager
> Generate tracking code specifically formatted for Google Tag Manager custom JavaScript with Snowtype's GTM target for easier tag implementation.
> Source: https://docs.snowplow.io/docs/event-studio/implement-tracking/snowtype/working-with-gtm/

> **Info:** This feature is available from version 0.5.0.

To make working with Google Tag Manager and event tracking easier, we created a specific target for Snowtype fitting the way Google Tag Manager handles custom JavaScript code.

A few extra benefits:

- Simple initialization and maintenance.
- Snowtype generated functions are available in `window.__snowtype` for all tags to use.
- Fully typed code documentation using JSDoc.

## Getting Started without a code repository

**Already using Snowtype?**

To generate code for usage in Google Tag Manager, you should use the option `Google Tag Manager` as the tracker option in your [init](/docs/event-studio/implement-tracking/snowtype/commands/#snowtype-init) flow or replace your `tracker` and `language` attributes with the following values:

```json
{
    // Rest of the attributes...
    "tracker": "google-tag-manager",
    "language": "javascript-gtm"
}
```

What we are going to showcase here is how you can set up a separate project that uses Snowtype to generate code for your Google Tag Manager tracking needs. _You can also use version control to better understand any changes you make across time._

The only requirement is to first have [Node.js](https://nodejs.org/en/download/package-manager) installed on your computer. After that, open your terminal and run the following commands:

```bash
# Navigate to a directory that you wish to create your project in.
cd ./directory/to/setup/the/project
# Create a folder for the project. Here you can replace
# 'container-id' with the GTM container the code will be used in.
mkdir snowtype-gtm-container-id
# Change directory to the newly created folder.
cd snowtype-gtm-container-id
# This will create a few needed files for the project.
npm init -y
# This will install the latest version of Snowtype.
npm install @snowplow/snowtype@latest
# This will start the Snowtype init flow,
# in which you should select 'Google Tag Manager' when
# prompted to select a tracker.
npx @snowplow/snowtype@latest init
```

After you have completed the `init` flow and have added your desired configuration, you can go ahead and generate the code you need to use:

```bash
npx @snowplow/snowtype@latest generate
```

After that you can find the code in the specified `outpath` attribute of your configuration file.

## Using in Google Tag Manager

The code that Snowtype will generate is included in a `snowplow.js` file and, as stated in the file as well, is the code to be **copied and pasted** into a Google Tag Manager [Custom JavaScript variable](https://support.google.com/tagmanager/answer/7683362?hl=en#custom_javascript).

Below you can see the steps you need to create the variable using the contents generated by Snowtype: ![](/assets/images/gtm-var-a0b2e5014dc5e14da1288ff2172b2037.gif)

> **Warning:** If the generated code is too large for Google Tag Manager, you will receive a warning from Snowtype. You can choose to use the `.minified.js` version created by Snowtype to reduce the file size. Alternatively, you can split the code into multiple variables and include them in the order they are generated, or place them directly into a Custom HTML tag.

### Naming and calling the Custom JavaScript

After selecting a name for the Custom JavaScript variable, you would need to include it in a [Custom HTML](https://support.google.com/tagmanager/answer/6107167?hl=en#CustomHTML) tag so that it is executed. Depending on your Google Tag Manager setup, there are a couple, or more, places the variable can be used. An example is a type of Custom HTML tag that runs during page initialization:

```html
<script>
// ... Rest of the initialization code you might have
{{NAME_OF_THE_CREATED_VARIABLE}}
</script>
```

### Using the Snowtype generated code

From there on, after the function has been executed you can access all the functions generated would be available through the `window.__snowtype` object.

For example:

```js
// Example Data Structure
window.__snowtype.trackExample({ ... });

// Or for an Example Event Specification
window.__snowtype.trackExampleSpec({ ... });
```

Keep in mind that at the bottom of the generated file, there are all the JSDoc type definitions required for you to use the functions correctly.

---

# Event Studio
> Design and implement behavioral data tracking with schema management, governance, code generation, and tracking plans in Snowplow Console.
> Source: https://docs.snowplow.io/docs/event-studio/

Event Studio is a comprehensive set of tools for designing and implementing behavioral data event tracking. It provides:

- **Schema management**: define and version data structures for events and entities
- **Ownership and governance**: assign ownership and establish data contracts
- **Observability**: monitor data quality and tracking implementation
- **Code generation**: automatically generate tracking code from your designs, using [Snowtype](/docs/event-studio/implement-tracking/snowtype/)
- **Tracking plans**: document and manage your tracking implementation

The Event Studio UI is included in [Snowplow Console](https://console.snowplowanalytics.com).

These tools help organizations move from ad-hoc tracking implementations to a structured, governed, collaborative approach.

> **Tip:** New to tracking design? Start with our [best practice](/docs/fundamentals/tracking-design-best-practice/) guide to learn how to approach designing your tracking implementation.

## Key concepts

To use Event Studio effectively, you should understand these core concepts:

- **[Events](/docs/fundamentals/events/)**: actions that occur in your systems
- **[Entities](/docs/fundamentals/entities/)**: the objects and context associated with events
- **[Event specifications](/docs/event-studio/tracking-plans/event-specifications/)**: documentation of business events you're tracking
- **[Tracking plans](/docs/event-studio/tracking-plans/)**: logical groupings of related business events with defined ownership

Each tracking plan is associated with one or more [source applications](/docs/event-studio/source-applications/). The events and entities are defined by their [data structures](/docs/event-studio/data-structures/).

This diagram illustrates how these concepts relate to each other within Event Studio:

![Tracking plan overview showing the relationship between tracking plans, event specifications, data structures](/assets/images/tracking-plan-overview-9f48fdeef0c5d4c593b416f9313c174c.png)

This example `Ecommerce Checkout Flow` tracking plan groups two event specifications for ecommerce checkout behavior:

- `Checkout Started` describes a `checkout_started` event, with an associated `cart` entity
- `Product Add To Cart` describes an `add_to_cart` event, with `cart` and `product` entities

The individual event and entity data structures can also be used in other event specifications and tracking plans.

---

# Manage data structures via the API
> Programmatically manage data structures through the API with endpoints for retrieving, validating, and deploying schemas to development and production registries.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/data-structures-api/

The data structures Console API endpoints focus on the main operations in the workflow around:

1. Retrieving existing data structures and their associated schemas
2. Creating or editing new or existing data structures
3. Validating a data structure
4. Deploying a data structure to a registry

## Retrieving data structures

The following `GET` requests allow you to retrieve data structures from both your development and production environment registries.

### Retrieve a list of all data structures

Use this request to:

- Retrieve a list of all data structures
- Retrieve a list of data structures filtered by `vendor` or `name` query parameters

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1`

### Retrieve a specific data structure

Use this request to retrieve a specific data structure by its hash (see 'Generating a data structure hash' below), which is generated on creation.

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}`

### Retrieve specific version of a specific data structure

Use this request to retrieve all versions of a specific data structure by its hash (see 'Generating a data structure hash' below)

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}/versions/{versionNumber}`

See the [detailed API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs) for all options.

#### Generating a data structure hash

To use the commands to retrieve information about a specific Data Structure, you need to encode its identifying parameters (`organization ID`, `vendor`, `name` and `format`) and hash it with SHA-256.

**Example:**

| Parameter       | Value                                  |
| --------------- | -------------------------------------- |
| Organization ID | `38e97db9-f3cb-404d-8250-cd227506e544` |
| Vendor          | `com.acme.event`                       |
| Schema name     | `search`                               |
| Format          | `jsonschema`                           |

First concatenate the information with a dash (-) as the separator: `38e97db9-f3cb-404d-8250-cd227506e544-com.acme.event-search-jsonschema`

And then hash them with SHA-256 to receive: `a41ef92847476c1caaf5342c893b51089a596d8ecd28a54d3f22d922422a6700`

## Validation

To validate that your schema is in proper JSON format and complies with warehouse loading requirements, you can use the validation `POST` requests.

`**POST** /api/msc/v1/organizations/{organizationId}/data-structures/v1/validation-requests`

### Example

```bash
curl 'https://console.snowplowanalytics.com/api/msc/v1/organizations/cad39ca5-3e1e-4e88-91af-87d977a4acd8/data-structures/v1/validation-requests' \
  -H 'authorization: Bearer YOUR_TOKEN' \
  -H 'content-type: application/json' \
  --data-binary '{
  "meta": {
    "hidden": false,
    "schemaType": "event",
    "customData": {}
  },
  "data": {
    "description": "Schema for an example event",
    "properties": {
      "example_field_1": {
        "type": "string",
        "description": "the example_field_1 means x",
        "maxLength": 128
      }
    },
    "additionalProperties": false,
    "type": "object",
    "required": [
      "example_field_1"
    ],
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
      "vendor": "com.acme",
      "name": "example_schema_name",
      "format": "jsonschema",
      "version": "1-0-0"
    }
  }
}'
```

Please note:

- the request's body has two parts:

  - one for data structure metadata as value to the `meta` key
  - one for the schema itself as value to the `data` key

- this example uses the synchronous version of validation that responds with the result immediately. There is also an asynchronous version available that returns a request ID that you can later poll to get the result.

- you can add metadata specific to your organization to the schema as key/value pairs in the `customData` object. See '[Managing Meta Data](#managing-meta-data)' for more information.

## Deployments

The deployment endpoints deal with getting a new or edited version of your data structure into your development and production environments.

`**GET** /api/msc/v1/organizations/{organizationId}/data-structures/v1/{dataStructureHash}/deployments`

`**POST** /api/msc/v1/organizations/{organizationId}/data-structures/v1/deployment-requests`

### Example

```bash
curl 'https://console.snowplowanalytics.com/api/msc/v1/organizations/cad39ca5-3e1e-4e88-91af-87d977a4acd8/data-structures/v1/deployment-requests' \
  -H 'authorization: Bearer MY_TOKEN' \
  -H 'content-type: application/json' \
  --data-binary '{
  "message": "",
  "source": "VALIDATED",
  "target": "DEV",
  "vendor": "com.acme",
  "name": "example_schema_name",
  "format": "jsonschema",
  "version": "1-0-0"
}'
```

Please note:

- This example demonstrates deployment from `VALIDATED` to `DEV`. The method is the same for Production, but you would change the variables where `"source": "DEV"` and `"target": "PROD"`
- The API enforces a workflow of validating, testing on development and then deploying to production. To achieve this you deploy from one environment to another; from (virtual environment) `VALIDATED` to `DEV`, then `DEV` to `PROD`.
- Only users designated as "admin" in the console have the permissions to promote from `DEV` to `PROD`.
- There is a sync option that will return the response of the deployment request directly. Otherwise you can poll for deployment responses using the deployment ID.
- The property for `message` can be sent with a deployment which will capture any change log notes that will be stored against the deployment. (Note that this specific property is a bolt-on feature and might not be available for your account.)

## Managing meta data

Meta data is used to add additional information to a Data Structure.

```json
"meta": {
  "hidden": false,
  "schemaType": "event",
  "customData": {}
}
```

The `hidden` property sets the data structure as visible (true) or not (false) in the console.

The `schemaType` property can be set as null | "event" | "entity".

The `customData` property is mapped as `[string, string]` and can be used to send across any key/value pairs you'd like to associate with the schema.

For example if you wanted to specify departmental ownership through meta data:

```json
"customData": { "department": "marketing" }
```

You can update the meta data for a data structure using the PUT endpoint:

`**PUT** ​/api​/msc​/v1​/organizations​/{organizationId}​/data-structures/v1/{dataStructureHash}​/meta`

## Migrating from the legacy API

The API described above supersedes the legacy one, which was to be found under `/api/schemas`. It offers under-the-hood improvements and a clear path forward to adding further value. To make the transition easier, we have kept the same models for the data being exchanged; it is therefore simply a matter of 1) updating the authentication method, and 2) switching to the new endpoints listed above.

The legacy API is currently tunneled to the new one, so there is no data differences to be expected after the upgrade. We will maintain this facade while supporting customers in upgrading. Your Customer Success Manager and our Engineers will be glad to assist you during this transition.

---

# Manage event specifications via the API
> Programmatically retrieve, create, edit, publish, deprecate, and delete event specifications using the Event Specifications API endpoints.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/event-specifications-api/

The event specifications Console API endpoints allow you to retrieve, create, edit, publish, deprecate, or delete event specifications.

> **Note:** In the previous API version (V1), event specifications were referred to as "tracking scenarios".

## Response Format

The event specifications API follows a specific response format for successful cases (`2xx`) and for scenarios where critical and non-critical errors may occur, such as (`422`) **Unprocessable Entity**.

```json
{
    "data": [
        // Event specifications
    ],
    "includes": [
        // Additional event specifications info
    ],
    "errors": [
        // Warnings or errors
    ]
}
```

- `data`: Contains the event specification/s, depending on the request
- `includes`: Provides additional information, such as the history of event specification changes
- `errors`: Holds a list of errors, which could be of type `Error` or `Warning`. **If the array field contains at least one error of type `Error`, the request will also return a `4xx` status code, indicating that it cannot perform the store operation. Any other severity different from `Error` will result in a `2xx` status code.**

## Compatibility Checks

Certain endpoints conduct a validation to verify the compatibility of a specific event specification event schema, `event.schema`, with the source data structure version referenced by `event.source`. When both `event.schema` and `event.source` are defined in the event specification, the compatibility checks will be executed.

```json
...
"event": {
    "source": "iglu:com.example/ui_actions/jsonschema/1-0-0",
    "schema": {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "description": "Event to capture search submit",
        "type": "object",
        "required": [
            "label",
            "action"
        ],
        "properties": {
            "action": {
                "type": "string",
                "enum": [
                    "click"
                ]
            },
            "label": {
                "type": "string",
                "enum": [
                    "Search"
                ]
            }
        },
        "additionalProperties": false
    }
}
...
```

However, the compatibility check will not only be performed against the version specified by the source data structure (`event.source` field, e.g., `1-0-0`), which we will refer to as the **current** version. It will also be conducted against the latest version available in Iglu, referred to as the **latest** version.

This approach is because it's common for a new event specification to utilize the latest version of the source data structure. However, as this data structure may evolve over time and become incompatible with the `event.schema` defined in the event specification, we provide a method to detect these compatibility issues. Consequently, customers can update the event specification to ensure compatibility.

In cases where an event specification is incompatible or we cannot determine it, some errors will be provided in the `errors` field of the [response](#response-format). These errors alerting of compatibility issues between the event specification and the source data structure will take a similar shape to the one below:

```json
...
"errors": [
    {
        "type":"Warning",
        "code":"SchemaIncompatible",
        "title":"Event specification with id: 59b5e250-91c4-45af-a63d-5f8cd39f4b67, event schema is INCOMPATIBLE with schema with name: test_event, vendor: com.snplow.msc.aws, version: 1-0-13",
        "source":"event.schema"
    }
]
...
```

Compatibility checks can result in three possible values: **Compatible**, **SchemaIncompatible**, or **SchemaUndecidable**.

- If **Compatible**, the event specification is compatible, and no errors will be appended to the `errors` response field
- If **SchemaIncompatible**, the event specification is incompatible against some version. If the check for the **current** version is incompatible, the `type` will be `Error`. For incompatibility with the **latest** version, the `type` will be `Warning`. If the requested operation involves persisting the event specification (create/update), an error of type `Error` will be appended to the response, the status code will be **422 Unprocessable Entity**, and the store operation will not persist. When fetching an event specification, the checks will run for both too, **current** and **latest** versions, and if incompatible, the error type will always be `Warning`, returning status code **200 Ok**.
- If **SchemaUndecidable**, it is indeterminable whether the event specification is compatible with a specific version due to the use of some advanced JSON-Schema features and the high computational cost of checking compatibility. The `type` will always be `Warning`, and the user is responsible for ensuring that the event specification is compatible with the source data structure. A warning will be attached to the `errors` response field.

> **Info:** The algorithm used to perform the compatibility check is based on the [Finding Data Compatibility Bugs with JSON Subschema Checking](https://dl.acm.org/doi/pdf/10.1145/3460319.3464796) paper, authored by Andrew Habib, Avraham Shinnar, and Michael Pradel.

## Retrieve a List of Event Specifications

Use this request to retrieve a list of event specifications within an organization, which will be wrapped into the `data` field of the [response](#response-format).

`GET /api/msc/v1/organizations/{organizationId}/event-specs/v1`

The `organizationId` parameter is required.

### Query Parameters and Filters

You can filter the results based on the following query parameters:

- `dataProductId`: Filters the event specifications associated with a particular tracking plan
- `sourceId`: Filters the event specifications associated with a particular data structure, inferred from the `event.source` field
- `sourceVersion`: Filters the event specifications associated with a specific data structure version when used with `dataStructureId`.
- `withLatestHistory`: When `true`, it will return a list of event specifications, with the latest change per event specification attached to the `includes` array field. The relation between event specifications in `data` and history in `includes` can be determined by `id = eventSpecId`.
- `status`: Filters the event specifications that match the specified status

> **Info:** If no query parameters are provided, it will return all the event specifications for an organization.

## Retrieve a Specific Event Specification

Use this request to retrieve a specific event specification within an organization. The retrieved event specification will be wrapped into the `data` field of the response.

`GET /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

Query parameters `organizationId` and `eventSpecId` are required:

- `withHistory`: When `true`, returns a list with the history for the event specification in the `includes` array field of the response, related to the event specification by its id
- `status`: Filters the event specifications that match the specified status.

## Creating an Event Specification

Use this request to create an event specification within an organization.

`POST /api/msc/v1/organizations/{organizationId}/event-specs/v1`

Query parameter `organizationId` is required.

Request body example:

```json
{
  "spec": {
    "name": "Search",
    "description": "Tracking the use of the search box",
    "event": {
      "source": "iglu:com.example/ui_actions/jsonschema/1-0-0"
    }
  },
  "message": "update"
}
```

The creation form has two fields at the top level, as shown in the example above:

- `message`: An optional field to provide a message
- `spec`: The definition of the event specification, which should comply with the [validations](#validations).

By default, the event specification will be created with `spec.status` set to `draft` and `spec.version` set to `0` if not provided. These values can be changed and managed after creation. Here is an example response:

```json
{
    "data": [
        {
            "id": "5a203ef8-939b-4fd1-914e-f12a3dd1a869",
            "version": 0,
            "status": "draft",
            "name": "Search",
            "description": "Tracking the use of the search box",
            "event": {
                "source": "iglu:com.example/ui_actions/jsonschema/1-0-0"
            }
        }
    ],
    "includes": [
        {
            "author": "39b81015-1bd5-4b37-96c7-3296cabaa36f",
            "message": "initial draft",
            "date": "2023-04-26T14:41:48.708191Z",
            "eventSpecId": "5a203ef8-939b-4fd1-914e-f12a3dd1a869",
            "version": 0,
            "status": "draft",
            "type": "History"
        }
    ],
    "errors": []
}
```

### Validations

- `spec.event.source`: If provided it should match a valid and existing Iglu URI
- `spec.name`: It validates that the `spec.name` of an event specification is unique within the data structure context, inferred from the source data structure `spec.event.source` if provided
- `spec.version`: If provided should be equal or greater than zero
- `spec.status`: If provided should match one of `draft`, `published` or `deprecated`
- `spec.entities`: If provided it will validate that the entities, `spec.entities.tracked` and `spec.entities.enriched`, are not duplicated and that they exist
- `spec.dataProductId`: If provided it will validate that the tracking plans exists. (Coming soon)

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

## Editing an Event Specification

Use this request to edit an event specification within an organization. The format of the request and response is the same as during creation.

The `organizationId` and `eventSpecId` parameters are required.

`PUT /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

### Publishing an Event Specification

When editing an event specification, it can be published by setting the `status` to `published`. Currently, this will indicate to the event specification consumers (for instance, front-end developers) that the tracking design is ready to be implemented or consumed.

By default, when an event specification is created and no value is provided for `spec.status`, it will be set to `draft`. With this, we suggest an event specification lifecycle that we recommend following, but we allow a certain degree of flexibility to accommodate unique customer use cases. Here is the suggested lifecycle:

```mermaid
graph LR
    style Start fill:#633EB5, stroke:#000000, stroke-width:0px;
    style Draft color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style Published color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style Deprecated color:#89251F, fill:#FDF3F2, stroke:#89251F, stroke-width:1px;
    style Deleted fill:#D63A31, stroke:#000000, stroke-width:1px;

    Start(( )) -->|Create| Draft
    Draft -->|Publish| Published
    Draft -->|Delete| Deleted
    Published -->|Deprecate| Deprecated
    Deprecated -->|Undeprecate| Draft
    Deleted((Deleted))
```

In addition to this lifecycle, and in conjunction with versioning, we enforce that when an event specification is **published**, the versions between two published versions are **discarded**. For example:

Publish new version, before squash:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style C color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style D color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style E color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    linkStyle default stroke-width:2px,fill:#F2F4F7,stroke:#633EB5,color:#633EB5

    A[Published 1] --> B[Draft 2]
    B[Draft 2] --> C[Draft 3]
    C[Draft 3] --> D[Draft 4]
    D[Draft 4] --> E[Published 5]
```

After discarding intermediate versions:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    linkStyle default stroke-width:2px,fill:#F2F4F7,stroke:#633EB5,color:#633EB5

    A[Published 1] --> B[Published 5]
```

### Deprecating an Event Specification

When editing an event specification, it can be deprecated by setting the `status` to `deprecated`. This is a way of informing event specifications consumers (for instance, developers) not to rely on the tracking anymore.

### Validations

- `spec.event.source`: If provided, it should match a valid and existing Iglu URI
- `spec.name`: It validates that the `spec.name` of an event specification is unique within the data structure context, inferred from the source data structure `spec.event.source` if provided
- `spec.version`: If provided, it should be equal to or greater than zero, should not exist, and be greater than the last published version
- `spec.status`: If provided, it should match one of `draft`, `published`, or `deprecated`
- `spec.entities`: If provided, it will validate that the entities, `spec.entities.tracked` and `spec.entities.enriched`, are not duplicated and that they exist
- `spec.dataProductId`: If provided, it will validate that the tracking plan exists

> **Info:** This endpoint will trigger [**compatibility checking**](#compatibility-checks) if `event.source` and `event.schema` are defined.

## Deleting an Event Specification

Use this request to delete an event specification within an organization.

`DELETE /api/msc/v1/organizations/{organizationId}/event-specs/v1/{eventSpecId}`

> **Warning:** Please note that this action is irreversible and will permanently delete the event specification.

---

# Programmatic management using CLI or API
> Automate Event Studio workflows using Snowplow CLI for git-ops integration or the APIs for custom tooling and CI/CD pipelines.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/

Data structures and tracking plans can be managed programmatically, enabling git-ops workflows, CI/CD integration, and custom tooling.

Partnered with other tools like the data structures [CI tool](/docs/testing/data-structures-ci-tool/) and the testing pipeline [Snowplow Micro](/docs/testing/snowplow-micro/), it's possible to have a very robust and automated data structure workflow that ensures data quality upstream of data hitting your pipeline.

## Snowplow CLI

The [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) provides file-based workflows for git-ops integration. It has commands to:

- Download and upload [data structures](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/) as local YAML/JSON files
- Manage [tracking plans](/docs/event-studio/programmatic-management/snowplow-cli/tracking-plans/), event specifications, and source applications
- Validate resources before publishing
- Integrate with version control and code review workflows

The [Snowplow CLI MCP server](/docs/llms-support/mcp-server/) enables AI assistants to interact with your tracking plan resources through natural language.

## Console API

Use the Console REST API for direct programmatic access to tracking plans and data structures. This is useful for integration with other tools, or when you need more control than file-based workflows allow.

It has endpoints for three resource types:

- [Data structures](/docs/event-studio/programmatic-management/data-structures-api/) (`/data-structures/v1`): retrieve, validate, and deploy schemas to development and production registries
- [Tracking plans](/docs/event-studio/programmatic-management/tracking-plans-api/) (`/data-products/v2`): create and update tracking plans, view change history, and manage subscriptions
- [Event specifications](/docs/event-studio/programmatic-management/event-specifications-api/) (`/event-specs/v1`): create, publish, deprecate, and delete event specifications within tracking plans

You can explore all available endpoints in the [Swagger API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs).

> **Note:** By default, Snowplow pipelines use Iglu Server schema registries. Each pipeline has a development and a production Iglu Server instance.
>
> The Console API only works with these registries. If you're using a custom static S3 registry instead, you'll need to update your registry manually.

### Authentication

Each request requires your organization ID and an authorization token. You can find your organization ID [on the **Manage organization** page](https://console.snowplowanalytics.com/settings) in Console.

Follow the instructions in the [Account management](/docs/account-management/) section to obtain an access token for API authentication.

To be able to post sample requests in the [Swagger API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs), click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

---

# Manage data structures via Snowplow CLI
> Use the Snowplow CLI data-structures command to generate, download, validate, and publish data structures with git-ops workflows and JSON Schema validation.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/snowplow-cli/data-structures/

The `data-structures` subcommand of [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) provides a collection of functionality to ease the integration of custom development and publishing workflows.

## Snowplow CLI Prerequisites

Installed and configured [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/)

## Available commands

### Creating data structures

```bash
snowplow-cli ds generate login_click ./folder-name
```

Will create a minimal data structure template in a new file `./folder-name/login_click.yaml`. Note that you will need to add a vendor name to the template before it will pass validation. Alternatively supply a vendor at creation time with the `--vendor com.acme` flag.

### Downloading data structures

```bash
snowplow-cli ds download
```

This command will retrieve all organization data structures. By default it will create a folder named `data-structures` in the current working directory to put them in. It uses a combination of vendor and name to further break things down.

Given a data structure with `vendor: com.acme` and `name: link_click` and assuming the default format of yaml the resulting folder structure will be `./data-structures/com.acme/link_click.yaml`.

> **Note:** The CLI download command only retrieves data structures that have been deployed to at least the development environment. **Draft data structures** that haven't been deployed yet will not be included in the download.

### Validating data structures

```bash
snowplow-cli ds validate ./folder-name
```

This command will find all files under `./folder-name` (if omitted then `./data-structures`) and attempt to validate them using Snowplow Console. It will assert the following

1. Is each file a valid format (yaml/json) with expected fields
2. Does the schema in the file conform to [snowplow expectations](/docs/fundamentals/schemas/#self-describing-json-schema-anatomy)
3. Given the organization's [loading configuration](/docs/destinations/warehouses-lakes/) will any schema version number choices have a potentially negative effect on data loading

If any validations fail the command will report the problems to stdout and exit with status code 1.

### Publishing data structures

```bash
snowplow-cli ds publish dev ./folder-name
```

This command will find all files under `./folder-name` (if omitted then `./data-structures`) and attempt to publish them to Snowplow Console in the environment provided (`dev` or `prod`).

Publishing to `dev` will also cause data structures to be validated with the `validate` command before upload. Publishing to `prod` will not validate but requires all data structures referenced to be present on `dev`.

---

# Snowplow CLI for data management
> Command-line tool for downloading and syncing data structures and tracking plans to Snowplow Console, enabling git-ops workflows with reviews and branching.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/snowplow-cli/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

Snowplow CLI brings data management elements of Snowplow Console into the command line. It allows you to download your data structures and tracking plans to YAML/JSON files and publish them back to Console. This enables git-ops-like workflows, where your tracking design lives in your repository alongside your application code. Changes are reviewed and deployed through your standard development process.

## Install

Snowplow CLI can be installed with [homebrew](https://brew.sh/):

```bash
brew install snowplow/taps/snowplow-cli
```

Verify the installation with

```bash
snowplow-cli --help
```

For systems where homebrew is not available binaries for multiple platforms can be found in [releases](https://github.com/snowplow/snowplow-cli/releases).

Example installation for `linux_x86_64` using `curl`

```bash
curl -L -o snowplow-cli https://github.com/snowplow/snowplow-cli/releases/latest/download/snowplow-cli_linux_x86_64
chmod u+x snowplow-cli
```

Verify the installation with

```bash
snowplow-cli --help
```

## Configure

### Automated Setup

The easiest way to configure Snowplow CLI is using the built-in `setup` command:

```bash
snowplow-cli setup
```

This command will:

- Guide you through device authentication with Snowplow Console
- Automatically create and configure your API credentials
- Set up your organization ID

**Prerequisites:** Your Snowplow Console account must have sufficient permissions to create API keys.

You can also use optional flags:

- `--read-only`: Create a read-only API key
- `--dotenv`: Store configuration as .env file in current working directory

### Manual Configuration

If you prefer manual configuration, you will need these values:

- An API Key ID and the corresponding API Key (secret), which are generated from the [credentials section](https://console.snowplowanalytics.com/credentials) in Console.
- Your Organization ID, which you can find [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

Snowplow CLI can take its configuration from a variety of sources. More details are available from `snowplow-cli data-structures --help`. Variations on these three examples should serve most cases.

**env variables or .dotenv file:**

```bash
SNOWPLOW_CONSOLE_API_KEY_ID=********-****-****-****-************
SNOWPLOW_CONSOLE_API_KEY=********-****-****-****-************
SNOWPLOW_CONSOLE_ORG_ID=********-****-****-****-************
```

**$HOME/.config/snowplow/snowplow\.yml:**

```yaml
console:
  api-key-id: ********-****-****-****-************
  api-key: ********-****-****-****-************
  org-id: ********-****-****-****-************
```

**inline arguments:**

```bash
snowplow-cli data-structures --api-key-id ********-****-****-****-************ --api-key ********-****-****-****-************ --org-id ********-****-****-****-************
```

***

Snowplow CLI defaults to yaml format. It can be changed to json by either providing a `--output-format json` flag or setting the `output-format: json` config value. It will work for all commands where it matters, not only for `generate`.

### Verify Configuration

After configuration, you can verify that everything is working correctly using the `status` command:

```bash
snowplow-cli status
```

This command will:

- Check that your API credentials are properly configured
- Verify connectivity to Snowplow Console
- Confirm your organization access
- Provide helpful troubleshooting information if issues are found

If the status check fails, the command will suggest the next steps such as running `snowplow-cli setup` to reconfigure your credentials.

### Configure YAML language server (optional)

During the tracking plans authoring process it's convenient to have the editor highlight errors, suggest possible values, and show examples. All files generated by Snowplow CLI's `generate` and `download` commands will contain a special comment as part of the file. This comment can be interpreted by a YAML Language server.

For VS Code you can get this authoring functionality using this [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml). Instructions for other editors can be found [here](https://github.com/redhat-developer/yaml-language-server?tab=readme-ov-file#clients).

After that running `snowplow-cli ds generate test` and opening the generated file in the configured editor of choice should look something like the following:

![](/assets/images/lspValidation-21bfbfb84fe78d4b98d42b0ef09eda5b.png)

## MCP server

The Snowplow CLI includes a local Model Context Protocol (MCP) server that enables natural language interaction with AI assistants like Claude Desktop, Cursor, GitHub Copilot, and other MCP-compatible clients for creating, validating, and managing your Snowplow tracking plans. This allows you to:

- Create and validate data structures through conversation
- Analyze tracking requirements and suggest implementations
- Validate tracking plans and source applications

For setup instructions and configuration examples for different MCP clients, see our [MCP tutorial](/tutorials/snowplow-cli-mcp/introduction/).

## Use cases

- [Manage your data structures with snowplow-cli](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/)
- [Set up a GitHub CI/CD pipeline to manage data structures and tracking plans](/tutorials/data-structures-in-git/introduction/)

---

# Snowplow CLI command reference
> Complete reference for Snowplow CLI commands including data-products and data-structures subcommands with options and usage examples.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/snowplow-cli/reference/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

This page contains the complete reference for the Snowplow CLI commands.

## Data-Products

Work with Snowplow tracking plans

### Examples

```text
  $ snowplow-cli data-products validate
```

### Options

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
  -h, --help                  help for data-products
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                          Then on:
                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                            Windows %AppData%\snowplow\snowplow.yml
      --debug             Log output level to Debug
      --env-file string   Environment file (.env). Defaults to .env in current directory
                          Then on:
                            Unix $HOME/.config/snowplow/.env
                            Darwin $HOME/Library/Application Support/snowplow/.env
                            Windows %AppData%\snowplow\.env
      --json-output       Log output as json
  -q, --quiet             Log output level to Warn
  -s, --silent            Disable output
```

## Data-Products Add-Event-Spec

Add new event spec to an existing tracking plan

### Synopsis

Adds one or more event specifications to an existing tracking plan file, or print them to the stdout. The command takes the path to a tracking plan file and adds the specified event specifications to it. When run without the argument - it will print generated event specs to the stdout. The command will attempt to keep the formatting and comments of the original file intact, but it's a best effort approach. Some comments might be deleted, some formatting changes might occur.

```text
snowplow-cli data-products add-event-spec {path} [flags]
```

### Examples

```text
  $ snowplow-cli dp add-event-spec --event-spec user_login --event-spec page_view ./my-data-product.yaml
  $ snowplow-cli dp add-es ./data-products/analytics.yaml -e "checkout_completed" -e "item_purchased"
```

### Options

```text
  -e, --event-spec stringArray   Name of event spec to add
  -h, --help                     help for add-event-spec
  -f, --output-format string     Format of stdout output. Only applicable when the file in not specified. Json or yaml are supported (default "yaml")
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Download

Download all tracking plans, event specs and source apps from Snowplow Console

### Synopsis

Downloads the latest versions of all tracking plans, event specs and source apps from Snowplow Console.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products download {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp download
  $ snowplow-cli dp download ./my-data-products
```

### Options

```text
  -h, --help                   help for download
  -f, --output-format string   Format of the files to read/write. json or yaml are supported (default "yaml")
      --plain                  Don't include any comments in yaml files
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Generate

Generate new tracking plans and source applications locally

### Synopsis

Will write new tracking plans and/or source application to file based on the arguments provided.

Example: $ snowplow-cli dp gen --source-app "Mobile app" Will result in a new source application getting written to './data-products/source-applications/mobile-app.yaml'

$ snowplow-cli dp gen --data-product "Ad tracking" --output-format json --data-products-directory dir1 Will result in a new tracking plan getting written to './dir1/ad-tracking.json'

```text
snowplow-cli data-products generate [paths...] [flags]
```

### Examples

```text
  $ snowplow-cli dp generate --source-app "Mobile app" --source-app "Web app" --data-product "Signup flow"
```

### Options

```text
      --data-product stringArray         Name of tracking plan to generate
      --data-products-directory string   Directory to write tracking plans to (default "data-products")
  -h, --help                             help for generate
      --output-format string             File format (yaml|json) (default "yaml")
      --plain                            Don't include any comments in yaml files
      --source-app stringArray           Name of source app to generate
      --source-apps-directory string     Directory to write source apps to (default "data-products/source-apps")
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Purge

Purges (permanently removes) all remote tracking plans and source apps that do not exist locally

### Synopsis

Purges (permanently removes) all remote tracking plans and source apps that do not exist locally.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products purge {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp purge
  $ snowplow-cli dp purge ./my-data-products
```

### Options

```text
  -h, --help   help for purge
  -y, --yes    commit to purge
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Release

Sync tracking plans, event specs and source apps to Snowplow Console, then release event specs to be available within your pipeline

### Synopsis

Sync tracking plans, event specs and source apps to Snowplow Console, then release event specs.

This command runs 'sync' first, then releases the event specs. Releasing sets the event spec status to 'published' and pushes them to the pipeline, enabling event spec inference. Only event specs that exist locally will be released. Each event spec must have an event defined, and all referenced events and entities must be published to the production environment.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products release {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp release
  $ snowplow-cli dp release ./my-data-products
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)
  -d, --dry-run           Only print planned changes without performing them
      --gh-annotate       Output suitable for github workflow annotation (ignores -s)
  -h, --help              help for release
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Sync

Sync tracking plans, event specs and source apps to Snowplow Console

### Synopsis

Sync tracking plans, event specs and source apps to Snowplow Console.

This command syncs local files with Snowplow Console. Tracking plans, event specs and source apps are created or updated as needed. Tracking plans and source apps that exist in Snowplow Console are updated in place. Structural changes to event specs (name, event, entities) will instead create a new draft version of the event spec. Use 'release' to also release event specs, which changes the status in Snowplow Console to "published" and enables event spec inference.

If no directory is provided then defaults to 'data-products' in the current directory. Source apps are stored in the nested 'source-apps' directory

```text
snowplow-cli data-products sync {directory ./data-products} [flags]
```

### Examples

```text
  $ snowplow-cli dp sync
  $ snowplow-cli dp sync ./my-data-products
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)
  -d, --dry-run           Only print planned changes without performing them
      --gh-annotate       Output suitable for github workflow annotation (ignores -s)
  -h, --help              help for sync
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Products Validate

Validate tracking plans and source applications with Snowplow Console

### Synopsis

Sends all tracking plans and source applications from \<path> for validation by Snowplow Console.

```text
snowplow-cli data-products validate [paths...] [flags]
```

### Examples

```text
  $ snowplow-cli dp validate ./data-products ./source-applications
  $ snowplow-cli dp validate ./src
```

### Options

```text
  -c, --concurrency int   The number of validation requests to perform at once (maximum 10) (default 3)
      --full              Perform compatibility check on all files, not only the ones that were changed
      --gh-annotate       Output suitable for github workflow annotation (ignores -s)
  -h, --help              help for validate
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures

Work with Snowplow data structures

### Examples

```text
  $ snowplow-cli data-structures generate my_new_data_structure
  $ snowplow-cli ds validate
  $ snowplow-cli ds publish dev
```

### Options

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
  -h, --help                  help for data-structures
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                          Then on:
                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                            Windows %AppData%\snowplow\snowplow.yml
      --debug             Log output level to Debug
      --env-file string   Environment file (.env). Defaults to .env in current directory
                          Then on:
                            Unix $HOME/.config/snowplow/.env
                            Darwin $HOME/Library/Application Support/snowplow/.env
                            Windows %AppData%\snowplow\.env
      --json-output       Log output as json
  -q, --quiet             Log output level to Warn
  -s, --silent            Disable output
```

## Data-Structures Download

Download all data structures from Snowplow Console

### Synopsis

Downloads the latest versions of all data structures from Snowplow Console.

Will retrieve schema contents from your development environment. If no directory is provided then defaults to 'data-structures' in the current directory.

By default, data structures with empty schemaType (legacy format) are skipped. Use --include-legacy to include them (they will be set to 'entity' schemaType).

```text
snowplow-cli data-structures download {directory ./data-structures} [flags]
```

### Examples

```text
  $ snowplow-cli ds download

  Download data structures matching com.example/event_name* or com.example.subdomain*
  $ snowplow-cli ds download --match com.example/event_name --match com.example.subdomain

  Download with custom output format and directory
  $ snowplow-cli ds download --output-format json ./my-data-structures

  Include legacy data structures with empty schemaType
  $ snowplow-cli ds download --include-legacy
```

### Options

```text
  -h, --help                   help for download
      --include-drafts         Include drafts data structures
      --include-legacy         Include legacy data structures with empty schemaType (will be set to 'entity')
      --match stringArray      Match for specific data structure to download (eg. --match com.example/event_name or --match com.example)
  -f, --output-format string   Format of the files to read/write. json or yaml are supported (default "yaml")
      --plain                  Don't include any comments in yaml files
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures Generate

Generate a new data structure locally

### Synopsis

Will write a new data structure to file based on the arguments provided.

Example: $ snowplow-cli ds gen login\_click --vendor com.example Will result in a new data structure getting written to './data-structures/com.example/login\_click.yaml' The directory 'com.example' will be created automatically.

$ snowplow-cli ds gen login\_click Will result in a new data structure getting written to './data-structures/login\_click.yaml' with an empty vendor field. Note that vendor is a required field and will cause a validation error if not completed.

```text
snowplow-cli data-structures generate login_click {directory ./data-structures} [flags]
```

### Examples

```text
  $ snowplow-cli ds generate my-ds
  $ snowplow-cli ds generate my-ds ./my-data-structures
```

### Options

```text
      --entity                 Generate data structure as an entity
      --event                  Generate data structure as an event (default true)
  -h, --help                   help for generate
      --output-format string   Format for the file (yaml|json) (default "yaml")
      --plain                  Don't include any comments in yaml files
      --vendor string          A vendor for the data structure.
                               Must conform to the regex pattern [a-zA-Z0-9-_.]+
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures Publish

Publishing commands for data structures

### Synopsis

Publishing commands for data structures

Publish local data structures to Snowplow Console.

### Options

```text
  -h, --help   help for publish
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures Publish Dev

Publish data structures to your development environment

### Synopsis

Publish modified data structures to Snowplow Console and your development environment

The 'meta' section of a data structure is not versioned within Snowplow Console. Changes to it will be published by this command.

```text
snowplow-cli data-structures publish dev [paths...] default: [./data-structures] [flags]
```

### Examples

```text
  $ snowplow-cli ds publish dev
  $ snowplow-cli ds publish dev --dry-run
  $ snowplow-cli ds publish dev --dry-run ./my-data-structures ./my-other-data-structures
```

### Options

```text
  -d, --dry-run       Only print planned changes without performing them
      --gh-annotate   Output suitable for github workflow annotation (ignores -s)
  -h, --help          help for dev
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures Publish Prod

Publish data structures to your production environment

### Synopsis

Publish data structures from your development to your production environment

Data structures found on \<path...> which are deployed to your development environment will be published to your production environment.

```text
snowplow-cli data-structures publish prod [paths...] default: [./data-structures] [flags]
```

### Examples

```text

	$ snowplow-cli ds publish prod
	$ snowplow-cli ds publish prod --dry-run
	$ snowplow-cli ds publish prod --dry-run ./my-data-structures ./my-other-data-structures
	
```

### Options

```text
  -d, --dry-run   Only print planned changes without performing them
  -h, --help      help for prod
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Data-Structures Validate

Validate data structures with Snowplow Console

### Synopsis

Sends all data structures from \<path> for validation by Snowplow Console.

```text
snowplow-cli data-structures validate [paths...] default: [./data-structures] [flags]
```

### Examples

```text
  $ snowplow-cli ds validate
  $ snowplow-cli ds validate ./my-data-structures ./my-other-data-structures
```

### Options

```text
      --gh-annotate   Output suitable for github workflow annotation (ignores -s)
  -h, --help          help for validate
```

### Options inherited from parent commands

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --config string         Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                              Then on:
                                Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                                Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                                Windows %AppData%\snowplow\snowplow.yml
      --debug                 Log output level to Debug
      --env-file string       Environment file (.env). Defaults to .env in current directory
                              Then on:
                                Unix $HOME/.config/snowplow/.env
                                Darwin $HOME/Library/Application Support/snowplow/.env
                                Windows %AppData%\snowplow\.env
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
      --json-output           Log output as json
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
  -q, --quiet                 Log output level to Warn
  -s, --silent                Disable output
```

## Mcp

Start an MCP (Model Context Protocol) stdio server for Snowplow validation and context

### Synopsis

Start an MCP (Model Context Protocol) stdio server that provides tools for:

- Validating Snowplow files (data-structures, data-products, source-applications)
- Retrieving the built-in schema and rules that define how Snowplow data structures, tracking plans, and source applications should be structured

```text
snowplow-cli mcp [flags]
```

### Examples

```text

  Claude Desktop config:
  {
    "mcpServers": {
      ...
      "snowplow-cli": {
        "command": "snowplow-cli", "args": ["mcp"]
      }
    }
  }

  VS Code '\<workspace\>/.vscode/mcp.json':
  {
    "servers": {
      ...
      "snowplow-cli": {
        "type": "stdio",
        "command": "snowplow-cli", "args": ["mcp"]
      }
    }
  }

  Cursor '\<workspace\>/.cursor/mcp.json':
  {
    "mcpServers": {
      ...
      "snowplow-cli": {
        "command": "snowplow-cli", "args": ["mcp", "--base-directory", "."]
      }
    }
  }

Note:
  This server's validation tools require filesystem paths to validate assets. For full
  functionality, your MCP client needs filesystem write access so created assets can be
  saved as files and then validated.

Setup options:
  - Enable filesystem access in your MCP client, or
  - Run alongside an MCP filesystem server (e.g., @modelcontextprotocol/server-filesystem)
```

### Options

```text
  -S, --api-key string          Snowplow Console api key
  -a, --api-key-id string       Snowplow Console api key id
      --base-directory string   The base path to use for relative file lookups. Useful for clients that pass in relative file paths.
      --dump-context            Dumps the result of the get_context tool to stdout and exits.
  -h, --help                    help for mcp
  -H, --host string             Snowplow Console host (default "https://console.snowplowanalytics.com")
  -m, --managed-from string     Link to a github repo where the data structure is managed
  -o, --org-id string           Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                          Then on:
                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                            Windows %AppData%\snowplow\snowplow.yml
      --debug             Log output level to Debug
      --env-file string   Environment file (.env). Defaults to .env in current directory
                          Then on:
                            Unix $HOME/.config/snowplow/.env
                            Darwin $HOME/Library/Application Support/snowplow/.env
                            Windows %AppData%\snowplow\.env
      --json-output       Log output as json
  -q, --quiet             Log output level to Warn
  -s, --silent            Disable output
```

## Setup

Set up Snowplow CLI with device authentication

### Synopsis

Authenticate with Snowplow Console using device authentication flow and create an API key

```text
snowplow-cli setup [flags]
```

### Examples

```text
  $ snowplow-cli setup
  $ snowplow-cli setup --read-only
```

### Options

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
      --auth0-domain string   Auth0 domain (default "id.snowplowanalytics.com")
      --client-id string      Auth0 Client ID for device auth (default "EXQ3csSDr6D7wTIiebNPhXpgkSsOzCzi")
      --dotenv                Store as .env file in current working directory
  -h, --help                  help for setup
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
      --read-only             Create a read-only API key
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                          Then on:
                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                            Windows %AppData%\snowplow\snowplow.yml
      --debug             Log output level to Debug
      --env-file string   Environment file (.env). Defaults to .env in current directory
                          Then on:
                            Unix $HOME/.config/snowplow/.env
                            Darwin $HOME/Library/Application Support/snowplow/.env
                            Windows %AppData%\snowplow\.env
      --json-output       Log output as json
  -q, --quiet             Log output level to Warn
  -s, --silent            Disable output
```

## Status

Check Snowplow CLI configuration and connectivity

### Synopsis

Verify that the CLI is properly configured and can connect to Snowplow Console

```text
snowplow-cli status [flags]
```

### Examples

```text
  $ snowplow-cli status
```

### Options

```text
  -S, --api-key string        Snowplow Console api key
  -a, --api-key-id string     Snowplow Console api key id
  -h, --help                  help for status
  -H, --host string           Snowplow Console host (default "https://console.snowplowanalytics.com")
  -m, --managed-from string   Link to a github repo where the data structure is managed
  -o, --org-id string         Your organization id
```

### Options inherited from parent commands

```text
      --config string     Config file. Defaults to $HOME/.config/snowplow/snowplow.yml
                          Then on:
                            Unix $XDG_CONFIG_HOME/snowplow/snowplow.yml
                            Darwin $HOME/Library/Application Support/snowplow/snowplow.yml
                            Windows %AppData%\snowplow\snowplow.yml
      --debug             Log output level to Debug
      --env-file string   Environment file (.env). Defaults to .env in current directory
                          Then on:
                            Unix $HOME/.config/snowplow/.env
                            Darwin $HOME/Library/Application Support/snowplow/.env
                            Windows %AppData%\snowplow\.env
      --json-output       Log output as json
  -q, --quiet             Log output level to Warn
  -s, --silent            Disable output
```

---

# Managing tracking plans via the CLI
> Use the Snowplow CLI data-products command to create, download, validate, sync, and release tracking plans, event specifications, and source applications with git-ops workflows.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/snowplow-cli/tracking-plans/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

The `data-products` subcommand of [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) provides a collection of functionality to ease the integration of custom development and publishing workflows.

## Snowplow CLI Prerequisites

Installed and configured [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/)

## Available commands

### Creating tracking plan

```bash
snowplow-cli dp generate --data-product my-data-product
```

This command creates a minimal tracking plan template in a new file `./data-products/my-data-product.yaml`.

### Creating source application

```bash
snowplow-cli dp generate --source-app my-source-app
```

This command creates a minimal source application template in a new file `./data-products/source-apps/my-source-app.yaml`.

### Creating event specification

To create an event specification, you need to modify the existing data-product file and add an event specification object. Here's a minimal example:

```yaml
apiVersion: v1
resourceType: data-product
resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0
data:
    name: test-cli
    sourceApplications: []
    eventSpecifications:
        - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca
          name: test
          event:
            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0
```

> **Warning:** The `source` fields of events and entities must refer to a deployed data structure. Referring to a locally created data structure is not yet supported.

### Linking tracking plan to a source application

To link a tracking plan to a source application, provide a list of references to the source application files in the `data.sourceApplications` field. Here's an example:

```yaml
apiVersion: v1
resourceType: data-product
resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0
data:
    name: test-cli
    sourceApplications:
        - $ref: ./source-apps/my-source-app.yaml
```

### Modifying the event specifications source applications

By default event specifications inherit all the source applications of the tracking plan. If you want to customise it, you can use the `excludedSourceApplications` in the event specification description to remove a given source application from an event specification.

```yaml
apiVersion: v1
resourceType: data-product
resourceName: 3d3059c4-d29b-4979-a973-43f7070e1dd0
data:
    name: test-cli
    sourceApplications:
        - $ref: ./source-apps/generic.yaml
        - $ref: ./source-apps/specific.yaml
    eventSpecifications:
        - resourceName: 11d881cd-316e-4286-b5d4-fe7aebf56fca
          name: All source apps
          event:
            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0
        - resourceName: b9c994a0-03b2-479c-b1cf-7d25c3adc572
          name: Not quite everything
          excludedSourceApplications:
            - $ref: ./source-apps/specific.yaml
          event:
            source: iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0
```

In this example event specification `All source apps` is related to both `generic` and `specific` source apps, but event specification `Not quite everything` is related only to the `generic` source application.

### Downloading tracking plans, event specifications and source apps

```bash
snowplow-cli dp download
```

This command retrieves all organization tracking plans, event specifications, and source applications. By default, it creates a folder named `data-products` in your current working directory. You can specify a different folder name as an argument if needed. The command creates the following structure:

- A main `data-products` folder containing your tracking plan files
- A `source-apps` subfolder containing source application definitions
- Event specifications embedded within their related tracking plan files.

### Validating tracking plans, event specifications and source applications

```bash
snowplow-cli dp validate
```

This command scans all files under `./data-products` and validates them using Snowplow Console. It checks:

1. Whether each file is in a valid format (YAML/JSON) with correctly formatted fields
2. Whether all source application references in the tracking plan files are valid
3. Whether event specification rules are compatible with their schemas If validation fails, the command displays the errors in the console and exits with status code 1.

### Syncing tracking plans, event specifications and source applications

```bash
snowplow-cli dp sync
```

This command locates all files under `./data-products`, validates them, and pushes local changes to Console. Tracking plans and source applications are updated in place. For event specifications, a new version in draft status may be created if the change is structural (name change, event change, or rules change).

> **Warning:** The old `publish` command has been renamed to `sync`. Running `dp publish` still works as an alias for backward compatibility, but it may be removed in a future release. Update your scripts to use `dp sync` instead.

### Releasing event specifications

```bash
snowplow-cli dp release
```

This command first syncs local files with Console (like `sync`), then releases any draft event specifications. Releasing marks event specifications as published and enables event spec inference. Only event specifications that are part of the local tracking plan files are affected — other event specifications in Console are left unchanged. Event specifications without an event are skipped.

Use `sync` if you only want to push changes without changing event specification status.

---

# Managing tracking plans via the API
> Programmatically manage tracking plans through the API with endpoints for creating, updating, retrieving, and managing subscriptions for automated workflows and version control integration.
> Source: https://docs.snowplow.io/docs/event-studio/programmatic-management/tracking-plans-api/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

Use the tracking plans Console API endpoints to programmatically manage your tracking plans.

## Retrieving Information about Tracking Plans

The following `GET` requests are designed to allow you to access information about tracking plans.

### Retrieve a List of All Tracking Plans

To retrieve a comprehensive list of all tracking plans in your organization, you can use the following GET request:

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2`

Path parameter `organizationId` is required.

### Retrieving Information about a Specific Tracking Plan

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

Path parameters `organizationId` and `dataProductId` are required.

When retrieving a tracking plan, it could also contain an array field `data[].tracking_scenarios` that will include the `id` and `url` of the associated event specifications. For example:

```json
"data": [
  ...
  "eventSpecs": [
    {
      "id": "d1336abc-1b60-46f7-be2d-2105f2daf283",
      "url": "https://console.snowplowanalytics.com/api/msc/v1/organizations/f51dada7-4f11-4b6a-bbbd-2cf6a3673035/event-specs/v1/d1336abc-1b60-46f7-be2d-2105f2daf283"
      }
  ]
  ...
]
```

Under the json path `includes.tracking_scenarios`, the API will also attach associated event specifications in their entirety:

```json
"includes": {
  ...
  "eventSpecs": [
    "id": "d1336abc-1b60-46f7-be2d-2105f2daf283",
     ...
  ]
  ...
}
```

### Retrieve History Information for a Tracking Plan

If you wish to retrieve the change log of a specific tracking plan, you can use the following GET request:

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}/history`

You can pass several parameters to control the result of the response:

- **before**: returns records equal or less than the timestamp in the ISO-8601 format
- **limit**: limits the number of records
- **offset**: skip the first N results
- **order**: order of returned records, `asc` or `desc`. Defaults to `desc`.

Path parameter `organizationId` is required.

## Creating and updating Tracking Plans

### Creating a Tracking Plan

This `POST` request allows you to create a new tracking plan within an organization.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2`

The request body is mandatory and should be in JSON format. The minimum payload would be a JSON with only the `name` of the tracking plan. The remaining fields are optional and not required on creation. Example:

```json
{
  "name": "Performance tracking",
  "description": "Tracks performance",
  "domain": "Marketing",
  "owner": "IT department",
  "accessInstructions": "The data can be accessed in the warehouse, in the atomic.events table"
}
```

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

### Updating a Tracking Plan

Use this request to update a tracking plan. The `dataProductId` is required, along with a valid request body.

The minimum payload on update would be the same as on creation but with the addition of the required `status` field. On creation, by default, it will set the `status` to `draft`.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

See the [detailed API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs) for all options.

### Delete a Tracking Plan

Use this request to delete a tracking plan. The `dataProductId` and `organizationId` are both required.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v2/{dataProductId}`

## Subscription Management for Tracking Plans

### Retrieve All Subscriptions for a Tracking Plan

To retrieve all subscriptions for a tracking plan, use the following request. The `organizationId` and `dataProductId` are required.

`**GET** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions`

### Add a Subscription

To add a subscription for a tracking plan, use the following request. The `organizationId`, `dataProductId` and a valid request body are required.

`**POST** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions`

The following is the minimum accepted payload. It will create a subscription for the user who issues the request, as inferred by the JWT in the request headers.

```json
{
  "reason": "Get notified on breaking changes",
  "receiveNotifications": true
}
```

If you want to subscribe a different user you will need to populate an additional field, `recipient`, with that user's email address.

When a subscription is created, it will send a confirmation email to the recipient (default user or third user). Clicking the confirmation link in that email will direct the recipient to the following URL and mark the subscription as confirmed:

`**POST** /organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}/actions/confirm`

Once a subscription is created and the email has been confirmed, the subscriber will start receiving a daily email digest referencing all the tracking plans that had changes in the last 24 hours.

### Update a Subscription

To update a subscription for a specific tracking plan, use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId`, and a valid request body are required.

`**PUT** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}`

### Delete a Subscription

To delete a subscription for a specific tracking plan (unsubscribe action), use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId`, and a valid request body are required.

`**DELETE** ​/api​/msc​/v1​/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}`

### Resend a Subscription Confirmation Email

To resend a subscription confirmation email, use the following request. Path parameters `organizationId`, `subscriptionId`, `dataProductId` are required.

`**POST** ​/api​/msc​/v1/organizations/{organizationId}/data-products/v1/{dataProductId}/subscriptions/{subscriptionId}/actions/resend-confirmation`

### Integration with the SDK Generator

To send emails with instructions for the SDK generator, use the following request. Path parameters `organizationId` and `dataProductId` and a valid request body are required.

`**POST** /organizations/{organizationId}/data-products/v2/{dataProductId}/share-instructions`

---

# Query your data
> Use example SQL queries from Console to retrieve and analyze your event data in your data warehouse.
> Source: https://docs.snowplow.io/docs/event-studio/query-sql/

When viewing an [event specification](/docs/event-studio/tracking-plans/event-specifications/) in Console, the **Working with this event** section provides ready-to-use code snippets.

The **Querying** tab provides example queries to help you retrieve and analyze your event data. Choose your warehouse to see appropriately optimized SQL.

![Querying SQL examples](/assets/images/sql-example-d64be1abac9a42e1162f0ed311e2db80.png)

---

# Organize data sources with source applications
> Document and manage your tracking implementation across different applications with source applications in Event Studio.
> Source: https://docs.snowplow.io/docs/event-studio/source-applications/

For data collection, you will often have different sources of information that correspond to applications designed for a particular purpose. These are what we refer to as source applications.

> **Tip:** A guideline, which will address your needs most of the time, is to think of a source application as an independently deployable application system. For example:
>
> - An Android mobile application
> - An iOS mobile application
> - A web application
>
> This will let you best manage changes you make to the available application entities and make sure it reflects as closely as possible the current data state.

To illustrate, consider Snowplow. We can identify several applications designed for distinct purposes, each serving as a separate data source for behavioral data, or in other words, a source application:

- The Snowplow website that corresponds to the application served under `www.snowplow.io`
- The Console application that is served under `console.snowplowanalytics.com`
- The documentation website serving as our information hub for all things related to our product, served under `docs.snowplow.io`

Source applications are a foundational component that enables you to establish the overarching relationships that connect application IDs and [application entities](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) and [tracking plans](/docs/event-studio/tracking-plans/).

## Application IDs

Each source application should have a unique application ID to distinguish it in analysis. For each of these applications you would set up a unique application ID using the [`app_id`](/docs/events/ootb-data/app-information/#application-atomic-event-properties) field to distinguish them later on in analysis.

> **Tip:** We often see, and recommend as a best practice, setting up a unique application ID for each deployment environment you are using. For example, `${appId}-qa` for staging and `${appId}-dev` for development environments.

## Application entities

Application entities, also referred to as [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/), are a set of entities that can be sent with every event recorded in the application. Using source applications you can document which application entities are expected. This is useful for tracking implementation, data discovery, and preventing information duplication in tracking plans.

> **Info:** Since application entities can also be set conditionally, you can mark any of them as optional with a note to better understand the condition or any extra information required. The method for conditionally adding an application entity is through [rulesets](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#rulesets), [filter functions](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#filter-functions) and [context generators](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#context-generators).

## Initialize tracking with the tracker configuration

Once you have created a source application, you can use the **Set up tracking** tab to configure tracking and generate ready-to-use code snippets. This guided configuration simplifies the instrumentation process and helps you start receiving events.

> **Info:** The Set up tracking tab currently supports the JavaScript tracker. You can always customize your tracking further by referring to the [tracker documentation](/docs/sources/).

### Configure tracker settings

The Set up tracking tab provides a visual interface to configure your tracker and generates code snippets based on your selections.

#### Initialize tracker

Configure the basic tracker settings:

- **Collector URL**: the endpoint where your events will be sent
- **App ID**: select one of the application IDs associated with your source application

![A screenshot from Console showing the initialize tracker section, with dropdowns for Collector URL and App ID](/assets/images/initialize-tracker-96764a0c520edf581afece137c0282dd.png)

#### Automatic tracking

Enable out-of-the-box tracking features to capture common user interactions without additional code:

- **Page views**: automatically track when pages are viewed
- **Link clicks**: capture clicks on links
- **Form interactions**: track form submissions and field interactions
- **Page pings**: monitor user engagement with periodic activity pings

Toggle these features based on your tracking requirements.

![A screenshot from Console showing automatic tracking configuration](/assets/images/automatic-tracking-8a40a05374e35316c3710f66a6ca9759.png)

#### Implementation

The code snippet at the bottom of the Set up tracking tab updates in real-time as you modify settings. Copy the code snippet and integrate it into your application to begin tracking.

Choose your implementation method:

**JavaScript (tag):**

Add a `<script>` tag to your website.

The SDK URL is pre-filled with the latest version from a CDN but can be edited if you are hosting on your own domain.

![JavaScript tag implementation showing an editable SDK URL field and a generated script tag code snippet](/assets/images/tag-implementation-f6870244702c1b66b03cedc8d63f4d06.png)

**Browser (npm):**

Install the tracker as an npm package for bundled applications and any required plugins using your preferred package manager.

![Browser npm implementation showing package installation commands and import code snippet](/assets/images/npm-implementation-a96a56c0913305c882771df8a58b5f8b.png)

***

---

# Managing tracking plans using Console
> Create and manage tracking plans in Snowplow Console with templates, event specifications, source applications, and automatic code generation for tracking implementation.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/create-and-manage/

To create a new tracking plan, navigate to the "Tracking plans" section from the navigation bar and click the "Create tracking plan" button.

![Create tracking plan](/assets/images/create-tracking-plan-v2-7dd09e1670e4fadd80c1d27912b423a4.png)

A modal will appear on the page, giving you the possibility to quickly create a tracking plan by using one of the existing templates or create one from scratch.

![Create tracking plan modal](/assets/images/create-tracking-plan-modal-effaa5d8fbffc01a04a09662fc6a9a46.png)

After selecting "Create new" a form will appear on the page. Enter your tracking plan information and click "Create and continue" to navigate to the event specification page.

> **Note:** The name of your tracking plan must be unique to ensure proper identification and avoid conflicts.

The next page allows you to create multiple event specifications. You can click on any row to enter the details on this screen, or you can complete the information later.

When clicking on an event specification row, a page will allow you to enter additional information into separate modals:

- **Event information**; describes information such as which applications the event fires in
- **Event data structure**; determines how your data is structured and define the types of properties of the event
- **Entity data structures**; describes which entities to attach to the event when it is triggered
- **Event triggers**; specify the places and circumstances under which the event is triggered
- **Properties**; specify instructions about how each schema property should be set for this event specification

The breadcrumb navigation allows you to quickly navigate to the tracking plan overview as well as to the list of tracking plans. Alternatively, you can access the list of available tracking plans by clicking `Tracking plans` prominently displayed in the navigation bar on the left.

In the image below, you can see an example of a tracking plan. It not only provides an overview of all the event specifications but also allows you to access three important pieces of functionality.

- **Share**; allow other members of your organization to access the tracking plan
- **Subscribe**; receive notifications of any changes in the tracking plan
- **Implement tracking**; automatically generate the code for your tracking plan to be included in your application (to learn more visit [Code Generation - automatically generate code for Snowplow tracking SDKs](/docs/event-studio/implement-tracking/snowtype/))

> **Note:** Sharing and subscribing is only available for users registered in Snowplow Console.

![Tracking plan overview](/assets/images/tracking-plan-overview-0d2201d440d341fe7fff177dd83fef8d.png)

![Event specification details](/assets/images/event-specification-details-1c54133bbc47f99ca82386d40f3f6284.png)

If you need to edit a tracking plan at any time, select it from the tracking plans listing accessible from the main menu.

## Upgrading Event Specification Instructions

When working with Event Specifications of a Tracking Plan, it's essential to account for the evolution of underlying [Data Structures](/docs/event-studio/data-structures/). Data Structures define reusable JSON schemas, which can be referenced by different Event Specifications (events and entities). Each Event Specification may contain instructions, which rely on a specific version of a Data Structure, adding another layer to specialize or constraint Event Specifications in a more granular way.

### Versioning of Data Structures

As data and events evolve, Data Structures may be updated to new versions, which can be either compatible or incompatible with previous ones. These changes may cause potential conflicts with the instructions in Event Specifications (both events and entities) that reference an older version of the Data Structure.

### Semi-Automatic Upgrade of Event Specifications via the UI

To streamline the process of upgrading an Event Specification to the latest version of a Data Structure, we've implemented a mechanism that allows you to update Event Specification instructions through the UI. Here's how it works:

When a new version of a Data Structure becomes available, the system will indicate that the event or entities referenced by the data structure has a new version available, showing an **'Upgrade'** button in the UI.

![Upgrade Event Specification warning](/assets/images/upgrade-event-specification-warning-c04429de0f17e3b4ddd05404d6d3b691.png)

Clicking the button navigates to a new page, informing the user of the new version they are upgrading to, along with a **'View Changes'** button.

![Upgrade Event Specification page](/assets/images/upgrade-event-specification-page-194f756d9e7d4eef25df6d940b4949bc.png)

When clicked it will show the differences between the current version of the Data Structure and the one the user intends to upgrade to.

![Upgrade Event Specification diff](/assets/images/upgrade-event-specification-diff-7cb68782706e2707a441dbb890508b3a.png)

At the bottom, a button will allow users to confirm the upgrade. One of two things can happen when the upgrade is confirmed:

#### 1. Successful automatic upgrade

- If the Event Specification instructions are compatible with the new Data Structure version, the system will automatically upgrade the Event Specification to the latest version of the Data Structure.
- All instructions will be updated seamlessly without any further user intervention.

![Automatic upgrade Event Specification](/assets/images/success_upgrade-ac3deb29a3bcf0154246aaf4d2416f03.png)

#### 2. Conflict detection and resolution

If the new version of the Data Structure introduces incompatibilities with the existing Event Specification instructions, the system will flag the conflicting properties.

- The UI will prompt the user to resolve these conflicts before the Event Specification can be upgraded.

- The conflict resolution UI provides options to the user to modify or delete each instruction depending on the type of incompatibility:

  - **Remove conflicting instructions**: If a specific property is no longer present in the new Data Structure.
  - **Modify conflicting instructions**: If a property in the new Data Structure has been changed in an incompatible way (e.g., type change, added/removed enum values, added pattern, etc.).

![Conflict resolution Event Specification](/assets/images/conflict_resolution-1eaa5c52b9eb95d0feb3f26d2d42e5f0.png)

This mechanism ensures that teams can benefit from updated Data Structures while maintaining the integrity and accuracy of their Event Specifications. Users are empowered to make informed decisions during the upgrade process, with clear visual cues and options to handle conflicts effectively.

---

# Event specification inference
> Event specification inference enables the Snowplow pipeline to automatically match incoming events to your published event specifications, surfacing business context and volume metrics without changes to your tracking implementation.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/event-specification-inference/

Event specification inference allows the Snowplow pipeline to automatically match incoming events against your published [event specifications](/docs/event-studio/tracking-plans/event-specifications/). Once a specification is published, the pipeline can attach business context to matching events and surface volume and recency data in the Console — without any changes to your tracking implementation.

## Understand event specification statuses

Each event specification uses an explicit publishing model, replacing the previous "Live" status that the Console assigned automatically when it first observed matching events. Each specification has one of three statuses:

- **Draft**: the specification is being edited and is not yet active in the pipeline. The pipeline does not match events against it, and no inference occurs.
- **Publishing**: a transitional state, lasting a few minutes, while the pipeline propagates the specification. You do not need to take any action during this phase.
- **Published**: the specification is active. The pipeline matches incoming events against it, attaches an `event_specification` entity to it, and surfaces volume data and "last seen" timestamps in the Console.

The tracking plan list view reflects the status of the specifications it contains. A tracking plan shows **Published** if all of its specifications are published, and **With Drafts** if any specification remains in draft.

## Publish an event specification

Publishing makes a specification active in the pipeline and enables inference. You can trigger publishing from two places:

- From the tracking plan view: click the **Publish All** button above the list of specifications. This publishes all specifications in the plan that are currently in Draft status.
- From within an individual event specification: click the **Publish** button at the top of the specification page.

The publishing flow also checks that any [data structures](/docs/fundamentals/schemas/) your specification references are available in the production pipeline before activating it.

### Synchronize data structures during publishing

Before publishing completes, the Console checks whether all [data structures](/docs/fundamentals/schemas/) referenced by the specification — either the event data structure or any entity data structures — have been promoted to production. If any have not, the publishing flow identifies them and requires you to promote them in the same step. This ensures the pipeline can perform inference correctly as soon as the specification reaches Published status.

## Understand how inference works

Once a specification is published, the pipeline evaluates every eligible incoming event against it. Events sent using Snowtype's [automatic tracking features](/docs/event-studio/implement-tracking/snowtype/) already contain an `event_specification` entity, so the pipeline bypasses inference for them entirely. All other events are evaluated using the following criteria, in order:

1. **Event schema**: the event's schema must match the event data structure defined in the specification.
2. **Entity set**: the event must carry all [entities](/docs/fundamentals/entities/) listed in the specification according to the cardinality rules. Extra entities with different schemas on the event are ignored.
3. **Property rules**: any property-level rules defined on those data structures in the specification — for example, `category = "product"` — must be satisfied.

The pipeline does not match on `appId`, environment, or any other source attribute. A single incoming event can match more than one specification if multiple published specifications share overlapping definitions.

When a match occurs, the pipeline:

- Attaches an [entity](/docs/fundamentals/entities/) to the event containing the specification name and ID, making the business context available for downstream consumers.
- Records the event against the specification in the Console, updating the total volume count and "last seen" timestamp.

### Example

Consider a specification named "Product page view" that defines:

- **Event**: `page_view` (`iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0`)
- **Entity**: `product` (`iglu:com.acme/product/jsonschema/1-0-0`) with a rule `category = "electronics"`

An incoming `page_view` event carrying a `product` entity where `category = "electronics"` matches the specification. The pipeline attaches an `event_specification` entity to the event and increments the specification's volume count in the Console.

The same `page_view` event carrying a `product` entity where `category = "clothing"` does not match, because the entity rule is not satisfied. A `page_view` event with no `product` entity also does not match, because the required entity is absent.

> **Tip:** You do not need to change your tracking implementation to benefit from inference. Events already flowing through your pipeline will be matched against newly published specifications automatically.

---

# Manage event specifications using Console
> Create and manage event specifications in Snowplow Console UI as documented events within tracking plans with data structures and implementation details.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/event-specifications/

Event specifications are complete definitions of business events. They define what data should be tracked, when the event occurs, along with additional metadata about the event such as ownership.

They act as data contracts between teams. When you create an event specification, you are defining exactly what data your applications should send, what your data warehouse will receive, and what your downstream consumers can rely on.

> **Note:** You can only make event specifications within tracking plans. Tracking plans are wrappers for related event specifications. Each event specification belongs to a single tracking plan.

Once an event specification is ready, you can publish it to make it active in the pipeline. Publishing enables [event specification inference](/docs/event-studio/tracking-plans/event-specification-inference/), which automatically matches incoming events to your published specifications and surfaces volume and observability data in the Console.

Each event specification has an event [data structure](/docs/event-studio/data-structures/), plus optional entity data structures. You can add instructions for how to populate each property, when to trigger the event, and how many of each entity to expect. You can also select which version of each data structure to use.

## Create an event specification in Console

To create a new event specification, follow these steps:

1. Select a tracking plan
2. Click the **Create event** button
3. A dialog will appear, prompting you to enter a name for your event specification and click **Save and continue**
4. Your first event specification will be displayed on the page

![Create an event specification](/assets/images/create-event-specification-1fb10126179bd86b5036a979b2d44b39.png)

To add more information or modify an existing event specification, follow these steps:

1. Navigate to the appropriate tracking plan
2. Select the desired event specification
3. This action will open an overview of the selected event specification containing the details that have been added to date

This interface is divided into focused sections; explore each section below for more details.

![Example of an event specification overview](/assets/images/event-specification-overview-f3c4beeb305a7329dbc7ed8376a57f0b.png)

### Event information

This section provides essential meta-information for your event specification, including the event name, description, and the applications in which this event is tracked.

### Event data structure

This section defines the event data structure that this event will validate against as it is processed by your pipeline.

You can choose from two types of data structures:

- **Standard**: provided by the Snowplow tracker by default
- **Custom**: provided by your organization

![Event data structure](/assets/images/event-data-structure-00f3d61c035b8d6e2261531677f57083.png)

### Entity data structures

Within this section, you have the flexibility to define the entities that should be associated with the event when it is triggered.

To facilitate making an informed selection of your entities, you can view a detailed breakdown of the properties associated with the chosen entities (and its version) by a specific version.

![Entity data structures](/assets/images/entity-data-structures-a2728bea5606b9e3aa9eb473ab676c56.png)

### Event triggers

This section defines the locations and circumstances under which this event is triggered.

To create your first trigger, simply select the **Add trigger** button. You can edit or delete an existing trigger by clicking the dedicated buttons beside each entry in the triggers list.

A dialog will appear, allowing you to upload an image and provide additional context, such as the URL to the page on which this trigger applies.

![Event triggers](/assets/images/event-triggers-f58df8f12d6e3816cbe742f2b7b523c8.png)

### Properties

This section allows you to specify how each property for a selected event or entity data structure should be populated.

The dialog displays the list of properties for the selected data structure will be listed here, with the ability to provide the exact value/s or a description of how to populate these properties when the event is triggered.

You can configure detailed instructions for any of the properties shown in the list by clicking the **Add instruction** or **Edit** buttons. Once you have selected the type of instruction you wish to add/edit and have filled in the required input fields, you are then able to click **Save and update instruction** and return to the properties list.

> **Note:** Instructions for required properties are added by default and can be edited but they cannot be deleted.

![Implementation instructions](/assets/images/implementation-instructions-51f52ed6ae5773cb275f0af83a076447.png)

---

# Defining the data to collect with Tracking Plans
> Create logical groupings of behavioral data by domain with explicit ownership, event specifications, and source applications to improve governance and collaboration across your organization.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/

As described in [Tracking Plans Introduction](/docs/fundamentals/tracking-plans/), a tracking plan is a logical grouping of the data you collect as an organization by domain, with an explicit owner.

With tracking plans, you can:

- Set clear ownership for the data being created
- Make tracking implementation easier
- Deliver better governance around your data
- More easily communicate what the data means and how to use it
- Collaborate more effectively with the various teams involved in delivering value from your data
- Drive a self-serve culture around data across your organization
- See event volume metrics helping to monitor data collection over time.

Each tracking plan contains event specifications that define the events you want to track. By grouping related event specifications together in tracking plans, you can better organize your data collection efforts by domain, with clear ownership and implementation instructions for each event specification.

Event specifications bridge the gap between tracking design and data collection:

- **Design phase**: you document your tracking requirements by creating event specifications that capture both technical structure and business context
- **Implementation phase**: developers use these specifications to instrument tracking code, either manually or through code generation within the Snowplow Console or using tools like Snowtype. Snowtype generated code ensures type-safety and alignment with specifications, reducing implementation errors and accelerating development time
- **Observability phase**: monitor event specification usage directly in the Console. See the total number of events collected for each specification and when each was last seen. This visibility helps you confirm implementations are live, identify unused specifications, and understand event volume patterns across your tracking plan
- **Data modeling phase**: event specifications enable automatically generated dbt models that transform atomic events into analysis-ready tables. These models understand the structure defined in your specifications, creating consistent table schemas and joining related [entities](/docs/fundamentals/entities/). As you update specifications, corresponding data models can be regenerated, keeping your warehouse transformations synchronized with your tracking design

## Elements of a Tracking Plan

**Tracking plan**

- **Name;** a descriptive and unique name for the tracking plan

- **Description;** a description of the data that the tracking plan captures

- **Owner;** the individual responsible for the tracking plan

- **Domain;** the team or business domain that owns the tracking plan

- **Source Application;** the [source application/s](/docs/event-studio/source-applications/) the Tracking Plan is implemented in

- **Event specifications**

  - **Name;** a descriptive name for the event
  - **Description;** a description to help people understand what action the event is capturing
  - **Source Applications;** the source application(s) inherited from the tracking plan containing the Application ID(s) that will be sent with this event
  - **Triggers;** specific instructions on where the event gets triggered (e.g. when a user clicks the "Add to basket" button)
  - **Event data structure;** the event data structure that this event will validate against as it is processed by your pipeline
  - **Entities;** the entities that should be attached to this event (e.g. user, product)
  - **Properties;** any specific rules for each property of the event

---

# Tracking plan templates
> Pre-defined tracking plan templates for Base Web, Base Mobile, Ecommerce, and Media tracking with included event specifications and implementation guidance.
> Source: https://docs.snowplow.io/docs/event-studio/tracking-plans/templates/

To create a new tracking plan based on a pre-defined template, navigate to the "Tracking plans" section from the navigation bar and click the "Create tracking plan" or "Templates" buttons.

> **Note:** _When creating a tracking plan based on a template, the name must be unique to ensure proper identification and avoid conflicts._

Below is a list of the current templates:

## Base Web

This tracking plan template contains all the tracking of Standard Events related to web tracking. It is expected that these events would be implemented using our [Web trackers](/docs/sources/web-trackers/) and the respective [button click](/docs/sources/web-trackers/tracking-events/button-click/) and [link click](/docs/sources/web-trackers/tracking-events/link-click/) plugins.

Note: The event volume counts are calculated differently for this Tracking Plan. They are counts of any of the following events sent for the app ID's of this tracking plan.

Included event specifications:

- [Page View](/docs/sources/web-trackers/tracking-events/page-views/)
- [Page Ping](/docs/sources/web-trackers/tracking-events/activity-page-pings/)
- [Button click](/docs/sources/web-trackers/tracking-events/button-click/)
- [Link click](/docs/sources/web-trackers/tracking-events/link-click/)

## Base Mobile

This tracking plan template contains all the tracking of Standard Events related to mobile tracking. It is expected that these events would be implemented using our [mobile trackers](/docs/sources/mobile-trackers/) with [installation](/docs/sources/mobile-trackers/tracking-events/installation-tracking/), [screen view](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) and [app lifecycle](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/) tracking capabilities enabled.

Note: The event volume counts are calculated differently for this Tracking Plan. They are counts of any of the following events sent for the app ID's of this tracking plan.

Included event specifications:

- [Screen view](/docs/events/ootb-data/page-and-screen-view-events/#screen-views)
- [Screen end](/docs/events/ootb-data/page-activity-tracking/#screen-end-event)
- [Application install](/docs/events/ootb-data/mobile-lifecycle-events/#install-events)
- [Application background](/docs/events/ootb-data/mobile-lifecycle-events/#background-event)
- [Application foreground](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event)

## Ecommerce (Web and Mobile)

This Tracking Plan template contains all the basic ecommerce interactions that help you analyze customer behavior, identify potential growth opportunities, and improve your sales performance.

Tracked using the [out-of-the-box e-commerce events](/docs/events/ootb-data/ecommerce-events/) for web, Android, and iOS. On web, this requires the [Snowplow Ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/).

Included event specifications:

- Add to cart
- Checkout step
- Internal promotion click
- Internal promotion view
- Product list click
- Product list view
- Refund
- Remove from cart
- Track product view
- Transaction
- Transaction Error

## Media Web

This tracking plan template contains a full set of media tracking interactions to measure video or audio data. Tracked using the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/).

Included event specifications:

- Ad Break End Event
- Ad Break Start Event
- Ad Click Event
- Ad Complete Event
- Ad Pause Event
- Ad Quartile Event
- Ad Resume Event
- Ad Skip Event
- Ad Start Event
- Buffer End Event
- Buffer Start Event
- Error Event
- Fullscreen Change Event
- Media Ping Event
- Pause Event
- Percent Progress Event
- Picture-in-picture Change Event
- Playback End Event
- Play Event
- Quality Change Event
- Ready Event
- Seek End Event
- Seek Start Event
- Volume Change Event

---

# Anonymous tracking
> Track events without collecting user identifiers, IP addresses, or cookies to support privacy compliance and user consent requirements.
> Source: https://docs.snowplow.io/docs/events/anonymous-tracking/

Snowplow enables you to track events without collecting personally identifiable information (PII). Use anonymous tracking to respect user privacy preferences, comply with regulations such as GDPR, or track events before a user has provided consent.

Every Snowplow event includes a number of identifiers. Read more about them in the [overview](/docs/events/identifiers/) and [out-of-the-box tracking](/docs/events/ootb-data/user-and-session-identification/) pages. You can configure your trackers to prevent the collection of these identifiers, as well as IP addresses and cookies, using anonymous tracking.

Some Snowplow trackers also provide options for tracking [user consent](/docs/events/ootb-data/consent-events/) interactions and GDPR basis for processing.

## Anonymization levels

Snowplow trackers support multiple levels of anonymization to balance privacy requirements with analytics needs. Combine client-side and server-side anonymization to achieve the desired level of privacy.

As well as anonymous tracking, you can also use enrichments to pseudonymize or mask identifiers for tracked events, before they arrive at the data warehouse. This uses the [PII pseudonymization](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) or [IP anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) enrichments.

For more details on web cookie behavior, see [Cookies and local storage](/docs/sources/web-trackers/cookies-and-local-storage/).

> **Note:** Some tracker APIs use the British English spelling: "anonymisation" rather than "anonymization".

### Full client-side anonymization

When configured for full client-side anonymization, the tracker does not track any client-side user or session identifiers. Events can still be collected, but can't be grouped by user or session.

Here's some example code for enabling full client-side anonymization when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  // No user or session identifiers will be added to events
  // Network identifiers will be tracked
  anonymousTracking: true,
});
```

By default, the tracker uses [cookies and local storage](/docs/sources/web-trackers/cookies-and-local-storage/) to persist user and session information, as well as for buffering events. When anonymous tracking is enabled, identifiers are not saved to cookies or local storage.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {
  TrackerConfiguration()
    .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked
    .sessionContext(false) // Session entity won't be added to events
    // Network identifiers will be tracked
}
```

**Android:**

```kotlin
createTracker(
    applicationContext,
    "appTracker",
    NetworkConfiguration("https://snowplow-collector-url.com"),
    TrackerConfiguration("appId")
        .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked
        .sessionContext(false) // Session entity won't be added to events
        // Network identifiers will be tracked
)
```

***

### Client-side anonymization with session tracking

In this mode, the tracker doesn't include user identifiers but preserves [session tracking](/docs/events/ootb-data/user-and-session-identification/). You can analyze session-level behavior without identifying individual users.

Here's some example code for enabling client-side anonymization with session tracking when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  // User identifiers won't be added to events
  // Session and network identifiers will be tracked
  anonymousTracking: { withSessionTracking: true },
});
```

The tracker uses cookies and local storage by default. With this configuration it will store session identifiers, but not user identifiers.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {
  TrackerConfiguration()
    .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked
    .sessionContext(true) // Session entity will be added to events but with userId as a null UUID
    // Network identifiers will be tracked
}
```

**Android:**

```kotlin
createTracker(
    applicationContext,
    "appTracker",
    NetworkConfiguration("https://snowplow-collector-url.com"),
    TrackerConfiguration("appId")
        .userAnonymisation(true) // User identifiers in mobile entity (IDFA and IDFV) will not be tracked
        .sessionContext(true) // Session entity will be added to events but with userId as a null UUID
        // Network identifiers will be tracked
)
```

***

### Server-side anonymization

Server-side anonymization prevents the event [Collector](/docs/pipeline/collector/) setting the `sp` cookie for `network_userid`, or from capturing IP addresses.

Setting server-side anonymization will add a `SP-Anonymous` HTTP header to requests sent to the Collector. This requires the request method to be `POST`, which is the default for most trackers. When the `SP-Anonymous` header is present, the Collector doesn't set or read the `sp` cookie.

An alternative method for preventing IP address tracking is to set a null value, such as `0.0.0.0` within the tracker. Most Snowplow trackers [support this option](/docs/events/ootb-data/user-and-session-identification/), although not the JavaScript trackers.

In the [JavaScript trackers](/docs/sources/web-trackers/anonymous-tracking/), setting server-side anonymization also sets full client-side anonymization. In the [native mobile trackers](/docs/sources/mobile-trackers/anonymous-tracking/), it's a separate configuration. You can choose either client-side anonymization, server-side anonymization, or both.

Here's some example code for enabling server-side anonymization when creating a tracker on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  // No user, session, or network identifiers will be tracked
  anonymousTracking: { withServerAnonymisation: true },
});
```

For fully [cookieless](/docs/sources/web-trackers/cookies-and-local-storage/) web tracking, configure the tracker with:

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  // No user, session, or network identifiers will be tracked
  // No data will be stored in the browser
  anonymousTracking: { withServerAnonymisation: true },
  stateStorageStrategy: 'none',
});
```

This configuration prevents the tracker from storing any data in cookies or local storage.

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {
  // User and session identifiers will be tracked
  // Network identifiers won't be added to events
  EmitterConfiguration().serverAnonymisation(true)
}
```

**Android:**

```kotlin
createTracker(
    applicationContext,
    "appTracker",
    NetworkConfiguration("https://snowplow-collector-url.com"),
    // User and session identifiers will be tracked
    // Network identifiers won't be added to events
    EmitterConfiguration().serverAnonymisation(true)
)
```

***

## Tracker support

This table shows the support for anonymous tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                                    | Supported | Since version | Client-side | Server-side | Notes                                                         |
| ------------------------------------------------------------------------------------------ | --------- | ------------- | ----------- | ----------- | ------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/anonymous-tracking/)                                      | ✅         | 3.0.0         | ✅           | ✅           |                                                               |
| [iOS](/docs/sources/mobile-trackers/anonymous-tracking/)                                   | ✅         | 4.0.0         | ✅           | ✅           |                                                               |
| [Android](/docs/sources/mobile-trackers/anonymous-tracking/)                               | ✅         | 4.0.0         | ✅           | ✅           |                                                               |
| [React Native](/docs/sources/react-native-tracker/anonymous-tracking/)                     | ❌/✅       | 1.3.0         | ❌/✅         | ✅           | Version 4.x only supports server-side anonymization           |
| [Flutter](/docs/sources/flutter-tracker/anonymous-tracking/)                               | ✅         | 0.3.0         | ✅           | ✅           |                                                               |
| [Roku](/docs/sources/roku-tracker/adding-data/#adding-user-and-platform-data-with-subject) | ✅         | 0.3.0         | ✅           | ✅           | Use Subject configuration to manage client-side anonymization |
| [Node.js](/docs/sources/node-js-tracker/initialization/)                                   | ✅         | 3.21.0        | ❌           | ✅           |                                                               |
| Golang                                                                                     | ❌         |               |             |             |                                                               |
| .NET                                                                                       | ❌         |               |             |             |                                                               |
| Java                                                                                       | ❌         |               |             |             |                                                               |
| Python                                                                                     | ❌         |               |             |             |                                                               |
| Scala                                                                                      | ❌         |               |             |             |                                                               |
| Ruby                                                                                       | ❌         |               |             |             |                                                               |
| Rust                                                                                       | ❌         |               |             |             |                                                               |
| [PHP](/docs/sources/php-tracker/emitters/)                                                 | ✅         | 0.9.0         | ❌           | ✅           |                                                               |
| C++                                                                                        | ❌         |               |             |             |                                                               |
| Unity                                                                                      | ❌         |               |             |             |                                                               |
| Lua                                                                                        | ❌         |               |             |             |                                                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)                  | ✅         | v4            | ✅           | ✅           |                                                               |

### Toggle anonymous tracking

As well as setting it at tracker initialization, you can enable or disable anonymous tracking during a session. Use this to start with anonymous tracking until a user accepts a cookie banner, or to enable full tracking when a user logs in.

Here's some example code for toggling anonymization at runtime on [web](/docs/sources/web-trackers/anonymous-tracking/) or [mobile](/docs/sources/mobile-trackers/anonymous-tracking/):

**JavaScript:**

```javascript
snowplow('enableAnonymousTracking');
snowplow('disableAnonymousTracking');
```

These methods also accept an [optional configuration object](/docs/sources/web-trackers/anonymous-tracking/#toggling-anonymous-tracking) to specify which levels of anonymization to enable.

**iOS:**

```swift
tracker.userAnonymisation = true
tracker.sessionContext = true
tracker.emitter?.serverAnonymisation = true
```

**Android:**

```kotlin
tracker.userAnonymisation = true
tracker.sessionContext = true
tracker.emitter.serverAnonymisation = true
```

***

On web, enabling anonymous tracking removes user identifiers from outgoing events but doesn't delete existing cookies. On mobile, toggling anonymization starts a new session.

### Clear user data

When enabling anonymous tracking, you may want to clear any existing user identifiers stored in cookies or local storage.

Use the `clearUserData()` method in the JavaScript trackers to delete all data stored in the `_sp_id` and `_sp_ses` cookies. It takes an [optional configuration object](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) to specify which identifiers to clear. To delete the `network_userid` in the Collector `sp` cookie, set server-side anonymization.

There's no equivalent API for deleting stored data on mobile.

## Anonymizable event properties

Use anonymous tracking to avoid collecting the following identifiers in your events. The table shows how you can anonymize each identifier using client-side anonymization, server-side anonymization, or enrichments.

> **Note:** Snowplow events have two properties with similar names: the `user_id` atomic event property and the `userId` property in the session entity.
>
> The former is a business user identifier you set via the tracker API, while the latter is a device ID generated by the tracker. Both can be anonymized using client-side anonymization.

| Property                   | Location in event                                                   | Identifier type | Description                                                 | Client-side anon | Server-side anon | Enrichment                                                                                                                                                                                    |
| -------------------------- | ------------------------------------------------------------------- | --------------- | ----------------------------------------------------------- | ---------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `domain_userid`            | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | User            | UUID stored in a first-party cookie                         | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `domain_sessionid`         | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Session         | UUID for the current session                                | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `domain_sessionidx`        | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Session         | Count of sessions for this user                             | ✅                |                  |                                                                                                                                                                                               |
| `network_userid`           | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Network         | UUID set by the Collector in a server-side cookie           |                  | ✅                | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `user_ipaddress`           | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | Network         | IP address captured by the Collector                        |                  | ✅                | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/), [IP anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) |
| `user_id` in atomic fields | [Atomic](/docs/fundamentals/canonical-event/#user-fields)           | User            | Business user identifier set via `setUserId` or similar API | ✅                |                  | ✅ [PII](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)                                                                                                    |
| `userId` in session entity | [Entity](/docs/events/ootb-data/user-and-session-identification/)   | User            | Device ID in the client session entity                      | ✅                |                  |                                                                                                                                                                                               |
| `sessionId`                | [Entity](/docs/events/ootb-data/user-and-session-identification/)   | Session         | UUID for the session                                        | ✅                |                  |                                                                                                                                                                                               |
| `appleIdfa`                | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Advertising identifier (IDFA) on iOS                        | ✅                |                  |                                                                                                                                                                                               |
| `appleIdfv`                | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Vendor identifier (IDFV) on iOS                             | ✅                |                  |                                                                                                                                                                                               |
| `androidIdfa`              | [Entity](/docs/events/ootb-data/device-and-browser/#mobile-entity)  | User            | Advertising identifier on Android                           | ✅                |                  |                                                                                                                                                                                               |
| `tabId`                    | [Entity](/docs/events/ootb-data/device-and-browser/#browser-entity) | User            | UUID for the browser tab                                    | ✅                |                  |                                                                                                                                                                                               |

The [PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) supports an `anonymousOnly` mode, available since Enrich version 5.3.0, that only pseudonymizes fields when the `SP-Anonymous` header is present. This lets you keep full identifiers for consented users while anonymizing non-consented users.

It can also pseudonymize fields for [structured](/docs/fundamentals/canonical-event/#structured-events) and [legacy ecommerce](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields) events, as well as these fields that are populated during enrichment:

| Property             | Location in event                                                          | Identifier type | Description                                                                                                                   |
| -------------------- | -------------------------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `refr_domain_userid` | [Atomic](/docs/fundamentals/canonical-event/#cross-domain-tracking-fields) | User            | From [cross-domain tracking](/docs/events/cross-navigation/)                                                                  |
| `ip_organization`    | [Atomic](/docs/fundamentals/canonical-event/#ip-address-fields)            | User            | Added by [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                              |
| `ip_domain`          | [Atomic](/docs/fundamentals/canonical-event/#ip-address-fields)            | User            | Added by [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                              |
| `mkt_term`           | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |
| `mkt_content`        | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |
| `mkt_clickid`        | [Atomic](/docs/fundamentals/canonical-event/#marketing-fields)             | Marketing       | Added by [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) |

---

# Cross-navigation tracking
> Configure cross-domain tracking to maintain user identity between different domains in your ecosystem
> Source: https://docs.snowplow.io/docs/events/cross-navigation/

When users navigate between different domains in your ecosystem, their user identity may get fragmented. Examples include navigation from your main website to a partner site, or to a mobile app. This creates gaps in your user journey data and makes it difficult to understand the complete customer experience across your digital properties.

> **Info:** This problem doesn't usually apply for navigation between web subdomains, for example between `www.example.com` and `blog.example.com`. This is because the web trackers have `discoverRootDomain` [enabled by default](/docs/sources/web-trackers/tracker-setup/initialization-options/).

Cross-navigation (also called cross-domain) tracking solves this problem by passing user identification data in URL parameters when users click links to other domains. This enables you to maintain user continuity across your applications.

To use cross-navigation tracking, configure the [web](/docs/sources/web-trackers/cross-domain-tracking/) or [native mobile](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) trackers to add an additional parameter, named `_sp`, to the querystring of outbound links. This process is called "link decoration".

Link decoration makes the added values visible in the `url` field of events on the destination page or URI. The querystring is added only to the events at the destination page: it doesn't persist throughout the user's session.

## Querystring properties

The `_sp` querystring parameter has two different formats: extended or short. You can also configure exactly which properties you want to include.

Available properties:

| Property         | Description                                    | Extended | Short |
| ---------------- | ---------------------------------------------- | -------- | ----- |
| `domainUserId`   | Current tracker-generated UUID user identifier | ✅        | ✅     |
| `timestamp`      | Current epoch timestamp, millisecond precision | ✅        | ✅     |
| `sessionId`      | Current session UUID identifier                | ✅        |       |
| `subjectUserId`  | Custom business user identifier                | ✅        |       |
| `sourceId`       | Application identifier                         | ✅        |       |
| `sourcePlatform` | Platform of the current device                 | ✅        |       |
| `reason`         | Custom information or identifier               | ✅        |       |

For example, the link `appSchema://path/to/page` would look like this after decoration with the full extended format:

```text
appSchema://path/to/page?_sp=domainUserId.timestamp.sessionId.subjectUserId.sourceId.sourcePlatform.reason
```

## How are the querystring parameters processed?

By default, your pipeline will process only the short format querystring `_sp={domainUserId}.{timestamp}`, even if you've decorated the links with the extended format. To process the extra properties, you'll need to configure the [cross-navigation enrichment](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/).

Both default and extended formats will populate the [atomic](/docs/fundamentals/canonical-event/) `refr_domain_userid` and `refr_dvce_tstamp` fields. The enrichment also adds a `cross_navigation` entity to the event.

| Feature                                                 | Default behavior | With cross-navigation enrichment |
| ------------------------------------------------------- | ---------------- | -------------------------------- |
| Processes short format `_sp={domainUserId}.{timestamp}` | ✅                | ✅                                |
| Processes extended format properties                    | ❌                | ✅                                |
| Populates `refr_domain_userid` field                    | ✅                | ✅                                |
| Populates `refr_dvce_tstamp` field                      | ✅                | ✅                                |
| Adds `cross_navigation` entity                          | ❌                | ✅                                |

## Tracker support

This table shows the support for cross-domain tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include link click tracking.

| Tracker                                                                                                                              | Supported | Since version |
| ------------------------------------------------------------------------------------------------------------------------------------ | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/link-click/)                                                                        | ✅         | 0.7.0         |
| [iOS](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking)     | ✅         | 6.0.0         |
| [Android](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) | ✅         | 6.0.0         |
| React Native                                                                                                                         | ❌         |               |
| Flutter                                                                                                                              | ❌         |               |

---

# How to track custom events and entities
> Track custom self-describing events and entities with JSON schemas for flexible data collection.
> Source: https://docs.snowplow.io/docs/events/custom-events/

Snowplow includes three ways to customize tracking:

- Custom [self-describing events](/docs/fundamentals/events/#self-describing-events)
- Custom [entities](/docs/fundamentals/entities/) added to out-of-the-box or custom events
- Structured events (not recommended)

Read our [tracking design best practice guide](/docs/fundamentals/tracking-design-best-practice/) to learn how to track and use custom data.

## Custom events

Self-describing [events](/docs/fundamentals/events/#self-describing-events) are [based on JSON schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

To define your own custom event, you will need to [create a corresponding schema](/docs/fundamentals/schemas/). Snowplow uses the schema to validate that the JSON containing the event properties is well-formed.

This code shows how you could track a custom `article_share` event using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
window.snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.example_company/article_share/jsonschema/1-0-0',
    data: {
      articleId: 'doc-12345',
      shareMethod: 'email',
      articleTitle: 'Getting Started with Snowplow'
    }
  }
});
```

In addition to populating the [standard atomic columns](/docs/fundamentals/canonical-event/), each type of self-describing event gets its own warehouse column (or its own table, in the case of Redshift) for event-specific fields defined in its schema. See the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page for more information.

## Custom entities

As with custom self-describing events, if you want to create your own custom [entity](/docs/fundamentals/entities/), you will need to [create a corresponding schema](/docs/fundamentals/schemas/). Snowplow uses the schema to validate that the JSON containing the entity properties is well-formed.

Here's an example that shows how you could track custom `user` and `product` entities along with a page view event, using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
// Track a page view with a custom entity
window.snowplow('trackPageView', {
  context: [
    {
      schema: 'iglu:com.example_company/user/jsonschema/1-0-0',
      data: {
        userId: 'user-12345',
        accountType: 'enterprise',
        subscriptionTier: 'premium'
      }
    },
    {
      schema: 'iglu:com.example_company/product/jsonschema/1-2-1',
      data: {
        productId: 'ASO01043',
        brand: 'ACME'
      }
    }]
});
```

See the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page to learn how entity data is structured in the data warehouse.

> **Info:** In the past, what we now call "entity" or "entities" was called "context". You'll still find `context` used in many of the existing APIs, database column names, and documentation, especially to refer to a set of multiple entities. For example, the `context` parameter in the JavaScript tracker API above is an array of entities.

### Add custom entities to all events

Certain Snowplow trackers provide the option to add custom entities to all events, or a configurable subset of events. These are called **application entities**. This feature is called "global context" in the trackers.

See the documentation for each tracker to learn how to configure it:

- [Web](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/)
- [Native mobile](/docs/sources/mobile-trackers/custom-tracking-using-schemas/global-context/) (iOS and Android)
- [React Native](/docs/sources/react-native-tracker/)
- [Scala](/docs/sources/scala-tracker/initialization/)

Use [source applications](/docs/event-studio/source-applications/) to document your expected application entities.

## Structured events

> **Info:** Structured event tracking is a legacy format used to track events that weren't natively supported by Snowplow.
>
> We recommend using self-describing events for custom event tracking.

Structured events are simpler to create than custom self-describing events, as you don't need to define a [schema](/docs/fundamentals/schemas/). However, they have a number of disadvantages:

|            | Structured events                           | Self-describing events                |
| ---------- | ------------------------------------------- | ------------------------------------- |
| Format     | ❌ Data must fit the fields below            | ✅ JSON, as complex as you want        |
| Validation | ❌ No validation (beyond field types)        | ✅ Schema includes validation criteria |
| Meaning    | ❌ Can only infer what each field represents | ✅ Schema includes field descriptions  |

Structured events have five custom event specific parameters:

| **Table column** | **Type** | **Description**                                                        | **Example values**            |
| ---------------- | -------- | ---------------------------------------------------------------------- | ----------------------------- |
| `se_category`    | text     | The category of event                                                  | `Ecomm`, `Media`              |
| `se_action`      | text     | The action / event itself                                              | `add-to-basket`, `play-video` |
| `se_label`       | text     | A label often used to refer to the 'object' the action is performed on | `dog-skateboarding-video`     |
| `se_property`    | text     | A property associated with either the action or the object             | `hd`                          |
| `se_value`       | decimal  | A value associated with the user action                                | `13.99`                       |

Here's how to track a structured event using the [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
snowplow('trackStructEvent', {
  category: 'Product',
  action: 'View',
  label: 'ASO01043',
  property: 'Dress',
  value: 49.95
});
```

In the warehouse, data tracked for any of these structured event-specific fields is stored in [standard atomic columns](/docs/fundamentals/canonical-event/#structured-events).

---

# Example tracker HTTP request headers
> Reference examples of HTTP requests to the Snowplow Collector using the Tracker Protocol with JSON payloads.
> Source: https://docs.snowplow.io/docs/events/http-requests/

This page gives examples of requests to the Snowplow [Collector](/docs/pipeline/collector/). This is a reference **for those who plan to implement their own tracker**, or want to understand the requests being sent by existing Snowplow trackers.

Below are a number of example [Tracker Protocol](/docs/fundamentals/canonical-event/) requests. All examples are POST requests with a JSON body. Requests to the Collector endpoint can contain more than one event, depending how your tracker is configured.

All requests are sent to `https://<your-collector-host>/com.snowplowanalytics.snowplow/tp2` unless you have specified a custom POST path when [configuring your Collector](/docs/pipeline/collector/).

All payloads should be wrapped in a `payload_data` self-describing JSON.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      ...
    }
  ]
}
```

> **Info:** Reserved parameter `u``u` is a reserved parameter because it is used for [click tracking in the Pixel Tracker](/docs/sources/pixel-tracker/#click-tracking).

## Self-describing event

This is an example of an HTTP request for a `viewed_product` event from web, using the recommended Base64 encoding and the `ue_px` property:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "ue",
      "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",
      "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",
      "tv": "js-3.5.0",
      "tna": "biz1",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665398554151",
      "vp": "693x1302",
      "ds": "678x9015",
      "vid": "29",
      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
      "duid": "85094061-f702-4b62-a46d-20f7226b4741",
      "url": "https://snowplow.io/",
      "stm": "1665398554153"
    }
  ]
}
```

The encoded base64 string (`ue_px`):

```text
eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==
```

represents the following JSON, when decoded:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
  "data": {
    "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",
    "data": {
      "product_id": "ASO01043",
      "price": 49.95
    }
  }
}
```

**cURL example**

```bash
curl --request POST \
    --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \
    --header 'Content-Type: application/json' \
    --data '{
    "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
    "data": [
      {
        "e": "ue",
        "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",
        "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",
        "tv": "js-3.5.0",
        "tna": "biz1",
        "aid": "website",
        "p": "web",
        "cookie": "1",
        "cs": "UTF-8",
        "lang": "en-GB",
        "res": "3440x1440",
        "cd": "30",
        "tz": "Europe/London",
        "dtm": "1665398554151",
        "vp": "693x1302",
        "ds": "678x9015",
        "vid": "29",
        "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
        "duid": "85094061-f702-4b62-a46d-20f7226b4741",
        "url": "https://snowplow.io/",
        "stm": "1665398554153"
      }
    ]
  }'
```

## Custom entities

An array of [entities](/docs/fundamentals/entities/) can be sent with each event. Entity payloads should be wrapped in a `contexts` self-describing JSON.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      ...
    },
    {
      ...
    },
  ]
}
```

This is an example of an HTTP request for the same `viewed_product` event as above but with a custom entity added, using the recommended Base64 encoding and the `cx` property:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "ue",
      "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",
      "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",
      "tv": "js-3.5.0",
      "tna": "biz1",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665398554151",
      "vp": "693x1302",
      "ds": "678x9015",
      "vid": "29",
      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
      "duid": "85094061-f702-4b62-a46d-20f7226b4741",
      "url": "https://snowplow.io/",
      "stm": "1665398554153",
      "cx": "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0="
    }
  ]
}
```

The encoded base64 string (`cx`):

```text
ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0=
```

represents the following JSON, when decoded:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.my_company/user/jsonschema/1-0-0",
      "data": {
        "my_company_user_id": "9999xyz"
      }
    }
  ]
}
```

**cURL example**

```bash
curl --request POST \
   --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \
   --header 'Content-Type: application/json' \
   --data '{
   "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
   "data": [
     {
       "e": "ue",
       "ue_px": "eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3ZpZXdlZF9wcm9kdWN0L2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InByb2R1Y3RfaWQiOiJBU08wMTA0MyIsInByaWNlIjo0OS45NX19fQ==",
       "eid": "5a305bf5-32fc-40d9-b0e8-66052f0f2c95",
       "tv": "js-3.5.0",
       "tna": "biz1",
       "aid": "website",
       "p": "web",
       "cookie": "1",
       "cs": "UTF-8",
       "lang": "en-GB",
       "res": "3440x1440",
       "cd": "30",
       "tz": "Europe/London",
       "dtm": "1665398554151",
       "vp": "693x1302",
       "ds": "678x9015",
       "vid": "29",
       "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
       "duid": "85094061-f702-4b62-a46d-20f7226b4741",
       "url": "https://snowplow.io/",
       "stm": "1665398554153",
       "cx": "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29udGV4dHMvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiBbCiAgICB7CiAgICAgICJzY2hlbWEiOiAiaWdsdTpjb20ubXlfY29tcGFueS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiZGF0YSI6IHsKICAgICAgICAibXlfY29tcGFueV91c2VyX2lkIjogIjk5OTl4eXoiCiAgICAgIH0KICAgIH0KICBdCn0="
     }
   ]
 }'
```

## Page view

This is an example HTTP request for a [page view](/docs/events/ootb-data/page-activity-tracking/) event:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "pv",
      "url": "https://snowplow.io/",
      "page": "Snowplow: behavioral data creation leader",
      "eid": "2ec13853-ef9e-414a-8976-e64a4693b134",
      "tv": "js-3.5.0",
      "tna": "biz1",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665398554084",
      "vp": "693x1302",
      "ds": "678x8188",
      "vid": "29",
      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
      "duid": "85094061-f702-4b62-a46d-20f7226b4741",
      "stm": "1665398554084"
    }
  ]
}
```

**cURL example**

```bash
curl --request POST \
  --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \
  --header 'Content-Type: application/json' \
  --data '{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "pv",
      "url": "https://snowplow.io/",
      "page": "Snowplow: behavioral data creation leader",
      "eid": "2ec13853-ef9e-414a-8976-e64a4693b134",
      "tv": "js-3.5.0",
      "tna": "biz1",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665398554084",
      "vp": "693x1302",
      "ds": "678x8188",
      "vid": "29",
      "sid": "be9520e7-16a5-4d4e-afa1-8e269f99a1cf",
      "duid": "85094061-f702-4b62-a46d-20f7226b4741",
      "stm": "1665398554084"
    }
  ]
}'
```

## Page ping

This is an example HTTP request for a [page ping](/docs/events/ootb-data/page-activity-tracking/) event:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "pp",
      "url": "https://snowplow.io/",
      "page": "Snowplow: behavioral data creation leader",
      "eid": "0a88e2bf-a2cd-48e7-877b-608c4b3d32a6",
      "tv": "js-3.5.0",
      "tna": "snplow5",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665401016632",
      "cx": "",
      "vp": "693x1302",
      "ds": "678x9015",
      "vid": "52",
      "sid": "6a3769de-da79-431d-85dd-0a1b96e2f7fd",
      "duid": "1f08df65-1558-46fc-b586-838460fc438e",
      "stm": "1665401016633"
    }
  ]
}
```

**cURL example**

```bash
curl --request POST \
  --url https://collector.website.com/com.snowplowanalytics.snowplow/tp2 \
  --header 'Content-Type: application/json' \
  --data '{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "e": "pp",
      "url": "https://snowplow.io/",
      "page": "Snowplow: behavioral data creation leader",
      "eid": "0a88e2bf-a2cd-48e7-877b-608c4b3d32a6",
      "tv": "js-3.5.0",
      "tna": "snplow5",
      "aid": "website",
      "p": "web",
      "cookie": "1",
      "cs": "UTF-8",
      "lang": "en-GB",
      "res": "3440x1440",
      "cd": "30",
      "tz": "Europe/London",
      "dtm": "1665401016632",
      "cx": "",
      "vp": "693x1302",
      "ds": "678x9015",
      "vid": "52",
      "sid": "6a3769de-da79-431d-85dd-0a1b96e2f7fd",
      "duid": "1f08df65-1558-46fc-b586-838460fc438e",
      "stm": "1665401016633"
    }
  ]
}'
```

---

# User and session identifiers
> Track user and session identifiers including domain_userid, network_userid, session IDs, and custom business user IDs.
> Source: https://docs.snowplow.io/docs/events/identifiers/

Snowplow tracks several identifiers to help you understand user behavior across sessions and devices. These include tracker-generated IDs stored on the client, server-generated IDs from the Collector, and business user IDs you set in your app. Understanding how these identifiers work enables you to build a complete picture of user journeys and perform identity stitching in your data models.

Read more about tracking identifiers in the [OOTB user and session data](/docs/events/ootb-data/user-and-session-identification/) page.

## User identifiers

### Tracker (in-browser/on-device) generated user identifier

This user identifier is **generated by the tracker**. It takes the form of a UUID. It is persisted either in the browser cookies or in the app user settings storage (e.g., UserDefaults on iOS).

It is the default identifier used in our dbt packages to identify users.

In the tracked events, it can appear either:

1. As the `domain_userid` parameter in the atomic event properties.

   - This is currently the default behavior on Web apps.
   - It is not supported in mobile apps.
   - This is referred to as the default `user_identifier` field in the dbt-snowplow-unified package

2. As the `userId` property in the `client_session` context entity ([see below](/docs/events/ootb-data/user-and-session-identification/#session-entity)).

   - This is an optional configuration in the JavaScript tracker for Web apps.
   - It is the default behavior on mobile apps.
   - This is referred to as the default `user_identifier` field in the dbt-snowplow-unified package (used to be referred to as `device_user_id` in our legacy dbt-snowplow-mobile package).

> **Tip:** Coalesced into the `user_identifier` in our unified dbt packageA good practice is to coalesce the `domain_userid` and the `userId` property in the client session context entity to make sure that the value can come from either place. This is also what our unified dbt packages does – it provides the information under a single `user_identifier` field.

> **Note:** In mobile apps, there are additional on-device identifiers provided by the platform – advertising ID (IDFA) and vendor ID (IDFV, app set ID). These can be tracked [in the mobile entity](/docs/events/ootb-data/device-and-browser/#mobile-entity).

### Server (Collector) generated user identifier

The Snowplow Collector generates a user identifier that is stored in cookies for the Collector domain. This is the `network_userid` property in the atomic events table and it has the format of a UUID.

The identifier is available both in Web and mobile apps. However, in Android apps, it is stored in memory so it is reset after the app restarts.

In most scenarios, this identifier may have a longer lifetime than the tracker generated identifier. However, browsers can restrict it's lifetime for different reasons, such as when the Snowplow Collector is on a third-party domain from the website (not recommended), or due to the ITP restrictions in Safari (Snowplow provides a solution to mitigate this problem – [the Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/)).

> **Info:** `network_userid` is captured via a cookie set by the Snowplow Collector. It can be overriden by setting `tnuid` on a Tracker request payload but is typically expected to be populated by the Collector cookies.

### Business user identifier

This is an external identifier given in the tracker by the app. Most commonly it is the username or e-mail address of the logged in user (e.g., `jon.doe@email.com`).

> **Tip:** Referred to as the `user_id` in the atomic events and our dbt packagesThe business user identifier is provided under the `user_id` field in the atomic events as well as our dbt packages.

All our trackers have an API to set this identifier. You can find it in [the JavaScript tracker docs here](/docs/sources/web-trackers/tracking-events/#business-user-id) and [the mobile trackers docs here](/docs/sources/mobile-trackers/client-side-properties/#set-the-subject-properties).

This identifier can be very useful to stitch the generated tracker identifiers together in order to identify the same user across multiple browsers or devices. See below for more information on user stitching.

### IP address

Although it's not a very reliable identifier, the events also contain the IP address of the user which can be used to identify them to some extent. The value is stored in the `user_ipaddress` property in the atomic events table.

## Session identifiers

In Snowplow events, sessions are tracked on the client-side – by the tracker. The session information is then attached to the events when they are tracked.

The session information consists of two main pieces of information (when using the client session entity discussed below, there is also extra information tracked with sessions):

1. **Session identifier**. UUID identifier generated by the tracker at the start of the session.
2. **Session index**. Index of the session for the current user (per the tracker generated user identifier).

Sessions have a configurable inactivity timeout (which defaults to 30 minutes). After no user activity for the set timeout, a new session is started. On the Web, activity is detected as any user interaction with the page (mouse movement, clicks, ...). In mobile apps, activity is detected using the tracked events.

## Identity stitching in our dbt packages

Identity stitching is the process of taking various user identifiers and combining them into a single user identifier, to better identify and track users throughout their journey on your site/app. It effectively allows you to attribute logged-in and non-logged-in sessions and page views back to a single user. Our dbt packages support identity stitching as [explained here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/).

In order for identity stitching to be reliable, it is necessary to follow two recommendations in tracking:

1. Set the business user identifier.
2. Reset generated identifiers after the user logs out.

### Setting the business user identifier

The stitching functionality in our dbt packages makes use of the business user identifier (see above) to stitch the logged in sessions and page views to non-logged in sessions. It stitches the business user ID with tracker generated user identifiers to identify sessions and page views from the same user.

When the business user ID is present in an event, identity stitching will attribute all events with the same tracker generated user ID with the same stitched user ID. This means that all previous sessions on the same browser or app will have the same stitched user ID as the one with the assigned business user ID.

The business user ID is therefore a requirement for identity stitching.

### Reset generated identifiers after the user logs out

In settings where multiple users share the same browser, identity stitching may wrongly attribute sessions coming from the same browser. Because the tracker generated user identifier will be consistent for all sessions from the browser, the stitched user identifier will also be consistent regardless of whether one or more users log in.

In case you want to make sure that after a user logs out, the following events won't be attributed to the same user ID, you can reset the user and session identifiers in the tracker. To do this on the JavaScript tracker, make use of the function to [clear user data](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) (such a feature is not yet available on the mobile trackers). Clearing the user data on the tracker will result in new tracker generated user and session identifiers. It will not clear the server generated `network_userid` user identifier.

## Where to find the identifiers in events?

The identifiers are either added in the [atomic event properties](/docs/fundamentals/canonical-event/#user-fields) or as a [client session entity](/docs/events/ootb-data/user-and-session-identification/).

## Anonymisation

By default, Snowplow captures identifiers with all events that can be considered personal identifiable information (PII) — user and session IDs as well as the IP address. However, Snowplow also allows you to track data without these identifiers. It provides two ways for limiting the amount of PII you capture and store:

- Not collecting the PII in the first place using the anonymous tracking feature on our trackers.
- Pseudonymizing the PII information during enrichment.

### Anonymous tracking

Anonymous tracking is a feature on our JavaScript and mobile trackers that lets you disable collecting PII information. In particular, it can disable the collection of the following identifiers:

- Tracker generated user identifier
- Tracker generated session identifier
- Server generated user
- The business user identifier
- The user’s IP address

There are several levels of anonymous tracking that our trackers provide:

1. Fully anonymous tracking that prevents collecting both client and server identifiers.
2. Client-only anonymisation – in this setting, the server assigned identifiers (network user ID and IP address) are still collected, but no other identifiers are collected on the tracker.
3. Anonymisation with session tracking – this prevents collecting all user identifiers (optionally server assigned ones can be enabled), but still tracks sessions and adds session information to events.

#### How to track?

- Anonymous tracking on Web [using the JavaScript tracker](/docs/sources/web-trackers/anonymous-tracking/).
- Anonymous tracking in mobile apps [using the iOS and Android trackers](/docs/sources/mobile-trackers/anonymous-tracking/) and [the React Native tracker](/docs/sources/react-native-tracker/anonymous-tracking/).

### PII pseudonymization using enrichment

PII (personally identifiable information) pseudonymization enrichment runs after all the other enrichments and pseudonymizes the fields that are configured as PIIs.

It enables the users of Snowplow to better protect the privacy rights of data subjects, therefore aiding in compliance for regulatory measures.

Full details of this enrichment are [available here](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/).

---

# Overview of Snowplow events
> Links to information about Snowplow event types, custom tracking, and out-of-the-box data collection capabilities.
> Source: https://docs.snowplow.io/docs/events/

This section gives an overview of the different kinds of Snowplow events, as well as general tracking concepts.

Read the [custom data](/docs/events/custom-events/) page for information on tracking custom events and entities.

Check out the [out-of-the-box data](/docs/events/ootb-data/) section to learn about built-in tracking APIs.

For an overview of the properties in the atomic events table, see the [atomic event fields reference](/docs/fundamentals/canonical-event/).

---

# Application error events
> Automatically track exceptions and application errors with the application_error event schema on web and mobile.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-error-events/

Exception (error) tracking captures exceptions within your application.

This table shows the support for exception tracking across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include error tracking APIs.

| Tracker                                                                      | Supported | Since version | Auto-tracking | Notes                                                                                                                                                                                                                                |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Web](/docs/sources/web-trackers/tracking-events/errors/)                    | ✅         | 3.0.0         | ✅/❌           | Requires error tracking plugin. Track handled exceptions manually.                                                                                                                                                                   |
| [iOS](/docs/sources/mobile-trackers/tracking-events/exception-tracking/)     | ✅         | 1.0.0         | ✅             |                                                                                                                                                                                                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/exception-tracking/) | ✅         | 1.0.0         | ✅             |                                                                                                                                                                                                                                      |
| React Native                                                                 | ❌         |               |               | Earlier versions of the React Native tracker supported error tracking for mobile platforms. This feature was removed in version 4.0.0 as it didn't work with JavaScript. Track errors manually using the `application_error` schema. |
| Flutter                                                                      | ❌         |               |               | Use the `application_error` schema for your own custom event.                                                                                                                                                                        |
| Roku                                                                         | ❌         |               |               | Use the `application_error` schema for your own custom event.                                                                                                                                                                        |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)    | ✅         | v3            | ❌             |                                                                                                                                                                                                                                      |

The Errors plugin for the JavaScript trackers provides configuration for automatic tracking of unhandled exceptions, as well as a method for manually tracking handled exceptions.

The mobile trackers will capture unhandled exceptions. These can crash the app, so it's likely that the event will be sent after restart. Note that in some situations, it may not be possible to capture all exception details before the app crashes.

All exception events use the same `application_error` schema, which has the following properties:

### `application_error`

**Type:** Event

Schema for an application error

**Schema:** `iglu:com.snowplowanalytics.snowplow/application_error/jsonschema/1-0-2`

**Example:**

```json
{
  "programmingLanguage": "JAVA",
  "message": "java.lang.OutOfMemoryError error raised",
  "exceptionName": "java.lang.OutOfMemoryError",
  "isFatal": true,
  "threadName": "main",
  "threadId": 1,
  "lineNumber": 10,
  "className": "android.graphics.BitmapFactory",
  "stackTrace": "java.lang.OutOfMemoryError"
}
```

**Properties:**

| Property                       | Description                                                                                                                                                                                                                                                            |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `programmingLanguage` _string_ | _Required._ Must be one of: `JAVA`, `SCALA`, `KOTLIN`, `GROOVY`, `RUBY`, `GOLANG`, `JAVASCRIPT`, `PHP`, `PYTHON`, `OBJECTIVEC`, `SWIFT`, `C`, `CPLUSPLUS`, `CSHARP`, `ACTIONSCRIPT`, `LUA`, `RUST`, `HASKELL`, `CLOJURE`, `ERLANG`, `ELIXIR`, `CRYSTAL`, `PONY`, `NIM` |
| `message` _string_             | _Required._                                                                                                                                                                                                                                                            |
| `threadName` _string_          | _Optional._                                                                                                                                                                                                                                                            |
| `threadId` _integer_           | _Optional._                                                                                                                                                                                                                                                            |
| `stackTrace` _string_          | _Optional._                                                                                                                                                                                                                                                            |
| `causeStackTrace` _string_     | _Optional._                                                                                                                                                                                                                                                            |
| `lineNumber` _integer_         | _Optional._                                                                                                                                                                                                                                                            |
| `className` _string_           | _Optional._                                                                                                                                                                                                                                                            |
| `exceptionName` _string_       | _Optional._                                                                                                                                                                                                                                                            |
| `isFatal` _boolean_            | _Optional._                                                                                                                                                                                                                                                            |
| `lineColumn` _integer_         | _Optional._                                                                                                                                                                                                                                                            |
| `fileName` _string_            | _Optional._                                                                                                                                                                                                                                                            |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_application_error_1 application_error_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'application_error'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# App and tracker information
> Track application and tracker metadata including version, build, platform, and namespace in atomic fields and context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-information/

Information about the application and the tracker instance that is sending the events to Snowplow can be useful for data analysis and troubleshooting issues.

You can track this in two ways: setting [atomic event properties](/docs/fundamentals/canonical-event/), or configuring the application entity.

## Application atomic event properties

These [properties](/docs/fundamentals/canonical-event/#application-fields) can be assigned across all our trackers, regardless of the platform.

| Atomic table field | Type | Description                                                                                          | Example values      |
| ------------------ | ---- | ---------------------------------------------------------------------------------------------------- | ------------------- |
| `name_tracker`     | text | The tracker namespace                                                                                | `tracker_1`         |
| `app_id`           | text | Unique identifier for website / application                                                          | `snow-game-android` |
| `platform`         | text | The platform the app runs on                                                                         | `web`, `mob`, `app` |
| `v_tracker`        | text | Identifier for Snowplow tracker. The format follows the convention of `TRACKER_NAME-TRACKER_VERSION` | `js-2.16.2`         |

You can specify the tracker namespace and app ID when creating a new tracker instance.

The tracker platform is set automatically but can be overriden in most of our trackers. The tracker version is also set automatically.

## Application entity

You can configure your web or mobile trackers to automatically include an application entity with all tracked events.

The included data is slightly different depending on the platform, as different schemas are used:

- Web: `version` only
- Mobile: `version` and `build`

This table shows the support for the application entity across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include this entity.

| Tracker                                                                                                        | Supported | Since version | Auto-tracking | Notes                                                |
| -------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#application-version)                                         | ✅         | 4.1.0         | ✅             |                                                      |
| [iOS](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#application-context)     | ✅         | 1.0.0         | ✅             |                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#application-context) | ✅         | 1.0.0         | ✅             |                                                      |
| [React Native](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/)           | ✅         | 4.0.0         | ✅             | Uses web or mobile schema depending on configuration |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                                     | ✅         | 0.3.0         | ✅             | Available for mobile only                            |
| [Roku](/docs/sources/roku-tracker/adding-data/)                                                                | ✅         | 0.1.0         | ✅             |                                                      |
| Google Tag Manager                                                                                             | ❌         |               |               |                                                      |

The native mobile trackers are able to extract the application version and build number automatically from the app metadata. For web, you need to provide the application version manually in the tracker configuration.

### Entity definitions

Web applications:

### `application`

**Type:** Entity

Schema for an application context which tracks the app version.

**Schema:** `iglu:com.snowplowanalytics.snowplow/application/jsonschema/1-0-0`

**Example:**

```json
{
  "version": "1.1.0"
}
```

**Properties:**

| Property           | Description                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| `version` _string_ | _Required._ Version of the application. Can be a semver-like structure (e.g 1.1.0) or a Git commit SHA hash. |

Mobile applications:

### `application`

**Type:** Entity

Schema for an application context which automatically tracks version number and build name when using our mobile SDK's.

**Schema:** `iglu:com.snowplowanalytics.mobile/application/jsonschema/1-0-0`

**Example:**

```json
{
  "version": "1.1.0",
  "build": "s9f2k2d"
}
```

**Properties:**

| Property           | Description                                                         |
| ------------------ | ------------------------------------------------------------------- |
| `version` _string_ | _Required._ Version number of the application e.g 1.1.0             |
| `build` _string_   | _Required._ Build name of the application e.g s9f2k2d or 1.1.0 beta |

---

# App performance
> Track web vitals, navigation timing, and custom performance metrics using automatic plugins or manual timing events.
> Source: https://docs.snowplow.io/docs/events/ootb-data/app-performance/

We provide two out-of-the-box solutions for tracking app performance metrics:

- Plugins for automatic performance tracking on web
- Timing events, available in most Snowplow trackers, for manual tracking

## Automatic tracking on web

The [JavaScript tracker](/docs/sources/web-trackers/) provides two configurable plugins that track performance automatically.

Use the web vitals plugin to track key user-centric performance metrics, and the performance navigation timing plugin to capture detailed navigation timing data.

### Web vitals

This [plugin](/docs/sources/web-trackers/tracking-events/web-vitals/) tracks web performance metrics categorized as [Web Vitals](https://web.dev/vitals/).

This table shows which versions of the trackers can use the web vitals plugins:

| Tracker            | Supported | Since version | Auto-tracking | Notes                                                             |
| ------------------ | --------- | ------------- | ------------- | ----------------------------------------------------------------- |
| Web                | ✅         | 3.13.0        | ✅             |                                                                   |
| React Native       | ❌         |               |               | Tracker can't access the required browser APIs                    |
| Google Tag Manager | ❌         |               |               | Not directly supported, but you can load it as an external plugin |

To process raw web vitals event data, use the core web vitals module in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/).

### `web_vitals`

**Type:** Event

Schema for a web vitals tracking event. For more information on web vitals you can visit https\://web.dev/vitals/.

**Schema:** `iglu:com.snowplowanalytics.snowplow/web_vitals/jsonschema/1-0-0`

**Example:**

```json
{
  "schema_name": "web_vitals",
  "cls": 0.05,
  "fcp": 0,
  "fid": "36:00.0",
  "inp": "36:00.0",
  "lcp": 1908,
  "navigationType": "navigate",
  "ttfb": 228.9
}
```

**Properties:**

| Property                                   | Description                                                                                                                                                                                                                                                         |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cls` (Cumulative Layout Shift) _number_   | _Optional._ A unitless metric for measuring visual stability because it helps quantify how often users experience unexpected layout shifts. For more information https\://web.dev/cls/.                                                                             |
| `fid` (First Input Delay) _number_         | _Optional._ A metric for measuring load responsiveness because it quantifies the experience users feel when trying to interact with unresponsive pages. Measured in milliseconds. For more information https\://web.dev/fid/.                                       |
| `lcp` (Largest Contentful Paint) _number_  | _Optional._ A metric for measuring perceived load speed because it marks the point in the page load timeline when the page's main content has likely loaded. Measured in milliseconds. For more information https\://web.dev/lcp/.                                  |
| `fcp` (First Contentful Paint) _number_    | _Optional._ A metric for measuring perceived load speed because it marks the first point in the page load timeline where the user can see anything on the screen. Measured in milliseconds. For more information https\://web.dev/fcp/.                             |
| `inp` (Interaction to Next Paint) _number_ | _Optional._ A metric that assesses responsiveness. INP observes the latency of all interactions a user has made with the page, and reports a single value which all (or nearly all) interactions were below that value. For more information https\://web.dev/inp/. |
| `ttfb` (Time To First Byte) _number_       | _Optional._ A DOMHighResTimeStamp referring to the time in milliseconds between the browser requesting a page and when it receives the first byte of information from the server. For more information https\://web.dev/ttfb/.                                      |
| `navigationType` _string_                  | _Optional._ The navigation type recognised from the Navigation Timing API https\://www\.w3.org/TR/navigation-timing-2/. E.g. 'navigate', 'reload', 'back-forward', 'back-forward-cache', 'prerender', 'restore'.                                                    |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1 web_vitals_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'web_vitals'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Performance navigation timing

This [plugin](/docs/sources/web-trackers/tracking-events/timings/) will add Performance Navigation Timing entities to all tracked events.

This table shows which versions of the trackers can use the performance plugins:

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                                                               |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------------------- |
| Web                                                                       | ✅         | 3.10.0        | ✅             |                                                                     |
| React Native                                                              | ❌         |               |               | Tracker can't access the required browser APIs to use these plugins |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/) | ✅         | v4            | ✅             | Integrates with the performance navigation timing plugin            |

### `PerformanceNavigationTiming`

**Type:** Entity

Schema for page navigation performance entity, based on the PerformanceNavigationTiming interface (see https\://w3c.github.io/navigation-timing/)

**Schema:** `iglu:org.w3/PerformanceNavigationTiming/jsonschema/1-0-0`

**Properties:**

| Property                              | Description                                                                                                                                                                                                                                                                                                             |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `decodedBodySize` _integer_           | _Optional._ A number that is the size (in octets) received from the fetch (HTTP or cache) of the message body, after removing any applied content encoding.                                                                                                                                                             |
| `encodedBodySize` _integer_           | _Optional._ A number representing the size (in octets) received from the fetch (HTTP or cache), of the payload body, before removing any applied content encodings.                                                                                                                                                     |
| `redirectStart` _number_              | _Optional._ A DOMHighResTimeStamp that represents the start time of the fetch which initiates the redirect in milliseconds.                                                                                                                                                                                             |
| `redirectEnd` _number_                | _Optional._ A DOMHighResTimeStamp immediately after receiving the last byte of the response of the last redirect in milliseconds.                                                                                                                                                                                       |
| `fetchStart` _number_                 | _Optional._ A DOMHighResTimeStamp immediately before the browser starts to fetch the resource in milliseconds.                                                                                                                                                                                                          |
| `domainLookupStart` _number_          | _Optional._ A DOMHighResTimeStamp immediately before the browser starts the domain name lookup for the resource in milliseconds.                                                                                                                                                                                        |
| `domainLookupEnd` _number_            | _Optional._ A DOMHighResTimeStamp representing the time immediately after the browser finishes the domain name lookup for the resource in milliseconds.                                                                                                                                                                 |
| `connectStart` _number_               | _Optional._ A DOMHighResTimeStamp immediately before the browser starts to establish the connection to the server to retrieve the resource in milliseconds.                                                                                                                                                             |
| `secureConnectionStart` _number_      | _Optional._ A DOMHighResTimeStamp immediately before the browser starts the handshake process to secure the current connection in milliseconds.                                                                                                                                                                         |
| `connectEnd` _number_                 | _Optional._ A DOMHighResTimeStamp immediately after the browser finishes establishing the connection to the server to retrieve the resource in milliseconds.                                                                                                                                                            |
| `requestStart` _number_               | _Optional._ A DOMHighResTimeStamp immediately before the browser starts requesting the resource from the server in milliseconds.                                                                                                                                                                                        |
| `responseStart` _number_              | _Optional._ A DOMHighResTimeStamp immediately after the browser receives the first byte of the response from the server in milliseconds.                                                                                                                                                                                |
| `responseEnd` _number_                | _Optional._ A DOMHighResTimeStamp immediately after the browser receives the last byte of the resource or immediately before the transport connection is closed in milliseconds, whichever comes first.                                                                                                                 |
| `unloadEventStart` _number_           | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's unload event handler starts in milliseconds.                                                                                                                                                                           |
| `unloadEventEnd` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's unload event handler completes in milliseconds.                                                                                                                                                                        |
| `domInteractive` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately before the user agent sets the document's readyState to 'interactive' in milliseconds.                                                                                                                                                              |
| `domContentLoadedEventStart` _number_ | _Optional._ A DOMHighResTimeStamp representing the time immediately before the current document's DOMContentLoaded event handler starts in milliseconds.                                                                                                                                                                |
| `domContentLoadedEventEnd` _number_   | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's DOMContentLoaded event handler completes in milliseconds.                                                                                                                                                              |
| `domComplete` _number_                | _Optional._ A DOMHighResTimeStamp representing the time immediately before the user agent sets the document's readyState to 'complete' in milliseconds.                                                                                                                                                                 |
| `loadEventStart` _number_             | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's load event handler starts in milliseconds.                                                                                                                                                                             |
| `loadEventEnd` _number_               | _Optional._ A DOMHighResTimeStamp representing the time immediately after the current document's load event handler completes in milliseconds.                                                                                                                                                                          |
| `entryType` _string_                  | _Optional._ The string 'navigation'.                                                                                                                                                                                                                                                                                    |
| `redirectCount` _integer_             | _Optional._ A number representing the number of redirects since the last non-redirect navigation in the current browsing context.                                                                                                                                                                                       |
| `type` _string_                       | _Optional._ A string representing the navigation type. Either 'navigate', 'reload', 'back\_forward' or 'prerender'.                                                                                                                                                                                                     |
| `workerStart` _number_                | _Optional._ Returns a DOMHighResTimeStamp immediately before dispatching the FetchEvent if a Service Worker thread is already running, or immediately before starting the Service Worker thread if it is not already running. If the resource is not intercepted by a Service Worker the property will always return 0. |
| `nextHopProtocol` _string_            | _Optional._ A string representing the network protocol used to fetch the resource, as identified by the ALPN Protocol ID (RFC7301)                                                                                                                                                                                      |
| `transferSize` _integer_              | _Optional._ A number representing the size (in octets) of the fetched resource. The size includes the response header fields plus the response payload body.                                                                                                                                                            |
| `duration` _number_                   | _Optional._ Returns a timestamp that is the difference between the loadEventEnd and startTime properties.                                                                                                                                                                                                               |
| `activationStart` _number_            | _Optional._ If the document is prerendered, activationStart represents the time between when the prerender was started and the document was actually activated.                                                                                                                                                         |
| `deliveryType` _string_               | _Optional._ Expose information about how a resource was delivered e.g. resources which were delivered from the cache.                                                                                                                                                                                                   |
| `serverTiming` _array_                | _Optional._ Array of PerformanceServerTiming entries.                                                                                                                                                                                                                                                                   |

## Manual timing events

Track things like resource load times or other performance measurements using manual timing events.

This table shows the support for timing events across Snowplow [tracker SDKs](/docs/sources/).

| Tracker                                                                                               | Supported | Since version |
| ----------------------------------------------------------------------------------------------------- | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timings/generic/)                                    | ✅         | 3.0.0         |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#creating-a-timing-event)                         | ✅         | 2.0.0         |
| [Android](/docs/sources/mobile-trackers/tracking-events/#creating-a-timing-event)                     | ✅         | 2.0.0         |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-timing-events)            | ✅         | 0.1.0         |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/#track-timing-events-withtiming)              | ✅         | 0.1.0         |
| Roku                                                                                                  | ❌         |               |
| [Java](/docs/sources/java-tracker/tracking-events/#creating-a-timing-event)                           | ✅         | 0.8.0         |
| [Golang](/docs/sources/golang-tracker/tracking-specific-events/#track-timing-events-with-tracktiming) | ✅         | 0.1.0         |
| Python                                                                                                | ❌         |               |
| Ruby                                                                                                  | ❌         |               |
| PHP                                                                                                   | ❌         |               |
| [.NET](/docs/sources/net-tracker/event-tracking/#track-timing-events-withtracktiming)                 | ✅         | 1.0.0         |
| [C++](/docs/sources/c-tracker/tracking-specific-events/#track-timing-events-withtimingevent)          | ✅         | 0.3.0         |
| [Rust](/docs/sources/rust-tracker/tracking-events/#track-timing-events-withtimingevent)               | ✅         | 0.1.0         |
| [Unity](/docs/sources/unity-tracker/event-tracking/#track-timing-events-withtracktiming)              | ✅         | 0.1.0         |
| Scala                                                                                                 | ❌         |               |
| Lua                                                                                                   | ❌         |               |
| Node.js                                                                                               | ❌         |               |
| Google Tag Manager                                                                                    | ❌         |               |

You can still track timing data using trackers without built-in timing event support. Use custom events with the `timing` schema.

### `timing`

**Type:** Event

Schema for a user timing event

**Schema:** `iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0`

**Properties:**

| Property            | Description |
| ------------------- | ----------- |
| `category` _string_ | _Required._ |
| `variable` _string_ | _Required._ |
| `timing` _number_   | _Required._ |
| `label` _string_    | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_timing_1 timing_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'timing'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Campaign and ad tracking
> Track marketing campaigns with UTM parameters and ad performance with impression, click, and conversion events for attribution analysis.
> Source: https://docs.snowplow.io/docs/events/ootb-data/campaigns-and-ads/

Campaign and ad tracking captures how users arrive at your site from marketing campaigns and how they interact with advertisements.

Use campaign and ad tracking to:

- Attribute traffic to specific marketing campaigns
- Measure ad impression and click performance
- Track conversions from ad campaigns
- Analyze the effectiveness of different marketing channels

## Campaign atomic event properties

The [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) can automatically capture UTM parameters from URLs and populate [atomic event properties](/docs/fundamentals/canonical-event/#marketing-fields) for campaign analysis.

## Advertisement events

Ad tracking captures impressions, clicks, and conversions from advertisements displayed on your site or elsewhere.

See also [media tracking](/docs/events/ootb-data/media-events/) for tracking advertising content consumed within video or audio media.

This table shows the support for ad tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                   | Supported | Since version                       | Auto-tracking | Notes                           |
| ------------------------------------------------------------------------- | --------- | ----------------------------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ads/)                    | ✅         | 1.1.0 (directly), 3.0.0 (as plugin) | ❌             | Requires the ad tracking plugin |
| iOS                                                                       | ❌         |                                     |               |                                 |
| Android                                                                   | ❌         |                                     |               |                                 |
| React Native                                                              | ❌         |                                     |               |                                 |
| Flutter                                                                   | ❌         |                                     |               |                                 |
| Roku                                                                      | ❌         |                                     |               |                                 |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                  | ❌             |                                 |

### Ad impression event

Track when an ad is displayed to a user.

### `ad_impression`

**Type:** Event

Schema for an ad impression event

**Schema:** `iglu:com.snowplowanalytics.snowplow/ad_impression/jsonschema/1-0-0`

**Example:**

```json
{
  "impressionId": "67965967893",
  "costModel": "cpm",
  "cost": 5.5,
  "targetUrl": "http://www.example.com",
  "bannerId": "23",
  "zoneId": "7",
  "advertiserId": "201",
  "campaignId": "12"
}
```

**Properties:**

| Property                | Description                                                                      |
| ----------------------- | -------------------------------------------------------------------------------- |
| `impressionId` _string_ | _Optional._ Identifier for the impression instance                               |
| `costModel` _string_    | _Optional._ The cost model: cpa, cpc, or cpm Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the impression                                           |
| `targetUrl` _string_    | _Optional._ The destination URL                                                  |
| `bannerId` _string_     | _Optional._ Adserver identifier for the ad banner                                |
| `zoneId` _string_       | _Optional._ Adserver identifier for the zone                                     |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser                               |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign                                 |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_ad_impression_1 ad_impression_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_impression'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Ad click event

Track when a user clicks on an ad.

### `ad_click`

**Type:** Event

Schema for an ad click event

**Schema:** `iglu:com.snowplowanalytics.snowplow/ad_click/jsonschema/1-0-0`

**Example:**

```json
{
  "targetUrl": "http://www.example.com",
  "clickId": "12243253",
  "costModel": "cpm",
  "cost": 2.5,
  "bannerId": "23",
  "zoneId": "7",
  "impressionId": "67965967893",
  "advertiserId": "201",
  "campaignId": "12"
}
```

**Properties:**

| Property                | Description                                                    |
| ----------------------- | -------------------------------------------------------------- |
| `targetUrl` _string_    | _Required._ The destination URL                                |
| `clickId` _string_      | _Optional._ Identifier for the click instance                  |
| `costModel` _string_    | _Optional._ The cost model Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the click                              |
| `bannerId` _string_     | _Optional._ Adserver identifier for the ad banner              |
| `zoneId` _string_       | _Optional._ Adserver identifier for the zone                   |
| `impressionId` _string_ | _Optional._ Links click to impression                          |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser             |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign               |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_ad_click_1 ad_click_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_click'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Ad conversion event

Track when a user completes a conversion action from an ad.

### `ad_conversion`

**Type:** Event

Schema for an ad conversion event

**Schema:** `iglu:com.snowplowanalytics.snowplow/ad_conversion/jsonschema/1-0-0`

**Example:**

```json
{
  "conversionId": "743560297",
  "costModel": "cpa",
  "cost": 10,
  "category": "ecommerce",
  "action": "purchase",
  "property": "",
  "initialValue": 99,
  "advertiserId": "201",
  "campaignId": "12"
}
```

**Properties:**

| Property                | Description                                                    |
| ----------------------- | -------------------------------------------------------------- |
| `conversionId` _string_ | _Optional._ Identifier for the conversion instance             |
| `costModel` _string_    | _Optional._ The cost model Must be one of: `cpa`, `cpc`, `cpm` |
| `cost` _number_         | _Optional._ The cost of the conversion                         |
| `category` _string_     | _Optional._ Conversion category                                |
| `action` _string_       | _Optional._ The type of user interaction, e.g. purchase        |
| `property` _string_     | _Optional._ Describes the object of the conversion             |
| `initialValue` _number_ | _Optional._ How much the conversion is initially worth         |
| `advertiserId` _string_ | _Optional._ Adserver identifier for the advertiser             |
| `campaignId` _string_   | _Optional._ Adserver identifier for the campaign               |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_ad_conversion_1 ad_conversion_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_conversion'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Consent events and GDPR
> Track GDPR consent preferences, CMP visibility, and marketing consent with enhanced consent events and legacy consent APIs.
> Source: https://docs.snowplow.io/docs/events/ootb-data/consent-events/

Snowplow provides out-of-the-box [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) to track user consent preferences and GDPR compliance. Use these to capture consent decisions, track changes to user preferences, and monitor Consent Management Platform (CMP) performance.

You can also configure your tracker not to track certain identifiers, depending on user consent, using [anonymous tracking](/docs/events/anonymous-tracking/).

## Consent API versions

Snowplow provides two versions of consent tracking APIs:

- Enhanced consent APIs: available for web only using the [Enhanced Consent plugin](/docs/sources/web-trackers/tracking-events/consent-gdpr/), these APIs track detailed consent preferences and CMP visibility events
- Basic consent APIs: available for mobile, these APIs track basic consent activity

To process raw events created by the Snowplow Enhanced Consent plugin, use the consent module in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/).

> **Note:** The Snowplow dbt packages don't support the basic consent events and entities. You can however build your own models based on the raw events.

## Tracker support

This table shows the support for consent tracking across the main client-side Snowplow [tracker SDKs](/docs/sources/). The server-side trackers don't include consent tracking APIs.

Older versions of the web trackers provided the `browser-plugin-consent` plugin for basic consent tracking. It was deprecated in version 4 in favor of `browser-plugin-enhanced-consent`.

| Tracker                                                                            | Supported | Since version                           | Auto-tracking | Notes                                                                               |
| ---------------------------------------------------------------------------------- | --------- | --------------------------------------- | ------------- | ----------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/consent-gdpr/)                    | ✅         | 3.0.0 - 4.0.0 (basic), 3.8.0 (enhanced) | ✅/❌           | Track enhanced consent events manually; configure automatic addition of GDPR entity |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#creating-a-consent-event)     | ✅         | 1.0.0                                   | ✅/❌           | Track basic consent events manually; configure automatic addition of GDPR entity    |
| [Android](/docs/sources/mobile-trackers/tracking-events/#creating-a-consent-event) | ✅         | 1.0.0                                   | ✅/❌           | Track basic consent events manually; configure automatic addition of GDPR entity    |
| [React Native](/docs/sources/react-native-tracker/tracking-events/)                | ✅         | 1.0.0                                   | ❌             | Basic API\*; no GDPR entity                                                         |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/)                          | ✅         | 0.1.0                                   | ❌             | Basic API                                                                           |
| Roku                                                                               | ❌         |                                         |               | Track custom events using the enhanced schemas                                      |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)          | ✅         | v3                                      | ❌             | Enhanced consent events                                                             |

\*It's also possible to use the JavaScript Enhanced Consent plugin with the React Native tracker.

## Events and entities

This section describes the events and entities used in Snowplow consent tracking.

### Enhanced

The [Enhanced Consent plugin](/docs/sources/web-trackers/tracking-events/consent-gdpr/) includes a number of tracking calls for different user consent actions:

| API                     | To track                                                |
| ----------------------- | ------------------------------------------------------- |
| `trackConsentAllow`     | Acceptance of user consent                              |
| `trackConsentSelected`  | A specific selection of consented scopes                |
| `trackConsentPending`   | The unconfirmed selection about user consent            |
| `trackConsentImplicit`  | The implicit consent on user consent preferences        |
| `trackConsentDeny`      | A denial of user consent                                |
| `trackConsentExpired`   | The expiration of a consent selection                   |
| `trackConsentWithdrawn` | The withdrawal of user consent                          |
| `trackCmpVisible`       | The render time of a consent management platform banner |

With the exception of the CMP visible event, these methods use the same `consent_preferences` event schema. The CMP visible event uses the `cmp_visible` schema.

### `consent_preferences`

**Type:** Event

Schema for consent preferences selection event

**Schema:** `iglu:com.snowplowanalytics.snowplow/consent_preferences/jsonschema/1-0-0`

**Example:**

```json
{
  "basisForProcessing": "consent",
  "consentVersion": "1",
  "consentScopes": [
    "necessary",
    "preferences",
    "statistics"
  ],
  "domainsApplied": [
    "https://www.example.com/"
  ],
  "consentUrl": "https://www.example.com/",
  "eventType": "allow_selected",
  "gdprApplies": false
}
```

**Properties:**

| Property                  | Description                                                                                                                                                                         |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eventType` __            | _Required._ The action for the consent preferences of a user. Must be one of: `deny_all`, `allow_all`, `allow_selected`, `pending`, `implicit_consent`, `withdrawn`, `expired`      |
| `basisForProcessing` __   | _Required._ GDPR lawful basis for data collection & processing. Must be one of: `consent`, `contract`, `legal_obligation`, `vital_interests`, `public_task`, `legitimate_interests` |
| `consentUrl` _string_     | _Required._ URI of the privacy policy related document.                                                                                                                             |
| `consentVersion` _string_ | _Required._ Version of the privacy policy related document.                                                                                                                         |
| `consentScopes` _array_   | _Required._ The scopes allowed after the user finalized his selection of consent preferences. E.g \['analytics', 'functional', 'advertisement'].                                    |
| `domainsApplied` _array_  | _Required._ The domains for which this consent allows these preferences to persist to.                                                                                              |
| `gdprApplies` _boolean_   | _Optional._ Determine if GDPR applies based on the user's geo-location.                                                                                                             |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_consent_preferences_1 consent_preferences_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'consent_preferences'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### `cmp_visible`

**Type:** Event

Tracks the load time of Consent Management Platform (CMP) banners.

**Schema:** `iglu:com.snowplowanalytics.snowplow/cmp_visible/jsonschema/1-0-0`

**Example:**

```json
{
  "elapsedTime": 1.5
}
```

**Properties:**

| Property               | Description                                                                 |
| ---------------------- | --------------------------------------------------------------------------- |
| `elapsedTime` _number_ | _Required._ The time taken for the consent popup to be shown to the screen. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_cmp_visible_1 cmp_visible_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'cmp_visible'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Basic

When you track a `consent_granted` or `consent_withdrawn` event, the tracker will automatically create and attach a `consent_document` entity.

The [native mobile](/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/) and [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) trackers also support automatic addition of the `gdpr` entity to all tracked events. To configure this, provide a `GdprConfiguration` object when setting up a new tracker.

The React Native tracker did include GDPR entity configuration in earlier versions. We [deprecated it in version 4](/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/).

### `consent_granted`

**Type:** Schema

Schema for consent granted

**Schema:** `iglu:com.snowplowanalytics.snowplow/consent_granted/jsonschema/1-0-0`

**Properties:**

| Property          | Description |
| ----------------- | ----------- |
| `expiry` _string_ | _Optional._ |

### `consent_withdrawn`

**Type:** Schema

Schema for consent withdrawn

**Schema:** `iglu:com.snowplowanalytics.snowplow/consent_withdrawn/jsonschema/1-0-0`

**Properties:**

| Property        | Description |
| --------------- | ----------- |
| `all` _boolean_ | _Required._ |

### `consent_document`

**Type:** Schema

Schema for consent document context

**Schema:** `iglu:com.snowplowanalytics.snowplow/consent_document/jsonschema/1-0-0`

**Properties:**

| Property               | Description |
| ---------------------- | ----------- |
| `id` _string_          | _Required._ |
| `version` _string_     | _Required._ |
| `name` _string_        | _Optional._ |
| `description` _string_ | _Optional._ |

### `gdpr`

**Type:** Schema

Schema for a web page context

**Schema:** `iglu:com.snowplowanalytics.snowplow/gdpr/jsonschema/1-0-0`

**Properties:**

| Property                       | Description                                                                                                                                                                 |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `basisForProcessing` _string_  | _Required._ GDPR basis for data collection & processing Must be one of: `consent`, `contract`, `legal_obligation`, `vital_interests`, `public_task`, `legitimate_interests` |
| `documentId` _string_          | _Optional._ ID for document detailing basis for processing                                                                                                                  |
| `documentVersion` _string_     | _Optional._ Version of document detailing basis for processing                                                                                                              |
| `documentDescription` _string_ | _Optional._ Description of document detailing basis for processing                                                                                                          |

---

# Device and browser information
> Track device, browser, and platform information in atomic fields and context entities including YAUAA parsing, Client Hints, and mobile context.
> Source: https://docs.snowplow.io/docs/events/ootb-data/device-and-browser/

Snowplow trackers track information about the device or browser that is sending the events to Snowplow in two ways: setting [atomic event properties](/docs/fundamentals/canonical-event/), or in browser or application entities. You can also choose to configure additional entities through [enrichment](/docs/pipeline/enrichments/).

## Browser atomic event properties

Some trackers can populate the [browser atomic event properties](/docs/fundamentals/canonical-event/#browser-fields).

Depending on the tracker, you may have to provide the values, as not all trackers capture these fields themselves. The tracker will add the values automatically to all sent events.

> **Note:** Only the JavaScript tracker can populate the `br_cookies` field.

| Tracker                                                                      | Supported | Since version | Auto-tracking | Notes                                         |
| ---------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/browsers/)                  | ✅         | 0.1.0         | ✅             | All values set automatically                  |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.5.0         | ✅             |                                               |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | ✅             |                                               |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | ✅             |                                               |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | ✅             |                                               |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.2.0         | ✅             | Set automatically; language and viewport only |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | ✅             |                                               |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | ✅             |                                               |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | ✅             |                                               |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.10.0        | ✅             |                                               |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.13.0        | ✅             |                                               |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.6.0         | ✅             |                                               |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         | ✅             |                                               |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | ✅             | Set language only                             |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | ✅             | Set language and color depth only             |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | ✅             |                                               |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | ✅             |                                               |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                   | ✅         | 0.1.0         | ✅             | Set color depth and viewport only             |
| Google Tag Manager                                                           | ❌         |               |               |                                               |

## Browser entity

You can configure the web trackers to automatically include a browser entity with all tracked events.

| Tracker                                                                              | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) | ✅         | 3.9.0         | ✅             |
| iOS                                                                                  | ❌         |               |               |
| Android                                                                              | ❌         |               |               |
| React Native                                                                         | ❌         |               |               |
| Flutter                                                                              | ❌         |               |               |
| Google Tag Manager                                                                   | ❌         |               |               |

The browser entity is only available on web trackers since it captures browser-specific information from the DOM and browser APIs. These APIs aren't available to React Native or Flutter.

### `browser_context`

**Type:** Entity

Schema for browser contexts

**Schema:** `iglu:com.snowplowanalytics.snowplow/browser_context/jsonschema/1-0-0`

**Example:**

```json
{
  "viewport": null,
  "documentSize": null,
  "resolution": null,
  "colorDepth": 1,
  "devicePixelRatio": 1,
  "cookiesEnabled": true,
  "online": true,
  "browserLanguage": null,
  "documentLanguage": null,
  "webdriver": true,
  "deviceMemory": 1,
  "hardwareConcurrency": 1,
  "tabId": null
}
```

**Properties:**

| Property                        | Description                                                                                                                                                   |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `viewport` _string_             | _Required._ Viewport dimensions of the browser. Arrives in the form of WidthxHeight e.g. 1200x900.                                                            |
| `documentSize` _string_         | _Required._ Document dimensions. Arrives in the form of WidthxHeight e.g. 1200x900                                                                            |
| `resolution` _string_           | _Required._ Device native resolution. Arrives in the form of WidthxHeight e.g. 1200x900                                                                       |
| `colorDepth` _integer_          | _Required._ The number of bits allocated to colors for a pixel in the output device, excluding the alpha channel.                                             |
| `devicePixelRatio` _number_     | _Optional._ Ratio of the resolution in physical pixels to the resolution in CSS pixels for the current display device.                                        |
| `cookiesEnabled` _boolean_      | _Required._ Indicates whether cookies are enabled or not. More info and caveats at https\://developer.mozilla.org/en-US/docs/Web/API/Navigator/cookieEnabled. |
| `online` _boolean_              | _Required._ Returns the online status of the browser. Important caveats are described in https\://developer.mozilla.org/en-US/docs/Web/API/Navigator/onLine.  |
| `browserLanguage` _string_      | _Optional._ The preferred language of the user, usually the language of the browser UI. RFC 5646 https\://datatracker.ietf.org/doc/html/rfc5646.              |
| `documentLanguage` _string_     | _Optional._ The language of the HTML document. RFC 5646 https\://datatracker.ietf.org/doc/html/rfc5646.                                                       |
| `webdriver` _boolean_           | _Optional._ Indicates whether the user agent is controlled by automation.                                                                                     |
| `deviceMemory` _integer_        | _Optional._ Approximate amount of device memory in gigabytes.                                                                                                 |
| `hardwareConcurrency` _integer_ | _Optional._ Number of logical processors available to run threads on the user's computer.                                                                     |
| `tabId` _string_                | _Optional._ An identifier for the client browser tab the event is sent from.                                                                                  |

## Mobile entity

The mobile entity gives information about the mobile device platform, including identifiers such as IDFA and IDFV. It's sometimes referred to as the platform entity.

It's enabled by default in all supported trackers. You can configure which of the optional properties to track.

This table shows the support for the mobile entity across the main client-side Snowplow [tracker SDKs](/docs/sources/).

| Tracker                                                                                                               | Supported | Since version | Auto-tracking | Notes                             |
| --------------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------- |
| Web                                                                                                                   | ❌         |               |               |                                   |
| [iOS](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#platform-mobile-context)        | ✅         | 1.0.0         | ✅             |                                   |
| [Android](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#platform-mobile-context)    | ✅         | 1.0.0         | ✅             |                                   |
| [React Native](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/#platform-context) | ✅         | 1.0.0         | ✅             | Only relevant for mobile tracking |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                                            | ✅         | 0.1.0         | ✅             | Only relevant for mobile tracking |
| Roku                                                                                                                  | ❌         |               |               |                                   |
| Google Tag Manager                                                                                                    | ❌         |               |               |                                   |

### `mobile_context`

**Type:** Entity

Schema for mobile contexts

**Schema:** `iglu:com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-3`

**Example:**

```json
{
  "deviceManufacturer": "Samsung",
  "deviceModel": "SM-N960N Galaxy Note9 TD-LTE KR 128GB",
  "osType": "Google Android 8.1 (Oreo)",
  "osVersion": 8.1,
  "androidIdfa": "00000000-0000-0000-0000-000000000000",
  "appleIdfa": null,
  "appleIdfv": null,
  "carrier": "Unknown",
  "openIdfa": null,
  "networkTechnology": "2G",
  "networkType": "wifi",
  "physicalMemory": 1,
  "systemAvailableMemory": 1,
  "appAvailableMemory": 1,
  "batteryLevel": 42,
  "batteryState": "charging",
  "lowPowerMode": true,
  "availableStorage": 1,
  "totalStorage": 128000000000,
  "isPortrait": true,
  "resolution": "1440x2960",
  "scale": 1,
  "language": null,
  "appSetId": null,
  "appSetIdScope": null
}
```

**Properties:**

| Property                          | Description                                                                                                                                                                                                                 |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `osType` _string_                 | _Required._ Operating system type (e.g., ios, tvos, watchos, osx, android)                                                                                                                                                  |
| `osVersion` _string_              | _Required._ The current version of the operating system                                                                                                                                                                     |
| `deviceManufacturer` _string_     | _Required._ The manufacturer of the product/hardware                                                                                                                                                                        |
| `deviceModel` _string_            | _Required._ The end-user-visible name for the end product                                                                                                                                                                   |
| `carrier` _string_                | _Optional._ The carrier of the SIM inserted in the device                                                                                                                                                                   |
| `networkType` _string_            | _Optional._ Type of network the device is connected to Must be one of: `mobile`, `wifi`, `offline`,``                                                                                                                       |
| `networkTechnology` _string_      | _Optional._ Radio access technology that the device is using                                                                                                                                                                |
| `openIdfa` _string_               | _Optional._ Deprecated tracking identifier for iOS                                                                                                                                                                          |
| `appleIdfa` _string_              | _Optional._ Advertising identifier on iOS                                                                                                                                                                                   |
| `appleIdfv` _string_              | _Optional._ UUID identifier for vendors on iOS                                                                                                                                                                              |
| `androidIdfa` _string_            | _Optional._ Advertising identifier on Android                                                                                                                                                                               |
| `physicalMemory` _integer_        | _Optional._ Total physical system memory in bytes                                                                                                                                                                           |
| `systemAvailableMemory` _integer_ | _Optional._ Available memory on the system in bytes (Android only)                                                                                                                                                          |
| `appAvailableMemory` _integer_    | _Optional._ Amount of memory in bytes available to the current app (iOS only)                                                                                                                                               |
| `batteryLevel` _integer_          | _Optional._ Remaining battery level as an integer percentage of total battery capacity                                                                                                                                      |
| `batteryState` _string_           | _Optional._ Battery state for the device Must be one of: `unplugged`, `charging`, `full`,``                                                                                                                                 |
| `lowPowerMode` _boolean_          | _Optional._ A Boolean indicating whether Low Power Mode is enabled (iOS only)                                                                                                                                               |
| `availableStorage` _integer_      | _Optional._ Bytes of storage remaining                                                                                                                                                                                      |
| `totalStorage` _integer_          | _Optional._ Total size of storage in bytes                                                                                                                                                                                  |
| `isPortrait` _boolean_            | _Optional._ A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                                                                 |
| `resolution` _string_             | _Optional._ Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes                                                                               |
| `scale` _number_                  | _Optional._ Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android)                                                         |
| `language` _string_               | _Optional._ System language currently used on the device (ISO 639)                                                                                                                                                          |
| `appSetId` _string_               | _Optional._ Android vendor ID scoped to the set of apps published under the same Google Play developer account (see https\://developer.android.com/training/articles/app-set-id)                                            |
| `appSetIdScope` _string_          | _Optional._ Scope of the \`appSetId\`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) Must be one of: `app`, `developer`,`` |

## Client hints entity

[Client hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-CH) are an alternative to user-agent strings for capturing browser and device information. You can configure the JavaScript tracker to automatically include a client hints entity with all tracked events.

| Tracker                                                         | Supported | Since version | Auto-tracking |
| --------------------------------------------------------------- | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/client-hints/) | ✅         | 3.0.0         | ✅             |
| iOS                                                             | ❌         |               |               |
| Android                                                         | ❌         |               |               |
| React Native                                                    | ❌         |               |               |
| Flutter                                                         | ❌         |               |               |
| Roku                                                            | ❌         |               |               |
| Google Tag Manager                                              | ❌         |               |               |

### `http_client_hints`

**Type:** Entity

Schema for HTTP Client Hints context

**Schema:** `iglu:org.ietf/http_client_hints/jsonschema/1-0-0`

**Example:**

```json
{
  "isMobile": false,
  "brands": [
    {
      "brand": "Google Chrome",
      "version": "89"
    },
    {
      "brand": "Chromium",
      "version": "89"
    }
  ]
}
```

**Properties:**

| Property                   | Description                                           |
| -------------------------- | ----------------------------------------------------- |
| `isMobile` _boolean_       | _Optional._ Whether the browser is on a mobile device |
| `brands` _array_           | _Optional._ List of browser brands and versions       |
| `architecture` _string_    | _Optional._ CPU architecture (high entropy)           |
| `model` _string_           | _Optional._ Device model (high entropy)               |
| `platform` _string_        | _Optional._ Operating system name (high entropy)      |
| `platformVersion` _string_ | _Optional._ Operating system version (high entropy)   |
| `uaFullVersion` _string_   | _Optional._ Full browser version (high entropy)       |

## Entities added during enrichment

You can configure your pipeline to add these entities to tracked web events.

### User agent parsing

The [YAUAA (Yet Another User Agent Analyzer) enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enables parsing the user agent string tracked in web events. It extract information about the user's device and browser.

### `yauaa_context`

**Type:** Entity

Schema for a context generated by the YAUAA enrichment after parsing the user agent

**Schema:** `iglu:nl.basjes/yauaa_context/jsonschema/1-0-4`

**Example:**

```json
{
  "schema_name": "yauaa",
  "agent_class": "Browser",
  "agent_information_email": "Unknown",
  "agent_name": "Chrome",
  "agent_name_version": "Chrome109",
  "agent_name_version_major": "Chrome109",
  "agent_version": "109",
  "agent_version_major": "109.00",
  "device_brand": "Apple",
  "device_class": "Desktop",
  "device_cpu": "Intel",
  "device_cpu_bits": "64",
  "device_name": "AppleMacintosh",
  "device_version": "Demo",
  "layout_engine_class": "Browser",
  "layout_engine_name": "Blink",
  "layout_engine_name_version": "Blink109",
  "layout_engine_name_version_major": "Blink109",
  "layout_engine_version": "109",
  "layout_engine_version_major": "109",
  "network_type": "Unknown",
  "operating_system_class": "Desktop",
  "operating_system_name": "MacOS",
  "operating_system_name_version": "MacOS>=10.15.7",
  "operating_system_name_version_major": "MacOS>=10.15",
  "operating_system_version": ">=10.15.7",
  "operating_system_version_build": "??",
  "operating_system_version_major": ">=10.15",
  "webview_app_name": "Unknown",
  "webview_app_name_version_major": "Unknown??",
  "webview_app_version": "??",
  "webview_app_version_major": "??"
}
```

**Properties:**

| Property                                   | Description                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `deviceClass` __                           | _Required._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Anonymized`, `Unknown`, `UNKNOWN`, `Mobile`, `Tablet`, `Phone`, `Watch`, `Virtual Reality`, `eReader`, `Set-top box`, `TV`, `Game Console`, `Home Appliance`, `Handheld Game Console`, `Voice`, `Car`, `Robot`, `Robot Mobile`, `Spy`, `Hacker`, `Augmented Reality`, `Robot Imitator` |
| `deviceName` _string_                      | _Optional._ Example: Google Nexus 6                                                                                                                                                                                                                                                                                                                                            |
| `deviceBrand` _string_                     | _Optional._ Example: Google                                                                                                                                                                                                                                                                                                                                                    |
| `deviceCpu` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceCpuBits` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceFirmwareVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `deviceVersion` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemClass` _string_            | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Desktop`, `Mobile`, `Cloud`, `Embedded`, `Game Console`, `Hacker`, `Anonymized`, `Unknown`                                                                                                                                                                                                        |
| `operatingSystemName` _string_             | _Optional._ Examples: Linux, Android.                                                                                                                                                                                                                                                                                                                                          |
| `operatingSystemVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionBuild` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineClass` _string_               | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Mobile App`, `Hacker`, `Robot`, `Unknown`, `Special`, `Cloud`, `eReader`                                                                                                                                                                                                               |
| `layoutEngineName` _string_                | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersion` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineVersionMajor` _string_        | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersion` _string_         | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineNameVersionMajor` _string_    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `layoutEngineBuild` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentClass` _string_                      | _Optional._ See https\://yauaa.basjes.nl/README-Output.html Must be one of: `Browser`, `Browser Webview`, `Mobile App`, `Robot`, `Robot Mobile`, `Cloud Application`, `Email Client`, `Voice`, `Special`, `Testclient`, `Hacker`, `Unknown`, `Desktop App`, `eReader`                                                                                                          |
| `agentName` _string_                       | _Optional._ Example: Chrome.                                                                                                                                                                                                                                                                                                                                                   |
| `agentVersion` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentVersionMajor` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentNameVersion` _string_                | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentNameVersionMajor` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentBuild` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentLanguage` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentLanguageCode` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentInformationEmail` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentInformationUrl` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `agentSecurity` _string_                   | _Optional._ Must be one of: `Weak security`, `Strong security`, `Unknown`, `Hacker`, `No security`                                                                                                                                                                                                                                                                             |
| `agentUuid` _string_                       | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppName` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersion` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppVersionMajor` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `webviewAppNameVersionMajor` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookCarrier` _string_                 | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceClass` _string_             | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceName` _string_              | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookDeviceVersion` _string_           | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookFBOP` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookFBSS` _string_                    | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemName` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `facebookOperatingSystemVersion` _string_  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `anonymized` _string_                      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `hackerAttackVector` _string_              | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `hackerToolkit` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `koboAffiliate` _string_                   | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `koboPlatformId` _string_                  | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersion` _string_          | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersion` _string_      | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `iECompatibilityNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `carrier` _string_                         | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `gSAInstallationID` _string_               | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `networkType` _string_                     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemNameVersionMajor` _string_ | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |
| `operatingSystemVersionMajor` _string_     | _Optional._                                                                                                                                                                                                                                                                                                                                                                    |

### Spiders and robots

The [IAB Spiders and Robots enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) uses the IAB/ABC International Spiders and Bots List to determine whether an event was produced by a user or a robot/spider based on its IP address and user agent.

### `spiders_and_robots`

**Type:** Entity

Schema for a context generated by the IAB Spiders & Robots enrichment

**Schema:** `iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0`

**Example:**

```json
{
  "category": "BROWSER",
  "primaryImpact": null,
  "reason": "PASSED_ALL",
  "spiderOrRobot": false
}
```

**Properties:**

| Property                  | Description                                                                                                                                                                                                                  |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spiderOrRobot` _boolean_ | _Required._ true if the IP address or user agent checked against the list is a spider or robot, false otherwise                                                                                                              |
| `category` __             | _Required._ Category based on activity if the IP/UA is a spider or robot, BROWSER otherwise Must be one of: `SPIDER_OR_ROBOT`, `ACTIVE_SPIDER_OR_ROBOT`, `INACTIVE_SPIDER_OR_ROBOT`, `BROWSER`                               |
| `reason` __               | _Required._ Type of failed check if the IP/UA is a spider or robot, PASSED\_ALL otherwise Must be one of: `FAILED_IP_EXCLUDE`, `FAILED_UA_INCLUDE`, `FAILED_UA_EXCLUDE`, `PASSED_ALL`                                        |
| `primaryImpact` __        | _Required._ Whether the spider or robot would affect page impression measurement, ad impression measurement, both or none Must be one of: `PAGE_IMPRESSIONS`, `AD_IMPRESSIONS`, `PAGE_AND_AD_IMPRESSIONS`, `UNKNOWN`, `NONE` |

---

# Ecommerce events and entities
> Track ecommerce transactions, carts, checkouts, products, and promotions with Snowplow ecommerce events and context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/ecommerce-events/

The Snowplow ecommerce tracking APIs enable you to track user behavior and interactions across your ecommerce store. Track product views, cart interactions, checkout steps, transactions, promotions, and more.

Use ecommerce tracking to answer questions such as:

- Which products do users view but not add to cart?
- Where do users drop off in the checkout process?
- How do promotions affect purchase behavior?
- What is the average order value by user segment?
- Which products are frequently purchased together?

## Tracker support

This table shows the support for ecommerce tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers can send ecommerce events as custom [self-describing events](/docs/fundamentals/events/#self-describing-events).

| Tracker                                                                        | Supported | Since version | Auto-tracking | Notes                                                |
| ------------------------------------------------------------------------------ | --------- | ------------- | ------------- | ---------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ecommerce/)                   | ✅         | 3.7.0         | ❌             | Requires ecommerce plugin                            |
| [iOS](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)       | ✅         | 5.4.0         | ❌             |                                                      |
| [Android](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)   | ✅         | 5.4.0         | ❌             |                                                      |
| [React Native](/docs/sources/react-native-tracker/)                            | ✅         | 4.2.0         | ❌             | Requires ecommerce plugin\*                          |
| Flutter                                                                        | ❌         |               |               |                                                      |
| Roku                                                                           | ❌         |               |               | Use the ecommerce schemas for your own custom events |
| [Google Tag Manager](/docs/sources/google-tag-manager/ecommerce-tag-template/) | ✅         | v3            | ❌             |                                                      |

\*You can use the [JavaScript ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/) and APIs for React Native ecommerce tracking.

The Snowplow ecommerce tracking APIs are supported by the [Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) dbt data model. It's a fully incremental model that transforms raw ecommerce event data into a set of derived tables based around carts, checkouts, products, and transactions.

We recommend using the [Ecommerce base tracking plan template](/docs/event-studio/tracking-plans/templates/#ecommerce-web-and-mobile) for ecommerce tracking.

## Available events

The ecommerce APIs include a number of tracking functions for different ecommerce actions. The exact API varies between trackers, but the available events are consistent across all supported trackers.

| Event behavior     | Used for                                                                                                                                       |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Product view       | Tracking a visit to a product detail screen. Also known as product detail view.                                                                |
| Add to cart        | Track an addition to cart.                                                                                                                     |
| Remove from cart   | Track a removal from cart.                                                                                                                     |
| Product list view  | Track an impression of a product list. The list could be a search results page, recommended products, upsells etc.                             |
| Product list click | Track the click/selection of a product from a product list.                                                                                    |
| Promotion view     | Track an impression for an internal promotion banner or slider or any other type of content that showcases internal products/categories.       |
| Promotion click    | Track the click/selection of an internal promotion.                                                                                            |
| Checkout step      | Track a checkout step completion in the checkout process together with common step attributes for user choices throughout the checkout funnel. |
| Transaction        | Track a transaction/purchase completion.                                                                                                       |
| Transaction error  | Track a failed transaction.                                                                                                                    |
| Refund             | Track a transaction partial or complete refund.                                                                                                |

All these events use the same underlying schema. They're distinguished by their `type` property.

### `snowplow_ecommerce_action`

**Type:** Event

Schema for an Ecommerce action

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/snowplow_ecommerce_action/jsonschema/1-0-2`

**Example:**

```json
{
  "type": "list_view",
  "name": "shop the look"
}
```

**Properties:**

| Property        | Description                                                                                                                                                                                                               |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` __       | _Required._ Standard ecommerce actions. Must be one of: `add_to_cart`, `remove_from_cart`, `product_view`, `list_click`, `list_view`, `promo_click`, `promo_view`, `checkout_step`, `transaction`, `refund`, `trns_error` |
| `name` _string_ | _Optional._ The name of the list presented to the user E.g. product list, search results, shop the look, frequently bought with.                                                                                          |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1 snowplow_ecommerce_action_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'snowplow_ecommerce_action'
and events.event_vendor = 'com.snowplowanalytics.snowplow.ecommerce'
```

## Automatically included entities

Every ecommerce event includes entities that describe the user interaction. These entities are attached automatically by the tracker. You don't need to track them yourself.

### Cart

The cart entity captures the current state of the shopping cart, including its total value and currency.

### `cart`

**Type:** Entity

Schema for a cart entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/cart/jsonschema/1-0-0`

**Example:**

```json
{
  "cart_id": null,
  "currency": "EUR",
  "total_value": 12
}
```

**Properties:**

| Property               | Description                                                     |
| ---------------------- | --------------------------------------------------------------- |
| `cart_id` _string_     | _Optional._ The unique ID representing this cart.               |
| `total_value` _number_ | _Required._ The total value of the cart after this interaction. |
| `currency` _string_    | _Required._ The currency used for this cart (ISO 4217).         |

### Checkout step

The checkout step entity tracks information about a specific step in the checkout process, including delivery details, payment method, and user preferences.

### `checkout_step`

**Type:** Entity

Schema for a checkout step entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/checkout_step/jsonschema/1-0-0`

**Example:**

```json
{
  "step": 2,
  "account_type": "guest",
  "billing_full_address": null,
  "billing_postcode": null,
  "coupon_code": null,
  "delivery_method": null,
  "delivery_provider": null,
  "marketing_opt_in": null,
  "payment_method": null,
  "proof_of_payment": null,
  "shipping_full_address": null,
  "shipping_postcode": null
}
```

**Properties:**

| Property                         | Description                                                                                                                         |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `step` _integer_                 | _Required._ Checkout step index.                                                                                                    |
| `shipping_postcode` _string_     | _Optional._ Shipping address postcode.                                                                                              |
| `billing_postcode` _string_      | _Optional._ Billing address postcode.                                                                                               |
| `shipping_full_address` _string_ | _Optional._ Full shipping address.                                                                                                  |
| `billing_full_address` _string_  | _Optional._ Full billing address.                                                                                                   |
| `delivery_provider` _string_     | _Optional._ Can be used to discern delivery providers DHL, PostNL etc.                                                              |
| `delivery_method` _string_       | _Optional._ Can be used to discern delivery methods selected E.g. store pickup, standard delivery, express delivery, international. |
| `coupon_code` _string_           | _Optional._ Coupon applied at checkout.                                                                                             |
| `account_type` _string_          | _Optional._ Type of account used on checkout. E.g. existing user, guest.                                                            |
| `payment_method` _string_        | _Optional._ Any kind of payment method the user selected to proceed E.g. card, PayPal, Alipay etc.                                  |
| `proof_of_payment` _string_      | _Optional._ Invoice or receipt.                                                                                                     |
| `marketing_opt_in` _boolean_     | _Optional._ If opted in to marketing campaigns.                                                                                     |

### Product

The product entity contains detailed information about a product, including its ID, category, price, and other attributes such as brand, variant, and inventory status.

### `product`

**Type:** Entity

Schema for a product entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/product/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "1236",
  "name": null,
  "category": "Hats",
  "price": 12,
  "list_price": null,
  "quantity": null,
  "size": null,
  "variant": null,
  "brand": "Snowplow",
  "inventory_status": null,
  "position": null,
  "currency": "EUR",
  "creative_id": null
}
```

**Properties:**

| Property                    | Description                                                                                                                       |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `id` _string_               | _Required._ The SKU or product ID.                                                                                                |
| `name` _string_             | _Optional._ The name or title of the product.                                                                                     |
| `category` _string_         | _Required._ The category the product belongs to. Use a consistent separator to express multiple levels. E.g. Woman/Shoes/Sneakers |
| `price` _number_            | _Required._ The price of the product at the current time.                                                                         |
| `list_price` _number_       | _Optional._ The list or recommended retail price of a product.                                                                    |
| `quantity` _integer_        | _Optional._ The quantity of the product taking part in the ecommerce action.                                                      |
| `size` _string_             | _Optional._ The size of the product.                                                                                              |
| `variant` _string_          | _Optional._ The variant of the product.                                                                                           |
| `brand` _string_            | _Optional._ The brand of the product.                                                                                             |
| `inventory_status` _string_ | _Optional._ The inventory status of the product E.g. in stock, out of stock, preorder, backorder.                                 |
| `position` _integer_        | _Optional._ The position the product was presented in a list of products E.g. search results, product list page.                  |
| `currency` _string_         | _Required._ The currency in which the product is being priced (ISO 4217).                                                         |
| `creative_id` _string_      | _Optional._ Identifier/Name/Url for the creative presented on a list or product view.                                             |

### Promotion

The promotion entity tracks internal promotional content such as banners, sliders, or featured product showcases.

### `promotion`

**Type:** Entity

Schema for a promotion entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/promotion/jsonschema/1-0-0`

**Properties:**

| Property               | Description                                                                                                                     |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `id` _string_          | _Required._ The ID of the promotion.                                                                                            |
| `name` _string_        | _Optional._ The name of the promotion.                                                                                          |
| `product_ids` _array_  | _Optional._ Array of SKUs or product IDs showcased in the promotion.                                                            |
| `position` _integer_   | _Optional._ The position the promotion was presented in a list of promotions E.g. banner, slider.                               |
| `creative_id` _string_ | _Optional._ Identifier/Name/Url for the creative presented on the promotion.                                                    |
| `type` _string_        | _Optional._ Type of the promotion delivery mechanism. E.g. popup, banner, intra-content                                         |
| `slot` _string_        | _Optional._ The website slot in which the promotional content was added to. E.g. Identifier for slot sidebar-1, intra-content-2 |

### Refund

The refund entity captures information about a transaction refund, including the amount refunded and the reason for the refund.

### `refund`

**Type:** Entity

Schema for a refund in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/refund/jsonschema/1-0-0`

**Properties:**

| Property                  | Description                                                               |
| ------------------------- | ------------------------------------------------------------------------- |
| `transaction_id` _string_ | _Required._ The ID of the transaction.                                    |
| `currency` _string_       | _Required._ The currency in which the product is being priced (ISO 4217). |
| `refund_amount` _number_  | _Required._ The monetary amount refunded.                                 |
| `refund_reason` _string_  | _Optional._ Reason for refunding the whole or part of the transaction.    |

### Transaction

The transaction entity contains details about a completed purchase, including revenue, payment method, shipping costs, and applied discounts.

### `transaction`

**Type:** Entity

Schema for a transaction entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction/jsonschema/1-0-0`

**Example:**

```json
{
  "transaction_id": "TtlZ3b",
  "currency": "EUR",
  "payment_method": "BNPL",
  "revenue": 13,
  "total_quantity": 1,
  "credit_order": null,
  "discount_amount": null,
  "discount_code": null,
  "shipping": 1,
  "tax": null
}
```

**Properties:**

| Property                   | Description                                                   |
| -------------------------- | ------------------------------------------------------------- |
| `transaction_id` _string_  | _Required._ The ID of the transaction.                        |
| `revenue` _number_         | _Required._ The revenue of the transaction.                   |
| `currency` _string_        | _Required._ The currency used for the transaction (ISO 4217). |
| `payment_method` _string_  | _Required._ The payment method used for the transaction.      |
| `total_quantity` _integer_ | _Required._ Total quantity of items in the transaction.       |
| `tax` _number_             | _Optional._ Total amount of tax on the transaction.           |
| `shipping` _number_        | _Optional._ Total cost of shipping on the transaction.        |
| `discount_code` _string_   | _Optional._ Discount code used.                               |
| `discount_amount` _number_ | _Optional._ Discount amount taken off.                        |
| `credit_order` _boolean_   | _Optional._ Whether the transaction is a credit order or not. |

### Transaction error

The transaction error entity captures information about failed transactions, including error codes, descriptions, and error types.

### `transaction_error`

**Type:** Entity

Schema for a transaction error or rejection entity in ecommerce.

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction_error/jsonschema/1-0-0`

**Properties:**

| Property                     | Description                                                                                                                                                                                                                                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `error_code` _string_        | _Optional._ Error-identifying code for the transaction issue. E.g. E522                                                                                                                                                                                                   |
| `error_shortcode` _string_   | _Optional._ Shortcode for the error occurred in the transaction. E.g. declined\_by\_stock\_api, declined\_by\_payment\_method, card\_declined, pm\_card\_radarBlock                                                                                                       |
| `error_description` _string_ | _Optional._ Longer description for the error occurred in the transaction.                                                                                                                                                                                                 |
| `error_type` _string_        | _Optional._ Hard error types mean the customer must provide another form of payment e.g. an expired card. Soft errors can be the result of temporary issues where retrying might be successful e.g. processor declined the transaction. Must be one of: `hard`, `soft`,`` |
| `resolution` _string_        | _Optional._ The resolution selected for the error scenario. E.g. retry\_allowed, user\_blacklisted, block\_gateway, contact\_user, default                                                                                                                                |

## Global ecommerce entities

You can configure the ecommerce `user` and `page` entities to automatically attach to **all** Snowplow events tracked by the tracker, not just the ecommerce events. This is useful if you want to have ecommerce user and page information available across your entire dataset.

### Page

The page entity describes the type of ecommerce page being viewed, such as homepage, product page, or checkout page.

### `page`

**Type:** Entity

Schema for a page entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/page/jsonschema/1-0-0`

**Example:**

```json
{
  "type": "checkout step 1",
  "language": null,
  "locale": null
}
```

**Properties:**

| Property            | Description                                                                                   |
| ------------------- | --------------------------------------------------------------------------------------------- |
| `type` _string_     | _Required._ The type of the page that was visited E.g. homepage, product page, checkout page. |
| `language` _string_ | _Optional._ The language that the web page is based in.                                       |
| `locale` _string_   | _Optional._ The locale version of the site that is running.                                   |

### User

The user entity provides information about the user making the purchase, including their ID, email address, and whether they're a guest.

### `user`

**Type:** Entity

Schema for an user entity in Ecommerce

**Schema:** `iglu:com.snowplowanalytics.snowplow.ecommerce/user/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "U12345",
  "email": "john@email.com",
  "is_guest": true
}
```

**Properties:**

| Property             | Description                                     |
| -------------------- | ----------------------------------------------- |
| `id` _string_        | _Required._ The user ID.                        |
| `is_guest` _boolean_ | _Optional._ Whether or not the user is a guest. |
| `email` _string_     | _Optional._ The user's email address.           |

## Legacy ecommerce events

Some Snowplow trackers also provide tracking calls for older ecommerce events. We strongly recommend using the new ecommerce tracking API instead.

The legacy transaction events populate the [transaction atomic event properties](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields): `tr_` fields for transaction events and `ti_` fields for transaction item events.

Transaction event properties:

| **Parameter** | **Table Column** | **Type** | **Description**                                     | **Example values** |
| ------------- | ---------------- | -------- | --------------------------------------------------- | ------------------ |
| `tr_id`       | `tr_orderid`     | text     | Order ID                                            | `12345`            |
| `tr_af`       | `tr_affiliation` | text     | Transaction affiliation (e.g. channel)              | `Web`              |
| `tr_tt`       | `tr_total`       | decimal  | Transaction total value                             | `9.99`             |
| `tr_tx`       | `tr_tax`         | decimal  | Transaction tax value (i.e. amount of VAT included) | `1.98`             |
| `tr_sh`       | `tr_shipping`    | decimal  | Delivery cost charged                               | `3.00`             |
| `tr_ci`       | `tr_city`        | text     | Delivery address: city                              | `London`           |
| `tr_st`       | `tr_state`       | text     | Delivery address: state                             | `Denver`           |
| `tr_co`       | `tr_country`     | text     | Delivery address: country                           | `United Kingdom`   |
| `tr_cu`       | `tr_currency`    | text     | Transaction Currency                                | `GBP`              |

Transaction item event properties:

| **Parameter** | **Table Column** | **Type** | **Description** | **Example values** |
| ------------- | ---------------- | -------- | --------------- | ------------------ |
| `ti_id`       | `ti_orderid`     | text     | Order ID        | `12345`            |
| `ti_sk`       | `ti_sku`         | text     | Item SKU        | \`pbz0025'         |
| `ti_nm`       | `ti_name`        | text     | Item name       | `black-tarot`      |
| `ti_ca`       | `ti_category`    | text     | Item category   | `tarot`            |
| `ti_pr`       | `ti_price`       | decimal  | Item price      | `7.99`             |
| `ti_qu`       | `ti_quantity`    | integer  | Item quantity   | `2`                |
| `ti_cu`       | `ti_currency`    | text     | Currency        | `USD`              |

---

# Geolocation and timezone tracking
> Track user location using IP-based geolocation enrichment or device GPS coordinates with the geolocation context entity.
> Source: https://docs.snowplow.io/docs/events/ootb-data/geolocation/

Snowplow provides several ways to capture location and timezone information:

- IP-based geolocation: the IP Lookup enrichment adds location data to the `geo_` atomic event properties based on the user's IP address.
- Device geolocation: client-side trackers can access GPS coordinates from the device and attach them as an entity. This requires user permission but provides more accurate location data.
- Timezone: most trackers can capture the device timezone and add it to the `os_timezone` atomic event property.

## Location atomic event properties

The [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) populates the `geo_` [atomic event properties](/docs/fundamentals/canonical-event/#location-fields), based on the IP address of the HTTP request to the event Collector.

## Timezone atomic event property

Some Snowplow trackers can populate the `os_timezone` [atomic event field](/docs/fundamentals/canonical-event/#time-and-date-fields) based on the device timezone.

Depending on the tracker, you may need to configure a plugin, set configuration on tracker initialization, or use the tracker Subject class. This table shows the support for timezone tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                      | Supported | Since version |
| ---------------------------------------------------------------------------- | --------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)      | ✅         | 3.0.0         |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.1.0         |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.1.0         |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 0.1.0         |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.4.0         |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.4.0         |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.1.0         |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         |
| Lua                                                                          | ❌         |               |
| Google Tag Manager                                                           | ❌         |               |

## Geolocation entity

Some Snowplow trackers can access device geolocation information, e.g. GPS coordinates, and attach it to events as an entity. This requires permission from the user, but is often more accurate than the IP-based geolocation.

This table shows the support for geolocation tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include geolocation tracking.

| Tracker                                                                    | Supported | Since version | Auto-tracking | Notes                           |
| -------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)    | ✅         | 3.0.0         | ✅             | Requires timezone plugin        |
| [iOS](/docs/sources/mobile-trackers/installation-and-set-up/)              | ✅         | 0.6.0         | ✅             | Set in tracker configuration    |
| [Android](/docs/sources/mobile-trackers/installation-and-set-up/)          | ✅         | 0.6.0         | ✅             | Set in tracker configuration    |
| React Native                                                               | ❌         |               |               |                                 |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) | ✅         | 0.1.0         | ✅             | Set in tracker configuration    |
| Roku                                                                       | ❌         |               |               |                                 |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)  | ✅         | v3            | ❌             | Integrates with timezone plugin |

The React Native tracker did include geolocation entity configuration in earlier versions. We [deprecated it in version 4](/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/).

### `geolocation_context`

**Type:** Entity

Schema for client geolocation contexts

**Schema:** `iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-1-0`

**Example:**

```json
{
  "latitude": 30.04335623,
  "longitude": 67.59633102,
  "latitude_longitude_accuracy": -24902753.22,
  "altitude": -19459.88,
  "altitude_accuracy": -29970651.08,
  "bearing": 21055653.32,
  "speed": -7127794.98
}
```

**Properties:**

| Property                             | Description |
| ------------------------------------ | ----------- |
| `latitude` _number_                  | _Required._ |
| `longitude` _number_                 | _Required._ |
| `latitudeLongitudeAccuracy` _number_ | _Optional._ |
| `altitude` _number_                  | _Optional._ |
| `altitudeAccuracy` _number_          | _Optional._ |
| `bearing` _number_                   | _Optional._ |
| `speed` _number_                     | _Optional._ |
| `timestamp` _integer_                | _Optional._ |

---

# Out-of-the-box data types
> Overview of Snowplow's built-in event types and entities for web and mobile tracking including ecommerce, media, and performance data.
> Source: https://docs.snowplow.io/docs/events/ootb-data/

Snowplow has provided these out-of-the-box [event](/docs/fundamentals/events/) and [entity](/docs/fundamentals/entities/) types so you can get started quickly with tracking user data.

This table shows the available out-of-the-box data grouped by category.

| Category                                                                                   | Description                                                                         | Events | Entities | [Atomic fields](/docs/fundamentals/canonical-event/) |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | ------ | -------- | ---------------------------------------------------- |
| [Application errors](/docs/events/ootb-data/app-error-events/)                             | Exception messages, stack traces, and error classification                          | ✅      |          |                                                      |
| [Application and tracker information](/docs/events/ootb-data/app-information/)             | App version, build number, platform, and tracker namespace                          |        | ✅        | ✅                                                    |
| [Application performance](/docs/events/ootb-data/app-performance/)                         | Web vitals, navigation timing, and performance metrics                              | ✅      | ✅        |                                                      |
| [Buttons](/docs/events/ootb-data/page-elements/)                                           | Button click events with label, ID, and element classes                             | ✅      |          |                                                      |
| [Campaigns and ads](/docs/events/ootb-data/campaigns-and-ads/)                             | UTM parameters, ad impressions, clicks, and conversions                             | ✅      |          | ✅                                                    |
| [Consent](/docs/events/ootb-data/consent-events/)                                          | GDPR consent and CMP visibility tracking                                            | ✅      | ✅        |                                                      |
| [Device and browser information](/docs/events/ootb-data/device-and-browser/)               | Track data such as language, viewport size, screen resolution, or client hints      |        | ✅        | ✅                                                    |
| [Ecommerce](/docs/events/ootb-data/ecommerce-events/)                                      | Complete ecommerce tracking including product views, cart actions, and transactions | ✅      | ✅        | ✅                                                    |
| [Forms](/docs/events/ootb-data/page-elements/)                                             | Form focus, field changes, and submission with field values                         | ✅      |          |                                                      |
| [Geolocation](/docs/events/ootb-data/geolocation/)                                         | IP-based location (city, region, country) and GPS coordinates                       |        | ✅        | ✅                                                    |
| [Referrers, links, and cross-navigation](/docs/events/ootb-data/links-and-referrers/)      | Page referrer URLs, link clicks, and mobile deep links                              | ✅      | ✅        | ✅                                                    |
| [Media](/docs/events/ootb-data/media-events/)                                              | Complete video or audio tracking, including play, pause, and ad breaks              | ✅      | ✅        |                                                      |
| [Mobile application lifecycle](/docs/events/ootb-data/mobile-lifecycle-events/)            | App install, foreground and background                                              | ✅      | ✅        |                                                      |
| [Page and screen engagement](/docs/events/ootb-data/page-activity-tracking/)               | Page pings, scroll depth, and time engaged on page or screen                        | ✅      | ✅        | ✅                                                    |
| [Page and screen views](/docs/events/ootb-data/page-and-screen-view-events/)               | Track page and screen views                                                         | ✅      | ✅        | ✅                                                    |
| [Page elements](/docs/events/ootb-data/page-elements/)                                     | Element visibility, scroll depth into elements, and lifecycle                       | ✅      | ✅        |                                                      |
| [Site search](/docs/events/ootb-data/site-search/)                                         | Search terms, filters applied, and result counts                                    | ✅      |          |                                                      |
| [Social media interactions](/docs/events/ootb-data/social-media/)                          | Likes, shares, and social widget interactions by network                            | ✅      |          |                                                      |
| [Third-party sources](/docs/events/ootb-data/third-party-sources/)                         | Optimizely experiments, GA cookies, Kantar Focal Meter, and Privacy Sandbox topics  |        | ✅        |                                                      |
| [Timezone](/docs/events/ootb-data/geolocation/)                                            | Track device timezone                                                               |        |          | ✅                                                    |
| [User and session identification](/docs/events/ootb-data/user-and-session-identification/) | Track identifiers such as user ID, domain user ID, session ID, and session index    |        | ✅        | ✅                                                    |
| [VisionOS and SwiftUI](/docs/events/ootb-data/visionos-swiftui/)                           | Window group and immersive space tracking for Apple                                 | ✅      | ✅        |                                                      |

---

# Referrers, links, and cross-navigation events
> Track page referrers, deep links in mobile apps, and cross-navigation links.
> Source: https://docs.snowplow.io/docs/events/ootb-data/links-and-referrers/

Understanding how users arrive at your pages and navigate between them is essential for analyzing traffic sources, marketing effectiveness, and user journeys. Snowplow captures referrer information, deep link data, and link click events to give you a complete picture of cross-page and cross-app navigation.

## Referrer atomic event property

Some Snowplow trackers can populate the `page_referrer` [atomic event property](/docs/fundamentals/canonical-event/#page-fields). This field captures the URL of the page that referred the user to the current page.

This table shows the support for page referrer tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include referrer tracking.

| Tracker                                                                                                | Supported | Since version                                      | Auto-tracking | Notes                              |
| ------------------------------------------------------------------------------------------------------ | --------- | -------------------------------------------------- | ------------- | ---------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/page-views/)                                          | ✅         | 0.1.0                                              | ✅             | Automatic for all page view events |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                              | ✅         | 3.0.0 (deep link events), 4.1.0 (deep link entity) | ❌             | Based on Deep Link events          |
| [Android](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                          | ✅         | 3.0.0 (deep link events), 4.1.0 (deep link entity) | ❌             | Based on Deep Link events          |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-deep-link-received-events) | ✅         | 1.1.0                                              | ❌             | Based on Deep Link events          |
| Flutter                                                                                                | ❌         |                                                    |               |                                    |
| Roku                                                                                                   | ❌         |                                                    |               |                                    |
| Google Tag Manager                                                                                     | ✅         | v3                                                 | ✅             | Automatic for all page view events |

The web trackers automatically populate the `page_referrer` [atomic event property](/docs/fundamentals/canonical-event/#page-fields) in all page view events.

### Deep links for mobile

Deep links are URLs or hyperlinks that take users directly to a particular location within a mobile app. They're received by the mobile operating system and passed to the related app.

Most events tracked on mobile unsurprisingly don't include webpage atomic properties. However, the mobile trackers can capture both the URL of the deep link and the referrer URL (if available).

You'll need to manually track any received deep links, using the `deep_link_received` event.

The `deep_link_received` event, as well as the first subsequent screen view event, include the `page_url` and `page_referrer` atomic event properties.

### `deep_link_received`

**Type:** Event

Represents a deep-link received in the app.

**Schema:** `iglu:com.snowplowanalytics.mobile/deep_link_received/jsonschema/1-0-0`

**Example:**

```json
{
  "url": "https://example.com/notes/123",
  "referrer": "https://snowplow.io"
}
```

**Properties:**

| Property            | Description                                        |
| ------------------- | -------------------------------------------------- |
| `url` _string_      | _Required._ URL in the received deep-link          |
| `referrer` _string_ | _Optional._ Referrer URL, source of this deep-link |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_deep_link_received_1 deep_link_received_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'deep_link_received'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

The tracker will automatically add this `deep_link` entity to the next screen view event after tracking the `deep_link_received` event.

### `deep_link`

**Type:** Entity

Entity that indicates a deep-link has been received and processed.

**Schema:** `iglu:com.snowplowanalytics.mobile/deep_link/jsonschema/1-0-0`

**Example:**

```json
{
  "url": "https://example.com/notes/123",
  "referrer": "https://snowplow.io"
}
```

**Properties:**

| Property            | Description                                        |
| ------------------- | -------------------------------------------------- |
| `url` _string_      | _Required._ URL in the received deep-link          |
| `referrer` _string_ | _Optional._ Referrer URL, source of this deep-link |

## Link clicks

The web trackers can automatically track link click events, capturing details about the clicked link such as its URL, element ID, classes, target, and content.

This table shows the support for link click tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include link click tracking.

| Tracker                                                                   | Supported | Since version                                 | Auto-tracking | Notes                             |
| ------------------------------------------------------------------------- | --------- | --------------------------------------------- | ------------- | --------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/link-click/)             | ✅         | 0.7.0 (limited functionality), 3.0.0 (plugin) | ✅             | Requires link click plugin        |
| iOS                                                                       | ❌         |                                               |               |                                   |
| Android                                                                   | ❌         |                                               |               |                                   |
| React Native                                                              | ❌         |                                               |               |                                   |
| Flutter                                                                   | ❌         |                                               |               |                                   |
| Roku                                                                      | ❌         |                                               |               |                                   |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                            | ✅             | Integrates with link click plugin |

To track link clicks on web using React Native or Flutter, you can implement custom event tracking when users interact with links in your app using the `link_click` event schema.

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes link clicks.

### `link_click`

**Type:** Event

Schema for a link click event

**Schema:** `iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1`

**Properties:**

| Property                  | Description |
| ------------------------- | ----------- |
| `elementId` _string_      | _Optional._ |
| `elementClasses` _array_  | _Optional._ |
| `elementTarget` _string_  | _Optional._ |
| `targetUrl` _string_      | _Required._ |
| `elementContent` _string_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_link_click_1 link_click_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'link_click'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Cross-domain tracking

Read more about tracking cross-domain in the [cross-navigation page](/docs/events/cross-navigation/).

---

# Media playback
> Track video and audio playback with media player events covering playback, buffering, quality changes, and ad tracking.
> Source: https://docs.snowplow.io/docs/events/ootb-data/media-events/

The Snowplow media tracking APIs enable you to track events from media playback on the web as well as in mobile apps. Track changes in the media playback e.g., play, pause, seek events, playback position with ping and percentage progress events, or ad playback events e.g., ad breaks, ad progress, or ad clicks.

While some trackers provide additional integrations with media players, the media tracking APIs are designed to be player agnostic.

Use media tracking to answer questions such as:

- Are users watching videos without sound?
- Which pieces of audio content do users play to completion?
- Where do users drop off during playback?
- How does buffering affect engagement?
- Which ads do users skip, click, or watch?

The media tracking works with a set of out-of-the-box events and entities. You can add custom entities to the events as required.

Media events fall into three categories: playback lifecycle events, player state changes, and advertising events. For most implementations, you'll need to track these events yourself.

Many media events have no properties of their own. The user behavior is captured in the event type, and the media player, session, ad, or ad break entities that are automatically attached to the event.

The event schema URIs have the format: `iglu:com.snowplowanalytics.snowplow.media/{EVENT_TYPE}/jsonschema/1-0-0`.

## Media API versions

The current set of media APIs has evolved from an earlier, more limited implementation. We refer to them as v1 and v2 of the media tracking APIs. Unless otherwise stated, all documentation refers to v2 of the media tracking APIs.

The [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt data model supports both versions since version 0.6.0. The v2 schemas are the default.

The newer v2 implementation has many more features than v1, including:

- Advertising tracking support
- Media ping events at configurable intervals
- Session entity with aggregated statistics
- Quality change tracking for bitrate, FPS, and quality level
- Buffer and seek start/end event pairs
- Filtering of repeated events (seek, volume)
- Page activity updates during playback

The v1 implementation used a single `media_player_event` schema for all events, distinguishing between actions using the `type` field. See the [Version 1 schemas](#version-1-schemas) section at the end of this page for details. The v2 implementation uses separate schemas for each event type.

## Tracker support

This table shows the support for media tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include media tracking APIs.

Auto-tracking means that the tracker can subscribe to and track events from a registered media player, without you needing to manually track each event.

| Tracker                                                                  | Plugin                                                                 | Supported | Since version                      | Auto-tracking | Notes                                                                                                                                               |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------- | --------- | ---------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Web                                                                      | [Snowplow](/docs/sources/web-trackers/tracking-events/media/snowplow/) | ✅         | 3.12.0 (Media v2)                  | ❌             |                                                                                                                                                     |
| Web                                                                      | [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/)       | ✅         | 3.2.0 (Media v1), 4.0.0 (Media v2) | ✅             |                                                                                                                                                     |
| Web                                                                      | [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/)   | ✅         | 3.2.0 (Media v1), 4.0.0 (Media v2) | ✅             |                                                                                                                                                     |
| Web                                                                      | [Vimeo](/docs/sources/web-trackers/tracking-events/media/vimeo/)       | ✅         | 3.14.0 (Media v2)                  | ✅             |                                                                                                                                                     |
| [iOS](/docs/sources/mobile-trackers/tracking-events/media-tracking/)     |                                                                        | ✅         | 5.3.0 (Media v2)                   | ✅/❌           | Auto-tracking for AVPlayer                                                                                                                          |
| [Android](/docs/sources/mobile-trackers/tracking-events/media-tracking/) |                                                                        | ✅         | 5.3.0 (Media v2)                   | ❌             |                                                                                                                                                     |
| React Native                                                             |                                                                        | ❌         |                                    |               | Use the media schemas for your own custom events, or the [media web tracking plan template](/docs/event-studio/tracking-plans/templates/#media-web) |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/media-tracking/) |                                                                        | ✅         | 0.7.0 (Media v2)                   | ❌             |                                                                                                                                                     |
| [Roku](/docs/sources/roku-tracker/media-tracking/)                       |                                                                        | ✅         | 0.1.0 (Media v1), 0.3.0 (Media v2) | ✅             |                                                                                                                                                     |
| Google Tag Manager                                                       | ❌                                                                      |           |                                    |               |                                                                                                                                                     |

The Snowplow media tracking APIs are supported by the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt data model.

We recommend using the [Media base tracking plan template](/docs/event-studio/tracking-plans/templates/#media-web) for media tracking on web.

## How to use the media APIs

Each media player instance needs to be registered with the tracker, using a unique session ID. Set the media session ID to a random string. In the browser, you could generate this with a function such as `crypto.randomUUID()`. The ID should be unique to a single session within a single player instance.

![Diagram showing media session timeline and which events to track](/assets/images/timeline_diagram-2fee2b04dd6ba2164e320661e324ae3b.svg)

> **Info:** Check out our [example React application](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media) to see tracked media events and entities live as you watch a video.
>
> The source code for the app is [available here](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

### Start a session

To start tracking user behavior involving that player, call the appropriate `startMediaTracking` function. The exact API is different for each tracker. Call `startMediaTracking` as soon as the player has loaded, not when playback begins. This ensures that session tracking begins immediately, and allows for accurate measurement of metrics such as buffering time or time to first frame.

A media session should correspond to a single media playback, including any ads that play before, during, or after the media. Metrics are collected against a media session, so it's important to set it correctly.

### Configure the player

You can provide additional player information and configuration when calling `startMediaTracking`, as well as in subsequent tracking calls.

We recommend providing a human-readable name for the media content as a `label` property. It'll be tracked as part of the media player entity. The label helps you quickly identify which content a media session relates to during analysis, and is used in combination with the media type, player type, and player ID to create the media stats table with the [Media Player dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/).

### Track playback

Track user interactions with the appropriate media tracking calls. Call `trackMediaPlay` when the main media content starts playing, even if there are pre-roll ads. The media session model accounts for both total play time and play time excluding ads.

We recommend continually updating the media player state by calling `updateMediaTracking` or equivalent `update` function **every second**. This function does not send events to the Collector, but updates internal attributes to ensure accurate metrics for the next event.

How you handle livestreams and playlists depends on whether you want to analyze them as a single session or split by content:

- **Analyze as a single session**: update the player label using `updateMediaTracking` when the content changes. This won't split the session in the data model.
- **Analyze as separate sessions per content segment**: end and restart the media session each time the content changes. Customize `media_session_id` to include segments, for example, `livestream-1`, `livestream-2` or `playlist-1`, `playlist-2`, and aggregate later using a shared prefix or a custom `livestream_id` / `playlist_id`.

### Filter events

You don't need to send the media ping events, or any other tracked media events, to the data warehouse. The media APIs include a configuration option to filter out certain events, or to keep only specified events. Filtered events aren't sent to the Collector.

To drop unwanted tracked events within the pipeline, use a [custom JavaScript enrichment](//docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/index.md#discarding-the-event).

### End a session

When playback ends, call the `endMediaTracking` function to end the session, clean up event listeners, and stop any background pings. Call this when you want to reset session stats, even if the media wasn't fully watched. It does not fire any events.

On web, if the user closes the browser or tab, you don't need to call `endMediaTracking` — pings stop automatically. In single-page applications, if `endMediaTracking` is missed, you may see one or two additional ping events, up to the `maxPausedPings` limit.

There's also a `trackMediaEnd` function that you can call to track when the media reaches 100% progress (fully watched). This is tracked as a media event. It's common to see sessions without a `trackMediaEnd` event if the user didn't finish the content.

If the content finishes and the user rewinds, continue the same session. Use the `timePlayed` and `contentWatched` metrics within the media session entity to understand rewatches compared to how much of the media content they watched.

## Playback lifecycle events

These events track the core playback flow from start to finish.

### Media pings

Media ping events are events sent in a regular interval while media tracking is active. They inform about the current state of the media playback, and keep the media session alive.

By default, ping events are sent every 30 seconds. They're sent in an interval that is unrelated to the media playback. However, to prevent sending too many events, there is a limit to how many ping events can be sent while the media is paused. By default, this is set to 1. You can change this by configuring `maxPausedPings` (not available on Roku).

### `ping_event`

**Type:** Event

Media player event fired periodically during main content playback, regardless of other API events that have been sent.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ping_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ping_event_1 ping_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ping_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Buffer end

Track a buffering end event when the the player finishes buffering content and resumes playback.

### `buffer_end_event`

**Type:** Event

Media player event fired when the the player finishes buffering content and resumes playback.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/buffer_end_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_buffer_end_event_1 buffer_end_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'buffer_end_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Buffer start

Track a buffering start event when the player goes into the buffering state and begins to buffer content.

The tracker will calculate the time since this event until a buffer end event, play event, or a change in playback position. It will add the duration to the `timeBuffering` property in the media session entity.

### `buffer_start_event`

**Type:** Event

Media player event fired when the player goes into the buffering state and begins to buffer content.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/buffer_start_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_buffer_start_event_1 buffer_start_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'buffer_start_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Pause event

Track a pause event when the user pauses the playback.

Tracking this event will automatically set the `paused` property in the media player entity to `true`.

### `pause_event`

**Type:** Event

Media player event sent when the user pauses the playback.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/pause_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_pause_event_1 pause_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'pause_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Playback end

Track a playback end event when playback stops, either when the end of the media is reached, or because no further data is available.

Tracking this event will automatically set the `ended` and `paused` properties in the media player entity to `true`.

### `end_event`

**Type:** Event

Media player event sent when playback stops when end of the media is reached or because no further data is available.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/end_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_end_event_1 end_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'end_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Percent progress

The tracker will track percentage progress events automatically, when the playback reaches configured percentage boundaries.

### `percent_progress_event`

**Type:** Event

Media player event fired when a percentage boundary set in tracking options is reached

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/percent_progress_event/jsonschema/1-0-0`

**Properties:**

| Property                    | Description                                          |
| --------------------------- | ---------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the media |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_percent_progress_event_1 percent_progress_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'percent_progress_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Play

Track a play event when the player changes state to playing from previously being paused.

Tracking this event will automatically set the `paused` property in the media player entity to `false`.

### `play_event`

**Type:** Event

Media player event sent when the player changes state to playing from previously being paused.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/play_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_play_event_1 play_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'play_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Seek end

Track a seek end event when a seek operation completes.

### `seek_end_event`

**Type:** Event

Media player event sent when a seek operation completes.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/seek_end_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_seek_end_event_1 seek_end_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'seek_end_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Seek start

Track a seek start event when a seek operation begins.

### `seek_start_event`

**Type:** Event

Media player event sent when a seek operation begins.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/seek_start_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_seek_start_event_1 seek_start_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'seek_start_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Player state change events

These events track changes to the player's display mode, quality, and settings.

### Error

Track an error event when the resource couldn't be loaded due to an error.

### `error_event`

**Type:** Event

Media player event tracked when the resource could not be loaded due to an error.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/error_event/jsonschema/1-0-0`

**Example:**

```json
{
  "errorCode": "E522",
  "errorName": "forbidden",
  "errorDescription": "Playback failed"
}
```

**Properties:**

| Property                    | Description                                                                          |
| --------------------------- | ------------------------------------------------------------------------------------ |
| `errorCode` _string_        | _Optional._ Error-identifying code for the playback issue. E.g. E522                 |
| `errorName` _string_        | _Optional._ Name for the type of error that occurred in the playback. E.g. forbidden |
| `errorDescription` _string_ | _Optional._ Longer description for the error that occurred in the playback.          |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_error_event_1 error_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'error_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Fullscreen change

Track a fullscreen change event when the media player fullscreen changes, fired immediately after the browser switches into or out of full-screen mode.

The `fullscreen` value is passed when tracking the event, and is automatically updated in the `player` entity.

### `fullscreen_change_event`

**Type:** Event

Media player event fired immediately after the browser switches into or out of full-screen mode.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/fullscreen_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "fullscreen": true
}
```

**Properties:**

| Property               | Description                                                          |
| ---------------------- | -------------------------------------------------------------------- |
| `fullscreen` _boolean_ | _Required._ Whether the video element is fullscreen after the change |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_fullscreen_change_event_1 fullscreen_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'fullscreen_change_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Picture-in-picture change

Track a picture-in-picture change event immediately after the platform switches into or out of picture-in-picture mode.

The `pictureInPicture` value is passed when tracking the event, and is automatically updated in the `player` entity.

### `picture_in_picture_change_event`

**Type:** Event

Media player event fired immediately after the browser switches into or out of picture-in-picture mode.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/picture_in_picture_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "pictureInPicture": true
}
```

**Properties:**

| Property                     | Description                                                                           |
| ---------------------------- | ------------------------------------------------------------------------------------- |
| `pictureInPicture` _boolean_ | _Required._ Whether the video element is showing picture-in-picture after the change. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_picture_in_picture_change_event_1 picture_in_picture_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'picture_in_picture_change_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Playback rate change

Track a playback rate change event when the playback rate has changed.

The `previousRate` is set automatically based on the last `playbackRate` value in the `player` entity. The `newRate` is passed when tracking the event, and is automatically updated in the `player` entity.

### `playback_rate_change_event`

**Type:** Event

Media player event sent when the playback rate has changed.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/playback_rate_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "previousRate": 1,
  "newRate": 1.5
}
```

**Properties:**

| Property                | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `previousRate` _number_ | _Optional._ Playback rate before the change (1 is normal) |
| `newRate` _number_      | _Required._ Playback rate after the change (1 is normal)  |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_playback_rate_change_event_1 playback_rate_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'playback_rate_change_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Quality change

Track a quality change event when the video playback quality changes.

The `previousQuality` is set automatically based on the last `quality` value in the `player` entity. The `newQuality` is passed when tracking the event, and is automatically updated in the `player` entity.

### `quality_change_event`

**Type:** Event

Media player event tracked when the video playback quality changes automatically.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/quality_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "previousQuality": "1080p",
  "newQuality": "720p",
  "bitrate": 1500,
  "framesPerSecond": 30,
  "automatic": true
}
```

**Properties:**

| Property                    | Description                                                            |
| --------------------------- | ---------------------------------------------------------------------- |
| `previousQuality` _string_  | _Optional._ Quality level before the change (e.g., 1080p).             |
| `newQuality` _string_       | _Optional._ Quality level after the change (e.g., 1080p).              |
| `bitrate` _integer_         | _Optional._ The current bitrate in bits per second.                    |
| `framesPerSecond` _integer_ | _Optional._ The current number of frames per second.                   |
| `automatic` _boolean_       | _Optional._ Whether the change was automatic or triggered by the user. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_quality_change_event_1 quality_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'quality_change_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ready

Track a ready event when the media tracking is successfully attached to the player and can track events.

### `ready_event`

**Type:** Event

Media player event fired when the media tracking is successfully attached to the player and can track events.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ready_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ready_event_1 ready_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ready_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Volume change

Track a volume change event when the user changes the volume.

The `previousVolume` is set automatically based on the last `volume` value in the `player` entity. The `newVolume` is passed when tracking the event, and is automatically updated in the `player` entity.

### `volume_change_event`

**Type:** Event

Media player event sent when the volume has changed.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/volume_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "previousVolume": 30,
  "newVolume": 50
}
```

**Properties:**

| Property                   | Description                                      |
| -------------------------- | ------------------------------------------------ |
| `previousVolume` _integer_ | _Optional._ Volume percentage before the change. |
| `newVolume` _integer_      | _Required._ Volume percentage after the change.  |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_volume_change_event_1 volume_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'volume_change_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Advertising events

These events track ad playback for monetized content. Ad events include the media ad and media ad break entities.

Tracking only `trackMediaAdStart` and `trackMediaAdComplete`, or equivalent API, is sufficient if you don't need to attribute ads to specific ad breaks.

If you skip `trackMediaAdBreakStart` and `trackMediaAdBreakEnd`, you won't have details like pod size and break ID to analyze all the ads within a break.

### Ad click

Track an ad click event when a user clicks on an ad.

Tracking this event will increase the counter of `adsClicked` in the session entity.

### `ad_click_event`

**Type:** Event

Media player event fired when the user clicked on the ad.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_click_event/jsonschema/1-0-0`

**Example:**

```json
{
  "percentProgress": 50
}
```

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_click_event_1 ad_click_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_click_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad break start

Track an ad break start event at the start of an ad break.

Tracking this event will increase the counter of `adBreaks` in the media session entity.

### `ad_break_start_event`

**Type:** Event

Media player event that signals the start of an ad break.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_break_start_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_break_start_event_1 ad_break_start_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_break_start_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad break end

Track an ad break end event to signal the end of an ad break.

### `ad_break_end_event`

**Type:** Event

Media player event that signals the end of an ad break.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_break_end_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_break_end_event_1 ad_break_end_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_break_end_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad complete

Track an ad complete event to signal that the ad creative was played to the end, at normal speed.

### `ad_complete_event`

**Type:** Event

Media player event that signals the ad creative was played to the end at normal speed.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_complete_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_complete_event_1 ad_complete_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_complete_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad pause

Track an ad pause event when a user clicks the pause control to stop the ad creative.

### `ad_pause_event`

**Type:** Event

Media player event fired when the user clicked the pause control and stopped the ad creative.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_pause_event/jsonschema/1-0-0`

**Example:**

```json
{
  "percentProgress": 50
}
```

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_pause_event_1 ad_pause_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_pause_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad quartile

The trackers have dedicated functions to track each ad quartile event: `trackMediaAdFirstQuartile`, `trackMediaAdMidpoint`, `trackMediaAdThirdQuartile`, or equivalent API. These events set the `percentProgress` property automatically to 25%, 50%, and 75% respectively.

### `ad_quartile_event`

**Type:** Event

Media player event fired when a quartile of ad is reached after continuous ad playback at normal speed.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_quartile_event/jsonschema/1-0-0`

**Example:**

```json
{
  "percentProgress": 50
}
```

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Required._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_quartile_event_1 ad_quartile_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_quartile_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad resume

Track an ad resume event when a user resumes playing the ad creative after it had been stopped or paused.

### `ad_resume_event`

**Type:** Event

Media player event fired when the user resumed playing the ad creative after it had been stopped or paused.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_resume_event/jsonschema/1-0-0`

**Example:**

```json
{
  "percentProgress": 50
}
```

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_resume_event_1 ad_resume_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_resume_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad skip

Track an ad skip event when the user activates a skip control to skip the ad creative.

Tracking this event will increase the counter of `adsSkipped` in the media session entity.

### `ad_skip_event`

**Type:** Event

Media player event fired when the user activated a skip control to skip the ad creative.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_skip_event/jsonschema/1-0-0`

**Example:**

```json
{
  "percentProgress": 50
}
```

**Properties:**

| Property                    | Description                                       |
| --------------------------- | ------------------------------------------------- |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the ad |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_skip_event_1 ad_skip_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_skip_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

### Ad start

Track an ad start event at the start of an ad.

Tracking this event will increase the counter of `ads` in the media session entity.

### `ad_start_event`

**Type:** Event

Media player event that signals the start of an ad.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_start_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_ad_start_event_1 ad_start_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'ad_start_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow.media'
```

## Custom events

Track your own custom events within the media session, using `trackMediaSelfDescribingEvent` or equivalent API. The tracker will automatically attach the media entities.

## Automatically included entities

Every media event includes entities that describe the current state of the player and session. These entities are attached automatically by the tracker. You don't need to track them yourself.

| Entity         | Attached to      | Purpose                                                      |
| -------------- | ---------------- | ------------------------------------------------------------ |
| Media player   | All media events | Current playback state (position, duration, volume, quality) |
| Media session  | All media events | Aggregated session metrics (time played, content watched)    |
| Media ad       | Ad events only   | Information about the current ad                             |
| Media ad break | Ad events only   | Information about the ad break                               |

You can also add custom entities to all media events, by providing them in the configuration. These entities will be attached to all subsequent media events for that player instance.

As with all Snowplow events, you can also attach custom entities to individual media events when tracking them.

### Media player

The media player entity captures the current state of playback at the moment each event fires. It's required for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model.

### `media_player`

**Type:** Entity

Schema for a context entity for media events that describes the media player and playback state

**Schema:** `iglu:com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0`

**Example:**

```json
{
  "currentTime": 109,
  "duration": 400,
  "ended": false,
  "fullscreen": false,
  "livestream": false,
  "label": "Big Bucks Bunny",
  "loop": false,
  "mediaType": "video",
  "muted": false,
  "paused": true,
  "pictureInPicture": false,
  "playbackRate": 1.5,
  "playerType": "com.vimeo-vimeo",
  "quality": "1080p",
  "volume": 100
}
```

**Properties:**

| Property                     | Description                                                                                                   |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `currentTime` _number_       | _Required._ The current playback time position within the media in seconds.                                   |
| `duration` _number_          | _Optional._ A floating-point value indicating the duration of the media in seconds.                           |
| `ended` _boolean_            | _Required._ Whether playback of the media has ended.                                                          |
| `fullscreen` _boolean_       | _Optional._ Whether the video element is fullscreen.                                                          |
| `livestream` _boolean_       | _Optional._ Whether the media is a live stream.                                                               |
| `label` _string_             | _Optional._ Human readable name given to tracked media content.                                               |
| `loop` _boolean_             | _Optional._ Whether the video should restart after ending.                                                    |
| `mediaType` _string_         | _Optional._ Type of media content. Must be one of: `video`, `audio`,``                                        |
| `muted` _boolean_            | _Optional._ Whether the media element is muted.                                                               |
| `paused` _boolean_           | _Required._ Whether the media element is paused                                                               |
| `pictureInPicture` _boolean_ | _Optional._ Whether the video element is showing picture-in-picture.                                          |
| `playbackRate` _number_      | _Optional._ Playback rate (1 is normal).                                                                      |
| `playerType` _string_        | _Optional._ Type of the media player (e.g., com.youtube-youtube, com.vimeo-vimeo, org.whatwg-media\_element). |
| `quality` _string_           | _Optional._ Quality level of the playback (e.g., 1080p).                                                      |
| `volume` _integer_           | _Optional._ Volume percentage at which the media will be played.                                              |

### Media session

The media session entity contains metrics calculated based on the tracked media events and the media update calls. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model.

The table below shows which media player properties and media events are used to calculate the metrics within the media session entity.

| Media player entity property | Media events                                               | Affected calculation of metric                                  |
| ---------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------- |
| `paused`                     | `play_event`, `pause_event`                                | `timePlayed`, `timePaused`, `timePlayedMuted`, `contentWatched` |
| `currentTime`                |                                                            | `timePlayed`, `timePaused`, `timePlayedMuted`, `contentWatched` |
| `muted`                      |                                                            | `timePlayedMuted`                                               |
| `playbackRate`               | `playback_rate_change_event`                               | `avgPlaybackRate`                                               |
|                              | `buffer_start_event`, `buffer_end_event`, `play_event`     | `timeBuffering`                                                 |
|                              | `ad_start_event`                                           | `ads`                                                           |
|                              | `ad_skip_event`                                            | `adsSkipped`                                                    |
|                              | `ad_click_event`                                           | `adsClicked`                                                    |
|                              | `ad_break_start_event`                                     | `adBreaks`                                                      |
|                              | `ad_start_event`, `ad_quartile_event`, `ad_complete_event` | `timeSpentAds`                                                  |
|                              | `ad_start_event`, `ad_complete_event`, `ad_skip_event`     | `timePlayed`, `timePlayedMuted`\*                               |

\* Play time stats aren't incremented while ads with type linear (default) are being played. Linear ads take over the video playback. For non-linear and companion ads, play time stats are still incremented while the ad is playing.

### `session`

**Type:** Entity

Schema for a context entity for media player events that tracks a session of a single media player usage

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0`

**Example:**

```json
{
  "mediaSessionId": "2d9bd9ac-abbd-419a-b934-9a2965cba339",
  "startedAt": "2023-11-03T09:55:29.920Z",
  "pingInterval": 15,
  "timePlayed": 143.12,
  "timePlayedMuted": 0,
  "timePaused": 8.12,
  "contentWatched": 120,
  "timeBuffering": 0.988,
  "timeSpentAds": 14.2,
  "ads": 4,
  "adsClicked": 0,
  "adsSkipped": 1,
  "adBreaks": 2,
  "avgPlaybackRate": 1.21
}
```

**Properties:**

| Property                   | Description                                                                                                                                                                                                   |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mediaSessionId` _string_  | _Required._ An identifier for the media session that is kept while the media content is played in the media player.                                                                                           |
| `startedAt` _null_         | _Optional._ Local date-time timestamp of when the session started.                                                                                                                                            |
| `pingInterval` _integer_   | _Optional._ Interval (seconds) in which the ping events will be sent. Default (specified in the tracker media docs) is assumed if not specified.                                                              |
| `timePlayed` _number_      | _Optional._ Total seconds user spent playing content (excluding linear ads).                                                                                                                                  |
| `timePlayedMuted` _number_ | _Optional._ Total seconds user spent playing content on mute (excluding linear ads).                                                                                                                          |
| `timePaused` _number_      | _Optional._ Total seconds user spent with paused content (excluding linear ads).                                                                                                                              |
| `contentWatched` _number_  | _Optional._ Total seconds of the content played. Each part of the content played is counted once (i.e., counts rewinding or rewatching the same content only once). Playback rate does not affect this value. |
| `timeBuffering` _number_   | _Optional._ Total seconds that playback was buffering during the session.                                                                                                                                     |
| `timeSpentAds` _number_    | _Optional._ Total seconds that ads played during the session.                                                                                                                                                 |
| `ads` _integer_            | _Optional._ Number of ads played.                                                                                                                                                                             |
| `adsClicked` _integer_     | _Optional._ Number of ads that the user clicked on.                                                                                                                                                           |
| `adsSkipped` _integer_     | _Optional._ Number of ads that the user skipped.                                                                                                                                                              |
| `adBreaks` _integer_       | _Optional._ Number of ad breaks played.                                                                                                                                                                       |
| `avgPlaybackRate` _number_ | _Optional._ Average playback rate (1 is normal speed).                                                                                                                                                        |

### Media ad

The media ad entity describes the currently playing ad. It's attached to ad events.

### `ad`

**Type:** Entity

Schema for a context entity with information about the currently played ad

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0`

**Example:**

```json
{
  "name": "Snowplow 5s ad",
  "adId": "2d9bd9ac-abbd-419a-b934-9a2965cba339",
  "creativeId": "fb819c48-5760-4d94-9c5b-4fa52f61a998",
  "podPosition": 2,
  "duration": 5,
  "skippable": true
}
```

**Properties:**

| Property                | Description                                                                      |
| ----------------------- | -------------------------------------------------------------------------------- |
| `name` _null_           | _Optional._ Friendly name of the ad.                                             |
| `adId` _string_         | _Required._ Unique identifier for the ad.                                        |
| `creativeId` _string_   | _Optional._ The ID of the ad creative.                                           |
| `podPosition` _integer_ | _Optional._ The position of the ad within the ad break, starting with 1.         |
| `duration` _number_     | _Optional._ Length of the video ad in seconds.                                   |
| `skippable` _boolean_   | _Optional._ Indicating whether skip controls are made available to the end user. |

### Media ad break

The media ad break entity describes a group of ads played together, whether pre-roll, mid-roll, or post-roll. It is attached to ad break events and all ad events within the break.

### `ad_break`

**Type:** Entity

Schema for a context entity, shared with all ad events belonging to the ad break.

**Schema:** `iglu:com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0`

**Example:**

```json
{
  "name": "pre-roll",
  "breakId": "fb819c48-5760-4d94-9c5b-4fa52f61a998",
  "startTime": 0,
  "breakType": "linear",
  "podSize": 2
}
```

**Properties:**

| Property             | Description                                                                                                                                                                                                                                                             |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` _null_        | _Optional._ Ad break name (e.g., pre-roll, mid-roll, and post-roll).                                                                                                                                                                                                    |
| `breakId` _string_   | _Required._ An identifier for the ad break.                                                                                                                                                                                                                             |
| `startTime` _number_ | _Required._ Playback time in seconds at the start of the ad break.                                                                                                                                                                                                      |
| `breakType` _string_ | _Optional._ Type of ads within the break: linear (take full control of the video for a period of time), nonlinear (run concurrently to the video), companion (accompany the video but placed outside the player). Must be one of: `linear`, `nonlinear`, `companion`,`` |
| `podSize` _integer_  | _Optional._ The number of ads to be played within the ad break.                                                                                                                                                                                                         |

## Player-specific data

The [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/), [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/), and [Vimeo](/docs/sources/web-trackers/tracking-events/media/vimeo/) media plugins automatically attach additional entities specific to those players. The Vimeo plugin also tracks additional events. See the documentation for each plugin for details.

The HTML5 and YouTube entities are used as part of the v1 media schemas for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package. If you're using the v2 schemas (default from Media Player 0.6+), the Media Player model doesn't use any fields from these player-specific entities. You can still use them for your own analysis.

The Vimeo events and entities aren't used by any version of the Media Player model.

## Version 1 schemas

The v1 media events use the `media_player_event` schema.

### `media_player_event`

**Type:** Event

Schema for a media event

**Schema:** `iglu:com.snowplowanalytics.snowplow/media_player_event/jsonschema/1-0-0`

**Example:**

```json
{
  "type": "play",
  "label": "Big Bucks Bunny"
}
```

**Properties:**

| Property         | Description                                     |
| ---------------- | ----------------------------------------------- |
| `type` _string_  | _Required._ The event fired by the media player |
| `label` _string_ | _Optional._ A custom identifier                 |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_media_player_event_1 media_player_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'media_player_event'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

The v1 media player entity uses the `1-0-0` version of the `media_player` schema, while v2 uses `2-0-0`.

### `media_player`

**Type:** Entity

Common Context Schema for a media player event

**Schema:** `iglu:com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0`

**Example:**

```json
{
  "currentTime": 52.3,
  "duration": 300,
  "ended": false,
  "isLive": false,
  "loop": false,
  "muted": false,
  "paused": false,
  "percentProgress": 17,
  "playbackRate": 1,
  "volume": 80
}
```

**Properties:**

| Property                    | Description                                                                                         |
| --------------------------- | --------------------------------------------------------------------------------------------------- |
| `currentTime` _number_      | _Required._ The current playback time                                                               |
| `duration` _number_         | _Required._ A double-precision floating-point value indicating the duration of the media in seconds |
| `ended` _boolean_           | _Required._ If playback of the media has ended                                                      |
| `isLive` _boolean_          | _Optional._ If the media is live                                                                    |
| `loop` _boolean_            | _Required._ If the video should restart after ending                                                |
| `muted` _boolean_           | _Required._ If the media element is muted                                                           |
| `paused` _boolean_          | _Required._ If the media element is paused                                                          |
| `percentProgress` _integer_ | _Optional._ The percent of the way through the media                                                |
| `playbackRate` _number_     | _Required._ Playback rate (1 is normal)                                                             |
| `volume` _integer_          | _Required._ Volume percent                                                                          |

---

# Mobile app install and lifecycle events
> Automatically track mobile app install, foreground, and background events with lifecycle entities for app usage analysis.
> Source: https://docs.snowplow.io/docs/events/ootb-data/mobile-lifecycle-events/

Mobile lifecycle events track key moments in your app's usage: when a user installs the app, opens it, or switches away. These events help you understand user engagement patterns, measure retention, and analyze session behavior.

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes install and lifecycle events.

## Install events

The mobile trackers can automatically track an install event when the app is first opened after installation.

This table shows the support for mobile application install tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Depending on your tracker and version, install autotracking may be enabled by default.

| Tracker                                                                                   | Supported | Since version | Auto-tracking | Notes                           |
| ----------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| Web                                                                                       | ❌         |               |               |                                 |
| [iOS](/docs/sources/mobile-trackers/tracking-events/installation-tracking/)               | ✅         | 1.1.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/installation-tracking/)           | ✅         | 1.1.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/installation-tracking/) | ✅         | 0.1.0         | ✅             | Only relevant for mobile events |
| Flutter                                                                                   | ❌         |               |               |                                 |
| Roku                                                                                      | ❌         |               |               |                                 |
| Google Tag Manager                                                                        | ❌         |               |               |                                 |

> **Note:** It's not possible to track when an app is uninstalled, since the mobile platforms don't provide a callback where such an event could be tracked.

This event has no properties since the relevant data is the timestamp, which is captured in the standard event fields.

### `application_install`

**Type:** Event

Schema for an event where a mobile application is installed.

**Schema:** `iglu:com.snowplowanalytics.mobile/application_install/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_application_install_1 application_install_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'application_install'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Android referrer details entity

On Android only, the Android tracker can attach an extra [entity with install referrer information](/docs/sources/mobile-trackers/tracking-events/installation-tracking/#android-only-tracking-referrer-information-from-the-google-play-referrer-library) to the install event. It makes use of the [Google Play Install Referrer library](https://developer.android.com/google/play/installreferrer) to retrieve the referrer information.

This entity is available from Android tracker version 5.2.0 onwards.

### `referrer_details`

**Type:** Event

Represents an install referrer details for Android apps installed from the Play Store (see https\://developer.android.com/reference/com/android/installreferrer/api/ReferrerDetails)

**Schema:** `iglu:com.android.installreferrer.api/referrer_details/jsonschema/1-0-0`

**Example:**

```json
{
  "installReferrer": "https://play.google.com/store/apps/details?id=com.example.myapp&referrer=someid%3Dsomedata",
  "referrerClickTimestamp": "2023-11-03T09:55:29.920Z",
  "installBeginTimestamp": "2023-11-03T10:55:29.920Z",
  "googlePlayInstantParam": true
}
```

**Properties:**

| Property                           | Description                                                                                                    |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `installReferrer` _string_         | _Optional._ The referrer URL of the installed package                                                          |
| `referrerClickTimestamp` _string_  | _Optional._ The timestamp when referrer click happens                                                          |
| `installBeginTimestamp` _string_   | _Optional._ The timestamp when installation begins                                                             |
| `googlePlayInstantParam` _boolean_ | _Required._ Boolean indicating if the user has interacted with the app's instant experience in the past 7 days |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_android_installreferrer_api_referrer_details_1 referrer_details_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'referrer_details'
and events.event_vendor = 'com.android.installreferrer.api'
```

## Foreground and background

Foreground and background events capture when the user switches to or away from your app. A foreground event fires when the app becomes visible on screen, while a background event fires when the user switches to another app or returns to the home screen.

The mobile trackers can track these events automatically. They also attach a lifecycle entity to all events, indicating whether the app was visible when each event occurred.

This table shows the support for mobile lifecycle event tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include lifecycle event tracking. Depending on your tracker and version, lifecycle autotracking may be enabled by default.

| Tracker                                                                                                | Supported | Since version | Auto-tracking | Notes                           |
| ------------------------------------------------------------------------------------------------------ | --------- | ------------- | ------------- | ------------------------------- |
| Web                                                                                                    | ❌         |               |               |                                 |
| [iOS](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                              | ✅         | 3.0.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/#tracking-deep-links)                          | ✅         | 3.0.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-deep-link-received-events) | ✅         | 0.1.0         | ✅             | Only relevant for mobile events |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                             | ✅         | 0.5.0         | ✅             | Only relevant for mobile events |
| Roku                                                                                                   | ❌         |               |               |                                 |
| Google Tag Manager                                                                                     | ❌         |               |               |                                 |

### Foreground event

The foreground event is tracked each time the app becomes visible on the screen. The `foregroundIndex` property counts how many times the app has been foregrounded since installation.

### `application_foreground`

**Type:** Event

Schema for an application foreground event

**Schema:** `iglu:com.snowplowanalytics.snowplow/application_foreground/jsonschema/1-0-0`

**Example:**

```json
{
  "foregroundIndex": 1
}
```

**Properties:**

| Property                    | Description |
| --------------------------- | ----------- |
| `foregroundIndex` _integer_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_application_foreground_1 application_foreground_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'application_foreground'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Background event

The background event is tracked when the app is no longer visible, such as when the user switches to another app or returns to the home screen. The `backgroundIndex` property counts how many times the app has been backgrounded since installation.

### `application_background`

**Type:** Event

Schema for an application background event

**Schema:** `iglu:com.snowplowanalytics.snowplow/application_background/jsonschema/1-0-0`

**Example:**

```json
{
  "backgroundIndex": 1
}
```

**Properties:**

| Property                    | Description |
| --------------------------- | ----------- |
| `backgroundIndex` _integer_ | _Optional._ |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_application_background_1 application_background_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'application_background'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Lifecycle entity

When lifecycle autotracking is enabled, this entity is automatically attached to all events. It indicates whether the app was in the foreground or background when the event occurred.

### `application_lifecycle`

**Type:** Entity

Entity that indicates the visibility state of the app (foreground, background)

**Schema:** `iglu:com.snowplowanalytics.mobile/application_lifecycle/jsonschema/1-0-0`

**Example:**

```json
{
  "isVisible": true,
  "index": 2
}
```

**Properties:**

| Property              | Description                                                                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `isVisible` _boolean_ | _Required._ Indicates if the app is in foreground state (true) or background state (false)                                                                               |
| `index` _integer_     | _Optional._ Represents the foreground index or background index (tracked with com.snowplowanalytics.snowplow application\_foreground and application\_background events. |

---

# Page and screen engagement
> Measure user engagement time and scrolling depth using page ping events on web and screen summary entities on mobile.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-activity-tracking/

Activity or engagement tracking enables you to measure the time a user spent engaged on a page or screen, and the extent of the page or screen they viewed.

Snowplow provides implementations for web and mobile platforms:

- Page ping events on web
- Screen summary entity on mobile

Both page and screen activity tracking are included in the [Snowplow Unified Digital package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) to calculate engagement metrics. The legacy [Web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) handles page pings as well.

## Page engagement

Page ping events record users engaging with content on a web page after it has initially loaded. This includes scroll position.

Where activity autotracking is available, you can configure your tracker to send page ping events at regular intervals. They're considered a type of "heartbeat" event that indicates the user is still present and engaging with the page.

Page ping events are [baked-in events](/docs/fundamentals/events/) that have no schema. They populate the [page offset atomic event parameters](/docs/fundamentals/canonical-event/#baked-in-event-fields).

This table shows the support for page ping tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Python is the only server-side tracker that provides page ping events.

| Tracker                                                                    | Supported | Since version | Auto-tracking | Notes                              |
| -------------------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/activity-page-pings/)     | ✅         | 0.10.0        | ✅             |                                    |
| iOS                                                                        | ❌         |               |               |                                    |
| Android                                                                    | ❌         |               |               |                                    |
| React Native                                                               | ❌         |               |               |                                    |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/) | ✅         | 0.1.0         | ✅             | Only on web                        |
| Roku                                                                       | ❌         |               |               |                                    |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)           | ✅         | 0.8.0         | ❌             | Detect activity and track manually |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)  | ✅         | v3            | ✅             |                                    |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes page pings.

## Screen engagement

Use screen engagement tracking to track a `screen_end` event when a user navigates away from a screen.

The trackers can also track a `screen_summary` entity that contains screen engagement data. This entity is sent along with [lifecycle](/docs/events/ootb-data/mobile-lifecycle-events/) and `screen_end` events.

This table shows the support for screen engagement tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). The server-side trackers don't include screen engagement tracking.

| Tracker                                                                                                        | Supported | Since version | Auto-tracking | Notes                           |
| -------------------------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/screen-views/#screen-engagement-tracking)                     | ✅         | 4.2.0         | ✅/❌           | Requires screen tracking plugin |
| [iOS](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking)               | ✅         | 6.0.0         | ✅             |                                 |
| [Android](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking)           | ✅         | 6.0.0         | ✅             |                                 |
| [React Native](/docs/sources/react-native-tracker/tracking-events/screen-tracking/#screen-engagement-tracking) | ✅         | 2.1.0         | ✅             | Only on mobile                  |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/#screen-engagement-tracking)                           | ✅         | 0.1.0         | ✅             | Only on mobile                  |
| Roku                                                                                                           | ❌         |               |               |                                 |
| Google Tag Manager                                                                                             | ❌         |               |               |                                 |

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes screen end events.

### Screen end event

This event can be tracked automatically just before the transition to the next screen, based on the firing of the next screen view event. It has no properties.

### `screen_end`

**Type:** Event

Schema for an event tracked before transitioning to a new screen

**Schema:** `iglu:com.snowplowanalytics.mobile/screen_end/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_screen_end_1 screen_end_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'screen_end'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### List item view event

Track this event to mark which items are visible in a list. When screen engagement tracking is enabled, this event won't be sent to the Collector but used to update the tracker state for the screen summary entity.

If you're using the iOS tracker with SwiftUI, you can configure it to [automatically track this event](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#list-item-view-tracking).

### `list_item_view`

**Type:** Event

Schema for an event tracked when an item is displayed in a list

**Schema:** `iglu:com.snowplowanalytics.mobile/list_item_view/jsonschema/1-0-0`

**Example:**

```json
{
  "index": 3,
  "items_count": 10
}
```

**Properties:**

| Property                | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `index` _integer_       | _Required._ Index of the item in a list on the screen     |
| `items_count` _integer_ | _Optional._ Total number of items in a list on the screen |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_list_item_view_1 list_item_view_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'list_item_view'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Scroll changed event

Track this event to mark changes in scroll position on a screen. When screen engagement tracking is enabled, this event won't be sent to the Collector but used to update the tracker state for the screen summary entity.

### `scroll_changed`

**Type:** Event

Schema for an event tracked when a scroll view's scroll position changes

**Schema:** `iglu:com.snowplowanalytics.mobile/scroll_changed/jsonschema/1-0-0`

**Example:**

```json
{
  "x_offset": 0,
  "y_offset": 250,
  "view_width": 375,
  "view_height": 667,
  "content_width": 375,
  "content_height": 1500
}
```

**Properties:**

| Property                   | Description                                                        |
| -------------------------- | ------------------------------------------------------------------ |
| `x_offset` _integer_       | _Optional._ Horizontal scroll offset in pixels                     |
| `y_offset` _integer_       | _Optional._ Vertical scroll offset in pixels                       |
| `view_width` _integer_     | _Optional._ The width of the scroll view in pixels                 |
| `view_height` _integer_    | _Optional._ The height of the scroll view in pixels                |
| `content_width` _integer_  | _Optional._ The width of the content in the scroll view in pixels  |
| `content_height` _integer_ | _Optional._ The height of the content in the scroll view in pixels |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_scroll_changed_1 scroll_changed_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'scroll_changed'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Screen summary entity

Entity that contains screen engagement information, including how long a user spends on a screen in the foreground and background, as well as information about scroll depth and list items.

The entity data is based on tracker state captured from lifecycle, list item view, and scroll changed events.

### `screen_summary`

**Type:** Entity

Schema for an entity tracked with foreground/background/screen\_end events with summary statistics about the screen view

**Schema:** `iglu:com.snowplowanalytics.mobile/screen_summary/jsonschema/1-0-0`

**Example:**

```json
{
  "foreground_sec": 10.2,
  "background_sec": 3.1,
  "last_item_index": 11,
  "items_count": 50,
  "min_x_offset": 0,
  "max_x_offset": 400,
  "min_y_offset": 0,
  "max_y_offset": 1000,
  "content_width": 400,
  "content_height": 5000
}
```

**Properties:**

| Property                    | Description                                                                             |
| --------------------------- | --------------------------------------------------------------------------------------- |
| `foreground_sec` _number_   | _Required._ Time in seconds spent on the current screen while the app was in foreground |
| `background_sec` _number_   | _Optional._ Time in seconds spent on the current screen while the app was in background |
| `last_item_index` _integer_ | _Optional._ Index of the last viewed item in the list on the screen                     |
| `items_count` _integer_     | _Optional._ Total number of items in the list on the screen                             |
| `min_x_offset` _integer_    | _Optional._ Minimum horizontal scroll offset on the scroll view in pixels               |
| `max_x_offset` _integer_    | _Optional._ Maximum horizontal scroll offset on the scroll view in pixels               |
| `min_y_offset` _integer_    | _Optional._ Minimum vertical scroll offset on the scroll view in pixels                 |
| `max_y_offset` _integer_    | _Optional._ Maximum vertical scroll offset on the scroll view in pixels                 |
| `content_width` _integer_   | _Optional._ Width of the scroll view in pixels                                          |
| `content_height` _integer_  | _Optional._ Height of the scroll view in pixels                                         |

---

# Page and screen views
> Track page views on web and screen views on mobile with page view IDs and screen context entities.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-and-screen-view-events/

Page and screen view tracking is key for understanding your users' navigation within your app.

Snowplow provides implementations for web and mobile platforms:

- Page view events on web
- Screen view events on mobile

Both page and screen views are included in the [Snowplow Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/). The legacy [Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) and [Mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/) packages handle page and screen views respectively.

## Page views

Track page views to record when a user views a web page.

Page view events are [baked-in events](/docs/fundamentals/events/) that have no schema. They populate the [page atomic event parameters](/docs/fundamentals/canonical-event/#page-fields).

This table shows the support for page view tracking across the main [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                                       | Supported | Since version | Auto-tracking | Notes                                                   |
| --------------------------------------------------------------------------------------------- | --------- | ------------- | ------------- | ------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/page-views/)                                 | ✅         | 0.1.0         | ✅/❌           | Web page entity can be automatically added              |
| iOS                                                                                           | ❌         |               |               |                                                         |
| Android                                                                                       | ❌         |               |               |                                                         |
| [React Native](/docs/sources/react-native-tracker/tracking-events/#tracking-page-view-events) | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)                    | ✅         | 0.1.0         | ✅/❌           | Only on web; web page entity can be automatically added |
| Roku                                                                                          | ❌         |               |               |                                                         |
| Node.js                                                                                       | ❌         |               |               |                                                         |
| Golang                                                                                        | ❌         |               |               |                                                         |
| .NET                                                                                          | ❌         |               |               |                                                         |
| [Java](/docs/sources/java-tracker/tracking-events/)                                           | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)                              | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| Scala                                                                                         | ❌         |               |               |                                                         |
| [Ruby](/docs/sources/ruby-tracker/tracking-events/)                                           | ✅         | 0.1.0         | ❌             | No web page entity                                      |
| Rust                                                                                          | ❌         |               |               |                                                         |
| PHP                                                                                           | ❌         |               |               |                                                         |
| C++                                                                                           | ❌         |               |               |                                                         |
| Unity                                                                                         | ❌         |               |               |                                                         |
| Lua                                                                                           | ❌         |               |               |                                                         |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/)                     | ✅         | v3            | ❌             |                                                         |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes page views.

### Web page entity

Some trackers assign a unique identifier to every page load: the page view ID. Where available, you can configure this UUID to be added to every event tracked on that page, by tracking the `web_page` entity. This identifier can be very useful for sessionization and user journey analysis.

### `web_page`

**Type:** Entity

Schema for a web page context

**Schema:** `iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "024629F0-6B9B-440C-82D4-DB3CF9D31533"
}
```

**Properties:**

| Property      | Description |
| ------------- | ----------- |
| `id` _string_ | _Required._ |

## Screen views

Screen view events are the mobile equivalent of page view events.

This table shows the support for screen view tracking across the main [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                                                                                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/screen-views/)           | ✅         | 4.2.0         | ✅/❌           | Requires screen tracking plugin; we recommend using page views instead                        |
| [iOS](/docs/sources/mobile-trackers/tracking-events/screen-tracking/)     | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; screen view autotracking depends on platform        |
| [Android](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; screen view autotracking depends on platform        |
| [React Native](/docs/sources/react-native-tracker/tracking-events/)       | ✅         | 0.1.0         | ✅/❌           | Screen entity can be automatically added; add navigation handler for screen view autotracking |
| [Flutter](/docs/sources/flutter-tracker/tracking-events/)                 | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Roku](/docs/sources/roku-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Node.js](/docs/sources/node-js-tracker/tracking-events/)                 | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Golang](/docs/sources/golang-tracker/)                                   | ✅         | 1.0.0         | ❌             | No screen entity                                                                              |
| [.NET](/docs/sources/net-tracker/)                                        | ✅         | 1.3.0         |               | No screen entity                                                                              |
| [Java](/docs/sources/java-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Python](/docs/sources/python-tracker/tracking-specific-events/)          | ✅         | 0.2.0         | ❌             | No screen entity                                                                              |
| Scala                                                                     | ❌         |               |               |                                                                                               |
| [Ruby](/docs/sources/ruby-tracker/tracking-events/)                       | ✅         | 0.2.0         | ❌             | No screen entity                                                                              |
| [Rust](/docs/sources/rust-tracker/tracking-events/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [PHP](/docs/sources/php-tracker/tracking-an-event/)                       | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [C++](/docs/sources/c-tracker/tracking-specific-events/)                  | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| [Unity](/docs/sources/unity-tracker/)                                     | ✅         | 0.8.0         | ❌             | No screen entity                                                                              |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                | ✅         | 0.1.0         | ❌             | No screen entity                                                                              |
| Google Tag Manager                                                        | ❌         |               |               |                                                                                               |

We recommend using the [Base mobile tracking plan template](/docs/event-studio/tracking-plans/templates/#base-mobile) for mobile tracking. It includes screen views.

Screen view events record the name and identifier of the current screen, as well as some optional properties such as the ID of the previous screen. Information about the previous screen can be useful for understanding user navigation within your application.

### `screen_view`

**Type:** Event

Schema for a screen view event

**Schema:** `iglu:com.snowplowanalytics.mobile/screen_view/jsonschema/1-0-0`

**Example:**

```json
{
  "name": "Screen 2",
  "type": "feed",
  "id": "024629F0-6B9B-440C-82D4-DB3CF9D31533",
  "previousName": "Screen 1",
  "previousId": "86ffd23c-b844-4ed4-aae9-7796b8b6f1e4",
  "previousType": "feed",
  "transitionType": "push"
}
```

**Properties:**

| Property                  | Description                                                             |
| ------------------------- | ----------------------------------------------------------------------- |
| `name` _string_           | _Required._ The name of the screen viewed.                              |
| `type` _string_           | _Optional._ The type of screen that was viewed e.g feed / carousel.     |
| `id` _string_             | _Required._ An ID from the associated screenview event.                 |
| `previousName` _string_   | _Optional._ The name of the previous screen.                            |
| `previousId` _string_     | _Optional._ A screenview ID of the previous screenview.                 |
| `previousType` _string_   | _Optional._ The screen type of the previous screenview.                 |
| `transitionType` _string_ | _Optional._ The type of transition that led to the screen being viewed. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_mobile_screen_view_1 screen_view_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'screen_view'
and events.event_vendor = 'com.snowplowanalytics.mobile'
```

### Screen entity

Some trackers can automatically attach a `screen` entity to all events after a screen view is tracked. This entity contains information about the current, or last seen, screen.

### `screen`

**Type:** Entity

Schema for a context that represents information pertaining to the current screen being viewed when an event occurs.

**Schema:** `iglu:com.snowplowanalytics.mobile/screen/jsonschema/1-0-0`

**Example:**

```json
{
  "name": "Screen 2",
  "type": "feed",
  "id": "024629F0-6B9B-440C-82D4-DB3CF9D31533"
}
```

**Properties:**

| Property                     | Description                                                            |
| ---------------------------- | ---------------------------------------------------------------------- |
| `name` _string_              | _Required._ The name of the screen viewed.                             |
| `type` _string_              | _Optional._ The type of screen that was viewed e.g feed / carousel.    |
| `id` _string_                | _Required._ An ID from the associated screenview event.                |
| `viewController` _string_    | _Optional._ iOS specific: class name of the view controller.           |
| `topViewController` _string_ | _Optional._ iOS specific: class name of the top level view controller. |
| `activity` _string_          | _Optional._ Android specific: name of activity.                        |
| `fragment` _string_          | _Optional._ Android specific: name of fragment.                        |

---

# Page element visibility and interaction tracking
> Track page element visibility, button clicks, and form interactions to understand how users engage with specific components on your web pages.
> Source: https://docs.snowplow.io/docs/events/ootb-data/page-elements/

Page element tracking captures user interactions with specific elements on your web pages, including buttons, forms, and any other visible components.

Use page element tracking to:

- Measure button click engagement across your site
- Track form interactions including focus, changes, and submissions
- Understand which page sections users actually see
- Build funnel analysis based on element visibility and interactions
- Measure content consumption and scroll depth

## Button clicks

Button click tracking automatically captures clicks on `<button>` and `<input type="button">` elements.

This table shows the support for button click tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                               |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/button-click/)           | ✅         | 3.18.0        | ✅             | Requires button click plugin        |
| iOS                                                                       | ❌         |               |               |                                     |
| Android                                                                   | ❌         |               |               |                                     |
| React Native                                                              | ❌         |               |               |                                     |
| Flutter                                                                   | ❌         |               |               |                                     |
| Roku                                                                      | ❌         |               |               |                                     |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v4            | ✅             | Integrates with button click plugin |

We recommend using the [Base web tracking plan template](/docs/event-studio/tracking-plans/templates/#base-web) for web tracking. It includes button clicks.

### `button_click`

**Type:** Event

Schema for a button click event

**Schema:** `iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0`

**Example:**

```json
{
  "label": "Submit",
  "id": "submit-btn",
  "classes": [
    "btn",
    "btn-primary"
  ],
  "name": "submit"
}
```

**Properties:**

| Property          | Description                                                     |
| ----------------- | --------------------------------------------------------------- |
| `label` _string_  | _Required._ The text on the button, or a user-provided override |
| `id` _string_     | _Optional._ The ID of the button                                |
| `classes` _array_ | _Optional._ The classes of the button                           |
| `name` _string_   | _Optional._ The name attribute of the button                    |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_button_click_1 button_click_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'button_click'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Form interactions

Form tracking captures three types of events: when a user focuses on a form field, when a field value changes, and when a form is submitted.

This table shows the support for form tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version                       | Auto-tracking | Notes                                                                                             |
| ------------------------------------------------------------------------- | --------- | ----------------------------------- | ------------- | ------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/form-tracking/)          | ✅         | 2.1.0 (directly), 3.0.0 (as plugin) | ✅             | Requires form tracking plugin                                                                     |
| iOS                                                                       | ❌         |                                     |               |                                                                                                   |
| Android                                                                   | ❌         |                                     |               |                                                                                                   |
| React Native                                                              | ❌         |                                     |               |                                                                                                   |
| Flutter                                                                   | ❌         |                                     |               |                                                                                                   |
| Roku                                                                      | ❌         |                                     |               |                                                                                                   |
| Python                                                                    | ❌         |                                     |               | The Python tracker has deprecated form tracking APIs; use custom events with form schemas instead |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3                                  | ✅             | Integrates with form tracking plugin                                                              |

### Focus form event

Triggered when a user focuses on a form field.

### `focus_form`

**Type:** Event

Schema for a form field focus event

**Schema:** `iglu:com.snowplowanalytics.snowplow/focus_form/jsonschema/1-0-0`

**Example:**

```json
{
  "formId": "contact-form",
  "elementId": "email",
  "nodeName": "INPUT",
  "elementType": "email",
  "elementClasses": [
    "form-control"
  ],
  "value": ""
}
```

**Properties:**

| Property                 | Description                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------- |
| `formId` _string_        | _Required._ The ID of the form                                                         |
| `elementId` _string_     | _Optional._ The ID of the element                                                      |
| `nodeName` _string_      | _Required._ The node name of the element Must be one of: `INPUT`, `TEXTAREA`, `SELECT` |
| `elementType` _string_   | _Optional._ The type of the element                                                    |
| `elementClasses` _array_ | _Required._ The classes of the element                                                 |
| `value` _string_         | _Optional._ The value of the element                                                   |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_focus_form_1 focus_form_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'focus_form'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Change form event

Triggered when a user changes the value of a form field.

### `change_form`

**Type:** Event

Schema for a form field change event

**Schema:** `iglu:com.snowplowanalytics.snowplow/change_form/jsonschema/1-0-0`

**Example:**

```json
{
  "formId": "contact-form",
  "elementId": "email",
  "nodeName": "INPUT",
  "elementType": "email",
  "elementClasses": [
    "form-control"
  ],
  "value": "user@example.com"
}
```

**Properties:**

| Property                 | Description                                                                            |
| ------------------------ | -------------------------------------------------------------------------------------- |
| `formId` _string_        | _Required._ The ID of the form                                                         |
| `elementId` _string_     | _Optional._ The ID of the element                                                      |
| `nodeName` _string_      | _Required._ The node name of the element Must be one of: `INPUT`, `TEXTAREA`, `SELECT` |
| `elementType` _string_   | _Optional._ The type of the element                                                    |
| `elementClasses` _array_ | _Required._ The classes of the element                                                 |
| `value` _string_         | _Required._ The value of the element                                                   |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_change_form_1 change_form_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'change_form'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Submit form event

Triggered when a user submits a form. Contains data about all form fields.

### `submit_form`

**Type:** Event

Schema for a form submission event

**Schema:** `iglu:com.snowplowanalytics.snowplow/submit_form/jsonschema/1-0-0`

**Example:**

```json
{
  "formId": "contact-form",
  "formClasses": [
    "contact-form",
    "validated"
  ],
  "elements": [
    {
      "name": "email",
      "value": "user@example.com",
      "nodeName": "INPUT",
      "type": "email"
    },
    {
      "name": "message",
      "value": "Hello",
      "nodeName": "TEXTAREA",
      "type": null
    }
  ]
}
```

**Properties:**

| Property              | Description                                    |
| --------------------- | ---------------------------------------------- |
| `formId` _string_     | _Required._ The ID of the form                 |
| `formClasses` _array_ | _Required._ The classes of the form            |
| `elements` _array_    | _Required._ The form elements and their values |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_submit_form_1 submit_form_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'submit_form'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

## Page element visibility and lifecycle

Element visibility tracking enables declarative tracking of page elements as they are created, become visible, leave view, or are removed from the page. It's only available for web.

This table shows the support for page element visibility tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                             | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------- | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/element-tracking/) | ✅         | 4.6.0         | ✅             |
| iOS                                                                 | ❌         |               |               |
| Android                                                             | ❌         |               |               |
| React Native                                                        | ❌         |               |               |
| Flutter                                                             | ❌         |               |               |
| Roku                                                                | ❌         |               |               |
| Google Tag Manager                                                  | ❌         |               |               |

### Events

The element tracking plugin generates four event types:

- `create_element`: when a matching element is added to the page
- `expose_element`: when a matching element scrolls into view
- `obscure_element`: when a matching element scrolls out of view
- `destroy_element`: when a matching element is removed from the page

All of these events have a single property, `element_name`. The plugin adds an `element` entity with information about the tracked element.

You can also configure the tracker to attach `element_statistics`, `element_content`, or `component_parents` entities.

#### Create element event

Triggered when an element matching a named configuration is added to the page or detected on page load.

### `create_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as existing in or being added to a document.

**Schema:** `iglu:com.snowplowanalytics.snowplow/create_element/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "newsletter signup"
}
```

**Properties:**

| Property                | Description                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was created. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_create_element_1 create_element_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'create_element'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Expose element event

Triggered when a tracked element scrolls into the viewport and becomes visible to the user.

### `expose_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as intersecting with the viewport, becoming visible to the user.

**Schema:** `iglu:com.snowplowanalytics.snowplow/expose_element/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "newsletter signup"
}
```

**Properties:**

| Property                | Description                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was exposed. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_expose_element_1 expose_element_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'expose_element'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Obscure element event

Triggered when a tracked element scrolls out of the viewport or becomes hidden from the user.

### `obscure_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as becoming hidden from a user or moving out of the viewport.

**Schema:** `iglu:com.snowplowanalytics.snowplow/obscure_element/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "newsletter signup"
}
```

**Properties:**

| Property                | Description                                                                                                                                   |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was obscured. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_obscure_element_1 obscure_element_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'obscure_element'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

#### Destroy element event

Triggered when a tracked element is removed from the page.

### `destroy_element`

**Type:** Event

Event that fires when an element matching a named configuration is detected as being removed from a document.

**Schema:** `iglu:com.snowplowanalytics.snowplow/destroy_element/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "newsletter signup"
}
```

**Properties:**

| Property                | Description                                                                                                                                    |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_ | _Required._ The name of the element that was destroyed. Should match the element name field in entities that describe this particular element. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_destroy_element_1 destroy_element_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'destroy_element'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

### Element entity

The `element` entity is attached to all element tracking events and contains information about the tracked element's position, dimensions, and attributes.

### `element`

**Type:** Entity

Schema for element context

**Schema:** `iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "newsletter signup",
  "width": 560,
  "height": 137,
  "position_x": 320,
  "position_y": 1953.5,
  "doc_position_x": 320,
  "doc_position_y": 5392.5,
  "element_index": 2,
  "element_matches": 2,
  "originating_page_view": "02e30714-a84a-42f8-8b07-df106d669db0",
  "attributes": []
}
```

**Properties:**

| Property                         | Description                                                       |
| -------------------------------- | ----------------------------------------------------------------- |
| `element_name` _string_          | _Required._ The name of the element                               |
| `width` _number_                 | _Optional._ The width of the element                              |
| `height` _number_                | _Optional._ The height of the element                             |
| `position_x` _number_            | _Optional._ The x position of the element in the viewport         |
| `position_y` _number_            | _Optional._ The y position of the element in the viewport         |
| `doc_position_x` _number_        | _Optional._ The x position of the element in the document         |
| `doc_position_y` _number_        | _Optional._ The y position of the element in the document         |
| `element_index` _integer_        | _Optional._ The index of this element among all matching elements |
| `element_matches` _integer_      | _Optional._ The total number of elements matching the rule        |
| `originating_page_view` _string_ | _Optional._ The page view ID when the element was first observed  |
| `attributes` _array_             | _Optional._ Custom attributes extracted from the element          |

### Element statistics entity

The `element_statistics` entity contains information about element visibility and scroll depth.

### `element_statistics`

**Type:** Entity

Schema for element statistics

**Schema:** `iglu:com.snowplowanalytics.snowplow/element_statistics/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "article_content",
  "element_index": 1,
  "element_matches": 1,
  "current_state": "unknown",
  "min_size": "800x3928",
  "current_size": "800x3928",
  "max_size": "800x3928",
  "y_depth_ratio": 0.203,
  "max_y_depth_ratio": 0.493,
  "max_y_depth": "1937/3928",
  "element_age_ms": 298379,
  "times_in_view": 0,
  "total_time_visible_ms": 185000
}
```

**Properties:**

| Property                          | Description |
| --------------------------------- | ----------- |
| `element_name` _string_           | _Required._ |
| `element_index` _integer_         | _Optional._ |
| `element_matches` _integer_       | _Optional._ |
| `current_state` _string_          | _Optional._ |
| `min_size` _string_               | _Optional._ |
| `current_size` _string_           | _Optional._ |
| `max_size` _string_               | _Optional._ |
| `y_depth_ratio` _number_          | _Optional._ |
| `max_y_depth_ratio` _number_      | _Optional._ |
| `max_y_depth` _string_            | _Optional._ |
| `element_age_ms` _integer_        | _Optional._ |
| `times_in_view` _integer_         | _Optional._ |
| `total_time_visible_ms` _integer_ | _Optional._ |

### Element content entity

The `element_content` entity captures data extracted from a tracked element, including its position within a parent component hierarchy.

### `element_content`

**Type:** Entity

Entity describing the content of an element matching a named configuration.

**Schema:** `iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0`

**Example:**

```json
{
  "parent_name": "product grid",
  "parent_index": 1,
  "element_name": "product card",
  "element_index": 3,
  "attributes": [
    {
      "source": "dataset",
      "attribute": "product_id",
      "value": "SKU-12345"
    },
    {
      "source": "content",
      "attribute": "price",
      "value": "$49.99"
    }
  ]
}
```

**Properties:**

| Property                  | Description                                                                                                                                                                               |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `parent_name` _string_    | _Required._ The name of the configuration for the element/component that contains the element described by this entity.                                                                   |
| `parent_index` _integer_  | _Required._ The index of this element's parent element/component's within it's parent's other matches for it's element configuration.                                                     |
| `element_name` _string_   | _Required._ The name of the configuration for the element/component that contains this data. This will usually match the corresponding element\_name in an event payload or other entity. |
| `element_index` _integer_ | _Required._ The position of this element's within it's parent's other matches for the element's configuration.                                                                            |
| `attributes` _array_      | _Optional._ Results of configured contents descriptions found on this element.                                                                                                            |

### Component parents entity

The `component_parents` entity records the hierarchy of parent components that contain a tracked element.

### `component_parents`

**Type:** Entity

Entity describing the list of components that were found to contain the element with the named configuration.

**Schema:** `iglu:com.snowplowanalytics.snowplow/component_parents/jsonschema/1-0-0`

**Example:**

```json
{
  "element_name": "add to cart button",
  "component_list": [
    "product grid",
    "product card",
    "card actions"
  ]
}
```

**Properties:**

| Property                 | Description                                                                                                                                                                      |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `element_name` _string_  | _Optional._ Name of the element that this entity relates to, if any. If not provided, may apply to a subject of some other event, such as Link, Button, or Form Tracking events. |
| `component_list` _array_ | _Required._ List of component names that were detected as containing the element that is the subject of this event.                                                              |

---

# Site search tracking
> Track internal site search queries with search terms, filters, and result counts to analyze user search behavior and content discovery.
> Source: https://docs.snowplow.io/docs/events/ootb-data/site-search/

Site search tracking captures how users search within your website. This data helps you understand what content users are looking for, whether they find it, and how search influences their journey through your site.

Use site search tracking to:

- Identify popular search terms and content gaps
- Measure search success rates based on result counts
- Analyze how search filters affect user behavior
- Optimize your internal search functionality

This table shows the support for site search tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/site-search/)            | ✅         | 3.0.0         | ❌             | Requires site tracking plugin |
| iOS                                                                       | ❌         |               |               |                               |
| Android                                                                   | ❌         |               |               |                               |
| React Native                                                              | ❌         |               |               |                               |
| Flutter                                                                   | ❌         |               |               |                               |
| Roku                                                                      | ❌         |               |               |                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3            | ❌             |                               |

The site search event captures the search terms, any filters applied, and the number of results returned.

### `site_search`

**Type:** Event

Schema for a site search event

**Schema:** `iglu:com.snowplowanalytics.snowplow/site_search/jsonschema/1-0-0`

**Example:**

```json
{
  "terms": [
    "unified",
    "log"
  ],
  "filters": {
    "category": "books",
    "sub-category": "non-fiction"
  },
  "totalResults": 14,
  "pageResults": 8
}
```

**Properties:**

| Property                 | Description                                   |
| ------------------------ | --------------------------------------------- |
| `terms` _array_          | _Required._ The search terms                  |
| `filters` _object_       | _Optional._ The search filters                |
| `totalResults` _integer_ | _Optional._ The total number of results       |
| `pageResults` _integer_  | _Optional._ The number of results on the page |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_site_search_1 site_search_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'site_search'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Social media interaction tracking
> Track user interactions with social media widgets including likes, shares, and retweets across Facebook, Twitter, and other platforms.
> Source: https://docs.snowplow.io/docs/events/ootb-data/social-media/

Social media interaction tracking captures how users engage with social widgets embedded on your website, such as like buttons, share buttons, and tweet buttons.

Use social interaction tracking to:

- Measure how often content is shared on social platforms
- Identify which social networks drive the most engagement
- Understand which content resonates with users enough to share
- Track the performance of social sharing features

This table shows the support for social interaction tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                   | Supported | Since version | Auto-tracking | Notes                         |
| ------------------------------------------------------------------------- | --------- | ------------- | ------------- | ----------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/social-media/)           | ✅         | 3.0.0         | ❌             | Requires site tracking plugin |
| iOS                                                                       | ❌         |               |               |                               |
| Android                                                                   | ❌         |               |               |                               |
| React Native                                                              | ❌         |               |               |                               |
| Flutter                                                                   | ❌         |               |               |                               |
| Roku                                                                      | ❌         |               |               |                               |
| [Google Tag Manager](/docs/sources/google-tag-manager/snowplow-template/) | ✅         | v3            | ❌             |                               |

The social interaction event captures the social network, the action performed, and optionally the target content.

### `social_interaction`

**Type:** Event

Schema for a social interaction event

**Schema:** `iglu:com.snowplowanalytics.snowplow/social_interaction/jsonschema/1-0-0`

**Example:**

```json
{
  "action": "like",
  "network": "facebook",
  "target": "pbz00123"
}
```

**Properties:**

| Property           | Description                                                               |
| ------------------ | ------------------------------------------------------------------------- |
| `action` _string_  | _Required._ The social action performed, e.g. like, retweet, share        |
| `network` _string_ | _Required._ The social network, e.g. facebook, twitter                    |
| `target` _string_  | _Optional._ The object of the social action, e.g. a page ID or product ID |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_social_interaction_1 social_interaction_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'social_interaction'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# Third-party sources
> Capture data from third-party tools including Optimizely experiments, Google Analytics cookies, and Kantar Focal Meter for unified analytics.
> Source: https://docs.snowplow.io/docs/events/ootb-data/third-party-sources/

Snowplow provides tracking integrations that let you capture data from third-party tools deployed on your web and mobile properties.

## Optimizely X

The Optimizely X web plugin captures experiment and variation data from [Optimizely](https://www.optimizely.com/) A/B tests. The tracker will add it to all tracked events as an entity.

This table shows the support for Optimizely X tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                        |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | ---------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/optimizely/) | ✅         | 3.0.0         | ✅             | Requires Optimizely X plugin |
| iOS                                                           | ❌         |               |               |                              |
| Android                                                       | ❌         |               |               |                              |
| React Native                                                  | ❌         |               |               |                              |
| Flutter                                                       | ❌         |               |               |                              |
| Roku                                                          | ❌         |               |               |                              |
| Google Tag Manager                                            | ❌         |               |               |                              |

### `summary`

**Type:** Entity

Schema for Optimizely X experiment context

**Schema:** `iglu:com.optimizely.optimizelyx/summary/jsonschema/1-0-0`

**Example:**

```json
{
  "experimentId": 12345,
  "variationName": "Variation A",
  "variationId": 67890,
  "isActive": true
}
```

**Properties:**

| Property                 | Description                                               |
| ------------------------ | --------------------------------------------------------- |
| `experimentId` _integer_ | _Optional._ The Optimizely experiment ID                  |
| `variationName` _string_ | _Optional._ The variation name (requires privacy setting) |
| `variationId` _integer_  | _Optional._ The variation ID                              |
| `isActive` _boolean_     | _Optional._ Whether the experiment is active              |

## Google Analytics cookies

The GA cookies web plugin captures Google Analytics cookie values (both GA4 and Universal Analytics) and attaches them to all events.

This table shows the support for GA cookie tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                      |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | -------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/ga-cookies/) | ✅         | 3.0.0         | ✅             | Requires GA cookies plugin |
| iOS                                                           | ❌         |               |               |                            |
| Android                                                       | ❌         |               |               |                            |
| React Native                                                  | ❌         |               |               |                            |
| Flutter                                                       | ❌         |               |               |                            |
| Roku                                                          | ❌         |               |               |                            |
| Google Tag Manager                                            | ❌         |               |               |                            |

The entity schema depends on whether the tracker finds GA4 or Universal Analytics cookies.

GA4 schema:

### `cookies`

**Type:** Entity

Schema for GA4 cookies context

**Schema:** `iglu:com.google.ga4/cookies/jsonschema/1-0-0`

**Example:**

```json
{
  "_ga": "G-1234",
  "cookie_prefix": "prefix",
  "session_cookies": [
    {
      "measurement_id": "G-1234",
      "session_cookie": "567"
    }
  ]
}
```

**Properties:**

| Property                  | Description                                         |
| ------------------------- | --------------------------------------------------- |
| `_ga` _string_            | _Optional._ The \_ga cookie value                   |
| `cookie_prefix` _string_  | _Optional._ The cookie prefix if configured         |
| `session_cookies` _array_ | _Optional._ Session cookie values by measurement ID |

Universal Analytics schema:

### `cookies`

**Type:** Entity

Schema for Universal Analytics cookies context

**Schema:** `iglu:com.google.analytics/cookies/jsonschema/1-0-0`

**Example:**

```json
{
  "_ga": "GA1.2.3.4"
}
```

**Properties:**

| Property          | Description                       |
| ----------------- | --------------------------------- |
| `_ga` _string_    | _Optional._ The \_ga cookie value |
| `__utma` _string_ | _Optional._ Classic GA cookie     |
| `__utmb` _string_ | _Optional._ Classic GA cookie     |
| `__utmc` _string_ | _Optional._ Classic GA cookie     |
| `__utmv` _string_ | _Optional._ Classic GA cookie     |
| `__utmz` _string_ | _Optional._ Classic GA cookie     |

## Kantar Focal Meter

Some Snowplow trackers integrate with [Kantar Focal Meter](https://www.virtualmeter.co.uk/focalmeter), a router-based audience measurement system. The plugin sends user IDs to Focal Meter endpoints to enable content audience measurement.

This integration sends data to Kantar's endpoint, rather than attaching an entity to events.

This table shows the support for Kantar Focal Meter tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                       | Supported | Since version | Auto-tracking | Notes                       |
| ------------------------------------------------------------- | --------- | ------------- | ------------- | --------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/focalmeter/) | ✅         | 3.16.0        | ✅             | Requires Focal Meter plugin |
| [iOS](/docs/sources/mobile-trackers/plugins/focal-meter/)     | ✅         | 5.6.0         | ❌             |                             |
| [Android](/docs/sources/mobile-trackers/plugins/focal-meter/) | ✅         | 5.6.0         | ❌             |                             |
| React Native                                                  | ❌         |               |               |                             |
| Flutter                                                       | ❌         |               |               |                             |
| Roku                                                          | ❌         |               |               |                             |
| Google Tag Manager                                            | ❌         |               |               |                             |

## Privacy Sandbox (deprecated)

> **Warning:** Privacy Sandbox has been deprecated by Google. This plugin remains available but the underlying browser API may no longer be supported.

The Privacy Sandbox plugin captures data from the [Topics API](https://developer.chrome.com/docs/privacy-sandbox/topics/overview/), which was designed to provide privacy-preserving interest-based advertising signals.

This table shows the support for Privacy Sandbox tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/).

| Tracker                                                            | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/privacy-sandbox/) | ✅         | 3.14.0        | ✅             |
| iOS                                                                | ❌         |               |               |
| Android                                                            | ❌         |               |               |
| React Native                                                       | ❌         |               |               |
| Flutter                                                            | ❌         |               |               |
| Roku                                                               | ❌         |               |               |
| Google Tag Manager                                                 | ❌         |               |               |

### `topics`

**Type:** Entity

Schema for Privacy Sandbox Topics context

**Schema:** `iglu:com.google.privacysandbox/topics/jsonschema/1-0-0`

**Example:**

```json
{
  "topics": [
    {
      "topic": 123,
      "taxonomyVersion": "1",
      "modelVersion": "2"
    }
  ]
}
```

**Properties:**

| Property         | Description                                     |
| ---------------- | ----------------------------------------------- |
| `topics` _array_ | _Optional._ List of topics for the current user |

---

# User and session identifiers
> Track user and session identifiers including domain_userid, network_userid, session IDs, and custom business user IDs.
> Source: https://docs.snowplow.io/docs/events/ootb-data/user-and-session-identification/

Snowplow tracks a number of different identifiers to help you understand user behavior across sessions and devices. Read more about these in the [identifiers overview](/docs/events/identifiers/) page.

The main user and session identifiers are tracked as:

- Atomic event properties
- Session entity properties

## User atomic event properties

These [properties](/docs/fundamentals/canonical-event/#user-fields) can be assigned using most Snowplow trackers, regardless of the platform.

Some trackers set certain user properties automatically, while others require you to set them manually using the tracker's API.

If you don't manually set an IP address or `network_userid`, the event [Collector](/docs/pipeline/collector/) will automatically add them to the event based on the request HTTP headers and the Collector cookie. You can turn this feature off using [server-side anonymization](/docs/events/anonymous-tracking/).

| Tracker                                                                      | Supported | Since version | Notes                                                                                                                                                                                                    |
| ---------------------------------------------------------------------------- | --------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/tracking-events/#business-user-id)          | ✅         | 0.1.0         | Set `user_id` only. Tracker sets the `domain_userid`, `network_userid`, `domain_sessionid`, and `domain_sessionidx` automatically from [cookies](/docs/sources/web-trackers/cookies-and-local-storage/). |
| [iOS](/docs/sources/mobile-trackers/client-side-properties/)                 | ✅         | 0.5.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [Android](/docs/sources/mobile-trackers/client-side-properties/)             | ✅         | 0.1.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [React Native](/docs/sources/react-native-tracker/client-side-properties/)   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Flutter](/docs/sources/flutter-tracker/initialization-and-configuration/)   | ✅         | 0.1.0         | Set `user_id` only on web. Set `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress` on mobile.                                                                                             |
| [Roku](/docs/sources/roku-tracker/adding-data/)                              | ✅         | 0.2.0         | Set `user_id` and `network_userid` only. You can automatically populate `domain_userid`, `domain_sessionid`, and `domain_sessionidx` by configuring `sessionAsCookies`.                                  |
| [Node.js](/docs/sources/node-js-tracker/configuration/)                      | ✅         | 0.8.0         | Set all properties.                                                                                                                                                                                      |
| [Golang](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/)  | ✅         | 1.0.0         | Set all properties.                                                                                                                                                                                      |
| [.NET](/docs/sources/net-tracker/subject/)                                   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/) | ✅         | 0.10.0        | Set all properties.                                                                                                                                                                                      |
| [Python](/docs/sources/python-tracker/subject/)                              | ✅         | 0.13.0        | Set all properties.                                                                                                                                                                                      |
| [Scala](/docs/sources/scala-tracker/subject-methods/)                        | ✅         | 0.6.0         | Set all properties except `domain_sessionid` and `domain_sessionidx`.                                                                                                                                    |
| [Ruby](/docs/sources/ruby-tracker/adding-data-events/)                       | ✅         | 0.3.0         | Set all properties.                                                                                                                                                                                      |
| [Rust](/docs/sources/rust-tracker/initialization-and-configuration/)         | ✅         | 0.1.0         | Set all properties except `domain_sessionidx`.                                                                                                                                                           |
| [PHP](/docs/sources/php-tracker/subjects/)                                   | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/)          | ✅         | 0.1.0         | Set `user_id` and `user_ipaddress` only.                                                                                                                                                                 |
| [Unity](/docs/sources/unity-tracker/subject/)                                | ✅         | 0.1.0         | Set all properties.                                                                                                                                                                                      |
| [Lua](/docs/sources/lua-tracker/tracking-specific-events/)                   | ✅         | 0.1.0         | Set `user_id` only.                                                                                                                                                                                      |
| Google Tag Manager                                                           | ❌         |               | Set using custom configuration.                                                                                                                                                                          |

## Session entity

Some trackers can track an entity containing information about the current session.

This table shows the support for `client_session` entity tracking across the main [Snowplow tracker SDKs](/docs/sources/):

| Tracker                                                                              | Supported | Since version | Auto-tracking |
| ------------------------------------------------------------------------------------ | --------- | ------------- | ------------- |
| [Web](/docs/sources/web-trackers/tracking-events/session/)                           | ✅         | 3.5.0         | ✅             |
| [iOS](/docs/sources/mobile-trackers/tracking-events/session-tracking/)               | ✅         | 0.5.0         | ✅             |
| [Android](/docs/sources/mobile-trackers/tracking-events/session-tracking/)           | ✅         | 0.5.0         | ✅             |
| [React Native](/docs/sources/react-native-tracker/tracking-events/session-tracking/) | ✅         | 1.0.0         | ✅             |
| [Flutter](/docs/sources/flutter-tracker/sessions-and-data-model/)                    | ✅         | 0.2.0         | ✅             |
| [Roku](/docs/sources/roku-tracker/adding-data/)                                      | ✅         | 0.3.0         | ✅             |
| Node.js                                                                              | ❌         |               |               |
| Golang                                                                               | ❌         |               |               |
| [.NET](/docs/sources/net-tracker/setup/)                                             | ✅         | 1.0.0         | ✅             |
| Java                                                                                 | ❌         |               |               |
| Python                                                                               | ❌         |               |               |
| Scala                                                                                | ❌         |               |               |
| Ruby                                                                                 | ❌         |               |               |
| Rust                                                                                 | ❌         |               |               |
| PHP                                                                                  | ❌         |               |               |
| [C++](/docs/sources/c-tracker/client-sessions/)                                      | ✅         | 1.0.0         | ✅             |
| Unity                                                                                | ❌         |               |               |
| Lua                                                                                  | ❌         |               |               |
| [Google Tag Manager](/docs/sources/google-tag-manager/settings-template/)            | ✅         | v4            | ✅             |

The `client_session` entity contains data such as the session identifier and session index, as well as information about the previous session to help analysis.

The `userId` property is the device ID.

### `client_session`

**Type:** Event

Schema for a client-generated user session

**Schema:** `iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2`

**Example:**

```json
{
  "sessionIndex": 7,
  "sessionId": "bca0fa0e-853c-41cf-9cc4-15048f6f0ff5",
  "previousSessionId": "fa008142-c427-4289-8424-6fb2b6576692",
  "userId": "7a62ec9d-2aa0-4426-b014-eba2d0dcfebb",
  "firstEventId": "1548BE58-4CE7-4A32-A5E8-2696ECE941F4",
  "eventIndex": 66,
  "storageMechanism": "SQLITE",
  "firstEventTimestamp": "2022-01-01T00:00:00Z"
}
```

**Properties:**

| Property                     | Description                                                                                                                                                         |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `userId` _string_            | _Required._ An identifier for the user of the session. It is set on app install.                                                                                    |
| `sessionId` _string_         | _Required._ An identifier for the session                                                                                                                           |
| `sessionIndex` _integer_     | _Required._ The index of the current session for this user                                                                                                          |
| `eventIndex` _null_          | _Optional._ Optional index of the current event in the session                                                                                                      |
| `previousSessionId` _null_   | _Required._ The previous session identifier for this user                                                                                                           |
| `storageMechanism` _string_  | _Required._ The mechanism that the session information has been stored on the device Must be one of: `SQLITE`, `COOKIE_1`, `COOKIE_3`, `LOCAL_STORAGE`, `FLASH_LSO` |
| `firstEventId` _null_        | _Optional._ The optional identifier of the first event for this session                                                                                             |
| `firstEventTimestamp` _null_ | _Optional._ Optional date-time timestamp of when the first event in the session was tracked                                                                         |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_snowplowanalytics_snowplow_client_session_1 client_session_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'client_session'
and events.event_vendor = 'com.snowplowanalytics.snowplow'
```

---

# visionOS and SwiftUI
> Track window groups and immersive spaces in visionOS apps on Apple Vision Pro using SwiftUI context entities and events.
> Source: https://docs.snowplow.io/docs/events/ootb-data/visionos-swiftui/

Use the window group and immersive space entities and events to understand user behavior within visionOS apps on Apple's Vision Pro headset.

Track the opening and dismissing of SwiftUI window groups using `OpenWindowEvent` and `DismissWindowEvent`. These events can be used in any SwiftUI app, not just visionOS. The event data is sent as a window group entity attached to these events.

Use the `OpenImmersiveSpaceEvent` and `DismissImmersiveSpaceEvent` to automatically add an immersive space entity to all events occurring within an immersive space. The entity will identify the immersive space in which the events occurred. This feature is enabled by default for the iOS tracker.

This table shows the support for VisionOS tracking across the main client-side [Snowplow tracker SDKs](/docs/sources/). Snowplow currently only supports visionOS apps via SwiftUI, not Unity.

| Tracker                                                        | Supported | Since version | Auto-tracking | Notes                            |
| -------------------------------------------------------------- | --------- | ------------- | ------------- | -------------------------------- |
| Web                                                            | ❌         |               |               |                                  |
| [iOS](/docs/sources/mobile-trackers/tracking-events/visionos/) | ✅         | 6.0.0         | ✅/❌           | Entities are automatically added |
| Android                                                        | ❌         |               |               |                                  |
| React Native                                                   | ❌         |               |               |                                  |
| Flutter                                                        | ❌         |               |               |                                  |
| Roku                                                           | ❌         |               |               |                                  |
| Unity                                                          | ❌         |               |               |                                  |
| Google Tag Manager                                             | ❌         |               |               |                                  |

## Available events

The following events track user interactions with SwiftUI windows and visionOS immersive spaces. These events have no properties themselves; the contextual data is captured in the attached entities.

### Open window

Tracks when a user opens a SwiftUI window or window group.

### `open_window`

**Type:** Event

Schema for an event for opening a SwiftUI window or window group.

**Schema:** `iglu:com.apple.swiftui/open_window/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_apple_swiftui_open_window_1 open_window_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'open_window'
and events.event_vendor = 'com.apple.swiftui'
```

### Dismiss window

Tracks when a user closes or dismisses a SwiftUI window or window group.

### `dismiss_window`

**Type:** Event

Schema for an event for dismissing a SwiftUI window or window group.

**Schema:** `iglu:com.apple.swiftui/dismiss_window/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_apple_swiftui_dismiss_window_1 dismiss_window_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'dismiss_window'
and events.event_vendor = 'com.apple.swiftui'
```

### Open immersive space

Tracks when a user enters an immersive space in a visionOS app on Apple Vision Pro.

### `open_immersive_space`

**Type:** Event

Schema for an event for opening a visionOS immersive space.

**Schema:** `iglu:com.apple.swiftui/open_immersive_space/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_apple_swiftui_open_immersive_space_1 open_immersive_space_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'open_immersive_space'
and events.event_vendor = 'com.apple.swiftui'
```

### Dismiss immersive space

Tracks when a user exits an immersive space in a visionOS app.

### `dismiss_immersive_space`

**Type:** Event

Schema for an event for dismissing a visionOS immersive space.

**Schema:** `iglu:com.apple.swiftui/dismiss_immersive_space/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_apple_swiftui_dismiss_immersive_space_1 dismiss_immersive_space_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'dismiss_immersive_space'
and events.event_vendor = 'com.apple.swiftui'
```

## Available entities

These entities provide context about the SwiftUI windows and visionOS immersive spaces where events occur.

### Window group

Describes a SwiftUI window group, including its identifier, title, and window style. This entity is attached to open and dismiss window events.

### `window_group`

**Type:** Entity

Schema for a window group entity, representing the SwiftUI window group that the event occurs in.

**Schema:** `iglu:com.apple.swiftui/window_group/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "group1",
  "window_id": "BC374B59-B8E7-4F09-B100-FD5F9AAC0E27",
  "title_key": "window1",
  "window_style": "automatic"
}
```

**Properties:**

| Property                | Description                                                                                                                                                               |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `window_id` _string_    | _Optional._ UUID for the current window within the group.                                                                                                                 |
| `id` _string_           | _Required._ A string that uniquely identifies the window group. Identifiers must be unique among the window groups in your app.                                           |
| `title_key` _string_    | _Optional._ A localized string key to use for the window's title in system menus and in the window's title bar. Provide a title that describes the purpose of the window. |
| `window_style` _string_ | _Optional._ A specification for the appearance and interaction of a window. Must be one of: `automatic`, `hiddenTitleBar`, `plain`, `titleBar`, `volumetric`,``           |

### Immersive space

Describes a visionOS immersive space, including its immersion style and upper limb visibility settings. This entity is automatically attached to all events that occur within an immersive space.

### `immersive_space`

**Type:** Entity

Schema for an immersive space entity, representing the VisionOS immersive space that the event occurs in.

**Schema:** `iglu:com.apple.swiftui/immersive_space/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "space1",
  "view_id": "C0A92B47-9654-4889-BC95-97E1C3721A53",
  "immersion_style": "mixed",
  "upper_limb_visibility": "visible"
}
```

**Properties:**

| Property                         | Description                                                                                                                                                  |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `view_id` _string_               | _Optional._ UUID for the view of the immersive space.                                                                                                        |
| `id` _string_                    | _Required._ The identifier of the immersive space to present.                                                                                                |
| `immersion_style` _string_       | _Optional._ The style of an immersive space. Must be one of: `automatic`, `full`, `mixed`, `progressive`,``                                                  |
| `upper_limb_visibility` _string_ | _Optional._ Preferred visibility of the user's upper limbs, while an immersive space scene is presented. Must be one of: `automatic`, `visible`, `hidden`,`` |

---

# Timestamps
> Understanding Snowplow event timestamps from creation through the pipeline, including derived timestamp calculation and usage recommendations.
> Source: https://docs.snowplow.io/docs/events/timestamps/

Snowplow events have multiple timestamps that are added as the payload moves through the pipeline. The set of timestamps is designed to account for devices with incorrectly set clocks, or delays in event sending due to network outages. All timestamps are converted to UTC for consistency across events.

The timestamps are:

| Timestamp             | Added by                                                    | Description                                                                                                                                                    | In all events |
| --------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `dvce_created_tstamp` | [Tracker](/docs/sources/)                                   | The device's timestamp when the event was created - not necessarily the correct time                                                                           | ✅             |
| `dvce_sent_tstamp`    | [Tracker](/docs/sources/)                                   | The device's timestamp when the event was successfully sent to the Collector endpoint - not necessarily the correct time                                       | ✅             |
| `true_tstamp`         | You                                                         | Exact timestamp, defined within your tracking code                                                                                                             | ❌             |
| `collector_tstamp`    | [Collector](/docs/pipeline/collector/)                      | When the Collector received the event payload                                                                                                                  | ✅             |
| `derived_tstamp`      | [Enrich](/docs/api-reference/enrichment-components/)        | Calculated from other timestamps, or the same as `true_tstamp`                                                                                                 | ✅             |
| `etl_tstamp`          | [Enrich](/docs/api-reference/enrichment-components/)        | The time at which Enrich started processing the event                                                                                                          | ✅             |
| `refr_device_tstamp`  | [Enrich](/docs/api-reference/enrichment-components/)        | Timestamp extracted from the [cross-domain navigation query string](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/), if present | ❌             |
| `load_tstamp`         | [Loader](/docs/destinations/warehouses-lakes/) or warehouse | Timestamp when the event was loaded into the warehouse                                                                                                         | ✅             |

The `load_tstamp` is added either by the Loader or by the warehouse at the point of loading, depending on the Loader/warehouse. Use this timestamp for incremental processing.

## Derived timestamp

The `derived_tstamp` corrects for inaccurate timestamps and delays.

It's calculated as `collector_tstamp` minus the difference between the `dvce_sent_tstamp` and the `dvce_created_tstamp`. Alternatively, if `true_tstamp` is available, `derived_tstamp` is the same as `true_tstamp`.

The calculation has two assumptions:

- We assume that, although `dvce_created_tstamp` and `dvce_sent_tstamp` can both be inaccurate, they're inaccurate in precisely the same way. If the device clock is 15 minutes fast at event creation, then it remains 15 minutes fast at event sending, whenever that might be.
- We assume that the time taken for an event to get from the device to the Collector is neglible, i.e. the lag between `dvce_sent_tstamp` and `collector_tstamp` is 0 seconds.

## What timestamp to use?

The choice of timestamp for downstream analysis depends on how you're using the data.

For reporting, we recommend using the `derived_tstamp`. Our data models use `derived_tstamp` as the default `start_tstamp` for creating derived tables.

To filter the `atomic.events` table directly, use the timestamp it's partitioned on. For example, if your warehouse is partitioned by `load_tstamp`, then use clauses like `WHERE load_tstamp > ?` in your analysis.

The `refr_device_tstamp` timestamp can help you understand cross-domain navigation by your users.

All of the timestamps can also be used for custom use cases, or in debugging, to closely follow how the events are created and move through the pipeline.

---

# Snowplow atomic event properties
> A summary of the Snowplow events table and its standard fields
> Source: https://docs.snowplow.io/docs/fundamentals/canonical-event/

This page provides a reference for the standard fields that are found in all Snowplow events. These fields define the Snowplow tracker protocol, and are often called "atomic properties" or "atomic fields" in reference to the [Snowplow `atomic.events` table](/docs/fundamentals/warehouse-tables/). They're also sometimes called "canonical fields".

Fields that are populated during tracking have two names associated with them:

- **Payload property**: a short payload name, to reduce the size of the HTTP request
- **Field name**: a more descriptive column or field name

The field **type** refers to the data type in the enriched event data. For some fields, the type is different for the raw tracker payload and the enriched event. Again, this is to reduce the size of the HTTP request.

Different event fields are populated by different applications, such as tracker SDKs or [enrichments](/docs/pipeline/enrichments/). The **source** of data for each field is indicated in the tables below.

## Validation

During enrichment, atomic property values are validated against the [atomic schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0). If a required field is missing or invalid, for example the wrong type or length, [Enrich](/docs/api-reference/enrichment-components/) will process the event as a [failed event](/docs/fundamentals/failed-events/). The **reqd?** values in these tables specify whether a field is required for a Snowplow event.

The tables show the default maximum lengths for string fields. You can configure different maximum lengths for specific fields by setting `atomicFieldsLimits` in the [Enrich configuration](/docs/api-reference/enrichment-components/configuration-reference/). These can be more permissive than the limits in the atomic schema. In fact, the `page_url`, `page_referrer`, and `mkt_clickid` fields already have larger maximum lengths in the default Enrich configuration than in the atomic schema.

Some identifiers require UUID strings. Snowplow components use UUIDv4, but the pipeline will accept any valid UUID, regardless of version.

Any payload that conforms to this protocol is a valid Snowplow event payload, whether it's sent by a Snowplow tracker SDK, a webhook, or a custom application. If you want to get into the details, check out these [example HTTP requests](/docs/events/http-requests/). In total, the tracker protocol defines 131 fields, of which 89 are in use by Snowplow applications.

## Common fields

These fields are common to all Snowplow events, regardless of platform or event type.

### Event fields

The `event_id` is the unique identifier (UUID) for the table row. This should be generated by trackers, but if missing will be generated by the enrichment process.

| Payload property | Field name          | Type                     | Description                                           | Reqd? | Example                                | Source                                                                                                         | Web | Mobile |
| ---------------- | ------------------- | ------------------------ | ----------------------------------------------------- | ----- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------- | --- | ------ |
| `e`              | `event`             | `string`, max length 128 | The type of event recorded                            | Yes   | `page_view`                            | Tracking                                                                                                       | ✅   | ✅      |
| `eid`            | `event_id`          | UUID `string`            | A UUID for each event                                 | Yes   | `c6ef3124-b53a-4b13-a233-0088f79dcbcb` | Tracking (or enrichment if empty)                                                                              | ✅   | ✅      |
|                  | `event_fingerprint` | `string`, max length 128 | Hash client-set event fields, used to de-dupe records | No    | `AADCE520E20C2899F4CED228A79A3083`     | [Event fingerprint enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/) | ✅   | ✅      |

This table shows the possible values for the `event` field:

| Event type              | Payload `e` value | Field `event` value                  |
| ----------------------- | ----------------- | ------------------------------------ |
| Self-describing event   | `ue`              | `unstruct`                           |
| Page view               | `pv`              | `page_view`                          |
| Page ping               | `pp`              | `page_ping`                          |
| Structured event        | `se`              | `struct`                             |
| Legacy ecommerce events | `tr` and `ti`     | `transaction` and `transaction_item` |

### User fields

Read more about tracking and using these fields in the [OOTB user and session data](/docs/events/ootb-data/user-and-session-identification/) and [identifiers](/docs/events/identifiers/) pages.

The `domain_userid` is regarded as the most reliable session based identifier for most use cases. It's treated as the primary `user_identifier` field in our data models that rely on sessionization, including the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) data model.

The `domain_sessionidx` is the number or index of the current user session. For example, an event occurring during a user's first session would have `domain_sessionidx` set to 1. The JavaScript tracker calculates this field by storing a visit count in a [first-party cookie](/docs/sources/web-trackers/cookies-and-local-storage/).

The equivalent values on mobile are tracked in a [session entity](/docs/events/ootb-data/user-and-session-identification/#session-entity).

The `network_userid` is set by a [Collector cookie](/docs/pipeline/collector/) by default. You can override it by setting a `network_userid` with your tracker.

The IP address is also added to the event by the Collector, if you didn't provide one in your tracking code. Snowplow supports both IPv4 and IPv6 addresses, as long as a domain is compatible with the IPv6 network.

| Payload property | Field name          | Type                     | Description                                                                    | Reqd? | Example                                | Source                                                                                                                   | Web | Mobile |
| ---------------- | ------------------- | ------------------------ | ------------------------------------------------------------------------------ | ----- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --- | ------ |
| `uid`            | `user_id`           | `string`, max length 255 | Unique identifier for user, set by the business using `setUserId`              | No    | `c94f860b-1266-4dad-ae57-3a36a414a521` | Tracking                                                                                                                 | ✅   | ✅      |
| `duid`           | `domain_userid`     | `string`, max length 128 | Unique identifier for a user, based on a first party cookie                    | No    | `4b0dfa75-9a8c-46a1-9691-01add9db4200` | Tracking                                                                                                                 | ✅   | ❌      |
| `tnuid`          | `network_userid`    | `string`, max length 128 | User ID set by Snowplow using server-set cookie                                | No    | `ecdff4d0-9175-40ac-a8bb-325c49733607` | Tracking or pipeline                                                                                                     | ✅   | ✅      |
| `sid`            | `domain_sessionid`  | UUID `string`            | Unique identifier (UUID) for this visit of this `domain_userid` to this domain | No    | `c6ef3124-b53a-4b13-a233-0088f79dcbcb` | Tracking                                                                                                                 | ✅   | ❌      |
| `vid`            | `domain_sessionidx` | `integer`                | Index of number of visits that this `domain_userid` has made to this domain    | No    | `3`                                    | Tracking                                                                                                                 | ✅   | ❌      |
| `ip`             | `user_ipaddress`    | `string`, max length 128 | User IP address, can be overwritten with the IP anonymization enrichment       | No    | `92.231.54.234`                        | Tracking or [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/) | ✅   | ✅      |

### Application fields

Use the application ID to distinguish events tracked from different applications or websites by the same Snowplow stack.

The platform ID is used to distinguish the same app running on different platforms, e.g. `iOS` vs `web`.

| Payload property | Field name     | Type                     | Description              | Reqd? | Example             | Source   | Web | Mobile |
| ---------------- | -------------- | ------------------------ | ------------------------ | ----- | ------------------- | -------- | --- | ------ |
| `aid`            | `app_id`       | `string`, max length 255 | Application ID           | Yes   | `snow-game-android` | Tracking | ✅   | ✅      |
| `p`              | `platform`     | `string`, max length 255 | Platform the app runs on | Yes   | `web`               | Tracking | ✅   | ✅      |
| `tna`            | `name_tracker` | `string`, max length 128 | Tracker namespace        | No    | `tracker_1`         | Tracking | ✅   | ✅      |

This table shows the possible values for the `platform` field:

| Platform                  | `platform` value |
| ------------------------- | ---------------- |
| Web, including mobile web | `web`            |
| Mobile, tablet            | `mob`            |
| Desktop, laptop           | `pc`             |
| Server-side application   | `srv`            |
| General application       | `app`            |
| Connected TV              | `tv`             |
| Games console             | `cnsl`           |
| Internet of Things        | `iot`            |
| Headset (e.g., AR, VR)    | `headset`        |

> **Info:** The tracker namespace parameter is used to distinguish between different trackers. The name can be any string that doesn't contain a colon or semicolon character. Tracker namespacing allows you to run multiple trackers, pinging to different collectors.

### Time and date fields

The `etl_tstamp` field records when the event was validated and enriched, not when it was loaded into the warehouse. The name is historical.

Most Snowplow trackers have built-in capability for tracking `os_timezone`. See the [geolocation and timezone tracking](/docs/events/ootb-data/geolocation/) page for tracker-specific configuration details.

| Payload property | Field name            | Type                     | Description                                                 | Reqd? | Example                   | Source             | Web | Mobile |
| ---------------- | --------------------- | ------------------------ | ----------------------------------------------------------- | ----- | ------------------------- | ------------------ | --- | ------ |
|                  | `collector_tstamp`    | `timestamp`              | Timestamp for the event recorded by the Collector           | Yes   | `2013-11-26 00:02:05.123` | Pipeline           | ✅   | ✅      |
| `dtm`            | `dvce_created_tstamp` | `timestamp`              | Timestamp for the event recorded on the client device       | No    | `2013-11-26 00:03:57.885` | Tracking           | ✅   | ✅      |
| `stm`            | `dvce_sent_tstamp`    | `timestamp`              | Timestamp when event occurred, as recorded by client device | No    | `2013-11-26 00:03:58.032` | Tracking           | ✅   | ✅      |
|                  | `etl_tstamp`          | `timestamp`              | Timestamp for when the event was validated and enriched     | No    | `2017-01-26 00:01:25.292` | Pipeline           | ✅   | ✅      |
|                  | `derived_tstamp`      | `timestamp`              | Timestamp making allowance for inaccurate device clock      | No    | `2013-11-26 00:02:04.123` | Default enrichment | ✅   | ✅      |
| `ttm`            | `true_tstamp`         | `timestamp`              | User-set "true timestamp" for the event                     | No    | `2013-11-26 00:02:04.123` | Tracking           | ✅   | ✅      |
|                  | `load_tstamp`         | `timestamp`              | Timestamp for when the data was loaded into the warehouse   | No    | `2013-11-26 00:02:04.123` | Pipeline           | ✅   | ✅      |
| `tz`             | `os_timezone`         | `string`, max length 255 | Client operating system timezone                            | No    | `Europe/London`           | Tracking           | ✅   | ✅      |

The trackers send timestamp properties as `int` in the payload, representing milliseconds since the Unix epoch, e.g. `1361553733313`. They're converted to `timestamp` type during enrichment. Similarly, the `tz` property is URL-encoded in the payload, e.g. `Europe%2FLondon`, and decoded during enrichment.

Read more about Snowplow timestamps [here](/docs/events/timestamps/).

### Device and operating system fields

The `dvce_screenheight` and `dvce_screenwidth` screen resolution fields originate from a single `res` payload property. It's a string with the format `"<width>x<height>"`, e.g. `1900x1024`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name          | Type                      | Description             | Reqd? | Example                                                                                | Source               | Web | Mobile |
| ---------------- | ------------------- | ------------------------- | ----------------------- | ----- | -------------------------------------------------------------------------------------- | -------------------- | --- | ------ |
| `ua`             | `useragent`         | `string`, max length 1000 | Raw useragent           | No    | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0` | Tracking or pipeline | ✅   | ✅      |
| `res`            | `dvce_screenheight` | `integer`                 | Screen height in pixels | No    | `1024`                                                                                 | Tracking             | ✅   | ✅      |
| `res`            | `dvce_screenwidth`  | `integer`                 | Screen width in pixels  | No    | `1900`                                                                                 | Tracking             | ✅   | ✅      |

For more information on this topic check out the [device data](/docs/events/ootb-data/device-and-browser/) page.

### IP address fields

These fields are populated by the [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/), based on the `user_ipaddress` field.

| Field name        | Type                     | Description                                                                                | Reqd? | Example              | Source                                                                                  | Web | Mobile |
| ----------------- | ------------------------ | ------------------------------------------------------------------------------------------ | ----- | -------------------- | --------------------------------------------------------------------------------------- | --- | ------ |
| `ip_isp`          | `string`, max length 100 | User's ISP                                                                                 | No    | `FDN Communications` | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_organization` | `string`, max length 128 | Organization associated with the user's IP address - defaults to ISP name if none is found | No    | `Bouygues Telecom`   | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_domain`       | `string`, max length 128 | Second level domain name associated with the user's IP address                             | No    | `nuvox.net`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `ip_netspeed`     | `string`, max length 100 | User's connection type                                                                     | No    | `Cable/DSL`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |

### Location fields

These fields are populated by the [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/), based on the IP address.

| Field name        | Type                     | Description                                            | Reqd? | Example              | Source                                                                                  | Web | Mobile |
| ----------------- | ------------------------ | ------------------------------------------------------ | ----- | -------------------- | --------------------------------------------------------------------------------------- | --- | ------ |
| `geo_country`     | `string`, max length 2   | ISO 3166-1 code for the country the user is located in | No    | `GB`, `US`           | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_region`      | `string`, max length 3   | ISO-3166-2 code for country region the user is in      | No    | `I9`, `TX`           | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_city`        | `string`, max length 75  | City the user is in                                    | No    | `New York`, `London` | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_zipcode`     | `string`, max length 15  | Postcode the user is in                                | No    | `94109`              | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_latitude`    | `float`                  | User location latitude                                 | No    | `37.443604`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_longitude`   | `float`                  | User location longitude                                | No    | `-122.4124`          | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_region_name` | `string`, max length 100 | User region name                                       | No    | `Florida`            | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |
| `geo_timezone`    | `string`, max length 64  | User timezone name                                     | No    | `Europe/London`      | [IP enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) | ✅   | ✅      |

For more information on this topic check out the [geolocation data](/docs/events/ootb-data/geolocation/) page.

### Marketing fields

These fields are populated by the [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/).

| Field name     | Type                      | Description                                                 | Reqd? | Example                                 | Source                                                                                                               | Web | Mobile |
| -------------- | ------------------------- | ----------------------------------------------------------- | ----- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --- | ------ |
| `mkt_medium`   | `string`, max length 255  | Type of traffic source                                      | No    | 'cpc', 'affiliate', 'organic', 'social' | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_source`   | `string`, max length 255  | The company or website where the traffic came from          | No    | 'Google', 'Facebook'                    | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_term`     | `string`, max length 255  | Any keywords associated with the referrer                   | No    | 'new age tarot decks'                   | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_content`  | `string`, max length 500  | The content of the ad, or an ID so that it can be looked up | No    | 13894723                                | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_campaign` | `string`, max length 255  | The campaign ID                                             | No    | 'diageo-123'                            | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_clickid`  | `string`, max length 1000 | The click ID                                                | No    | 'ac3d8e459'                             | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |
| `mkt_network`  | `string`, max length 64   | The ad network to which the click ID belongs                | No    | 'DoubleClick'                           | [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) | ✅   | ✅      |

### Cross-domain tracking fields

The pipeline populates these fields by default if the event's `url` has a [cross-navigation querystring](/docs/events/cross-navigation/).

| Payload property | Field name           | Type                     | Description                                                   | Reqd? | Example               | Source             | Web | Mobile |
| ---------------- | -------------------- | ------------------------ | ------------------------------------------------------------- | ----- | --------------------- | ------------------ | --- | ------ |
|                  | `refr_domain_userid` | `string`, max length 128 | The Snowplow `domain_userid` of the referring website         | No    | `bc2e92ec6c204a14`    | Default enrichment | ✅   | ✅      |
|                  | `refr_dvce_tstamp`   | timestamp                | The time of attaching the `domain_userid` to the inbound link | No    | `2013-11-26 00:02:05` | Default enrichment | ✅   | ✅      |

### Snowplow versions fields

These fields record the versions of the various Snowplow components involved in processing the event.

| Payload property | Field name    | Type                     | Description              | Reqd? | Example                             | Source             | Web | Mobile |
| ---------------- | ------------- | ------------------------ | ------------------------ | ----- | ----------------------------------- | ------------------ | --- | ------ |
| `tv`             | `v_tracker`   | `string`, max length 100 | Tracker version          | Yes   | `js-3.0.0`                          | Tracking           | ✅   | ✅      |
|                  | `v_collector` | `string`, max length 100 | Collector version        | Yes   | `ssc-2.1.0-kinesis`                 | Pipeline           | ✅   | ✅      |
|                  | `v_etl`       | `string`, max length 100 | ETL application versions | Yes   | `snowplow-micro-1.1.0-common-1.4.2` | Default enrichment | ✅   | ✅      |

## Web-specific fields

These fields apply only to events from web.

### Page fields

The `url`, `refr` and `page` payload properties are URL-encoded in the payload, e.g. `https%3A%2F%2Fwww.snowplow.io%2F` or `Snowplow%20Behavoral%20Data`, and decoded during enrichment.

The `url` property is further parsed during enrichment to populate the various `page_url*` fields. Similarly, the `refr` property is parsed to populate the various `refr_*` fields.

| Payload property | Field name         | Type                       | Description                                                            | Reqd? | Example                                                             | Source                                                                                                     | Web | Mobile |
| ---------------- | ------------------ | -------------------------- | ---------------------------------------------------------------------- | ----- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | --- | ------ |
| `url`            | `page_url`         | `string`, max length 10000 | The page URL                                                           | No    | `https://www.snowplow.io/product/index.html?id=GTM-DLRG#conclusion` | Tracking                                                                                                   | ✅   | ❌      |
|                  | `page_urlscheme`   | `string`, max length 16    | Scheme, also known as protocol                                         | No    | `https`                                                             | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlhost`     | `string`, max length 255   | Host, also known as domain                                             | No    | `www.snowplow.io`                                                   | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlport`     | `integer`                  | Port if specified, scheme dependent if not; 443 for HTTPS, 80 for HTTP | No    | `443`                                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlpath`     | `string`, max length 3000  | Path to page                                                           | No    | `/product/index.html`                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlquery`    | `string`, max length 6000  | Querystring                                                            | No    | `id=GTM-DLRG`                                                       | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `page_urlfragment` | `string`, max length 3000  | Fragment, also known as anchor                                         | No    | `conclusion`                                                        | Default enrichment                                                                                         | ✅   | ❌      |
| `page`           | `page_title`       | `string`, max length 2000  | Web page title                                                         | No    | `Snowplow Behavioral Data`                                          | Tracking                                                                                                   | ✅   | ❌      |
| `refr`           | `page_referrer`    | `string`, max length 10000 | URL of the referrer                                                    | No    | `https://www.google.com/search?q=psychic+oracle+cards`              | Tracking                                                                                                   | ✅   | ❌      |
|                  | `refr_urlscheme`   | `string`, max length 16    | Referrer scheme                                                        | No    | `https`                                                             | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlhost`     | `string`, max length 255   | Referrer host                                                          | No    | `www.google.com`                                                    | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlport`     | `integer`                  | Referrer port                                                          | No    | `443`                                                               | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlpath`     | `string`, max length 6000  | Referrer page path                                                     | No    | `/search`                                                           | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlquery`    | `string`, max length 6000  | Referrer URL querystring                                               | No    | `q=psychic+oracle+cards`                                            | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_urlfragment` | `string`, max length 3000  | Referrer URL fragment                                                  | No    |                                                                     | Default enrichment                                                                                         | ✅   | ❌      |
|                  | `refr_medium`      | `string`, max length 25    | Type of referrer                                                       | No    | `search`                                                            | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |
|                  | `refr_source`      | `string`, max length 50    | Name of referrer if recognized                                         | No    | `Google`                                                            | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |
|                  | `refr_term`        | `string`, max length 255   | Keywords if source is a search engine                                  | No    | `psychic oracle cards`                                              | [Referrer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) | ✅   | ❌      |

### Document fields

The `doc_width` and `doc_height` page size fields originate from a single `ds` payload property. It's a string with the format `"<width>x<height>"`, e.g. `3000x1024`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name    | Type                     | Description                   | Reqd? | Example | Source   | Web | Mobile |
| ---------------- | ------------- | ------------------------ | ----------------------------- | ----- | ------- | -------- | --- | ------ |
| `cs`             | `doc_charset` | `string`, max length 128 | The page's character encoding | No    | `UTF-8` | Tracking | ✅   | ❌      |
| `ds`             | `doc_width`   | `integer`                | The page's width in pixels    | No    | `1024`  | Tracking | ✅   | ❌      |
| `ds`             | `doc_height`  | `integer`                | The page's height in pixels   | No    | `3000`  | Tracking | ✅   | ❌      |

### Browser fields

The `br_viewwidth` and `br_viewheight` page size fields originate from a single `vp` payload property. It's a string with the format `"<width>x<height>"`, e.g. `1000x1000`. During enrichment, this string is split into the two separate integer fields.

| Payload property | Field name      | Type                     | Description                              | Reqd? | Example | Source   | Web | Mobile |
| ---------------- | --------------- | ------------------------ | ---------------------------------------- | ----- | ------- | -------- | --- | ------ |
| `lang`           | `br_lang`       | `string`, max length 255 | Language the browser is set to           | No    | `en-GB` | Tracking | ✅   | ❌      |
| `cookie`         | `br_cookies`    | `integer boolean`        | Whether the browser is accepting cookies | No    | `1`     | Tracking | ✅   | ❌      |
| `cd`             | `br_colordepth` | `string`, max length 12  | Bit depth of the browser color palette   | No    | `24`    | Tracking | ✅   | ❌      |
| `vp`             | `br_viewwidth`  | `integer`                | Viewport width                           | No    | `1000`  | Tracking | ✅   | ❌      |
| `vp`             | `br_viewheight` | `integer`                | Viewport height                          | No    | `1000`  | Tracking | ✅   | ❌      |

For more information on this topic check out the [device and browser data](/docs/events/ootb-data/device-and-browser/) page.

## Baked-in event fields

[Baked-in event](/docs/fundamentals/events/#baked-in-events) data is stored in specific default fields, rather than in self-describing event or entity columns.

### Page views

There are no fields that are specific to page view events. All the relevant fields are included in the standard fields available for any web-based event e.g. `page_url`, or `page_title`.

### Page pings

Page ping events include four additional fields. They indicate how a user has scrolled over a web page since the last page ping:

| Payload property | Field name       | Type      | Description                                        | Reqd? | Example |
| ---------------- | ---------------- | --------- | -------------------------------------------------- | ----- | ------- |
| `pp_mix`         | `pp_xoffset_min` | `integer` | Minimum page x offset seen in the last ping period | No    | `10`    |
| `pp_max`         | `pp_xoffset_max` | `integer` | Maximum page x offset seen in the last ping period | No    | `100`   |
| `pp_miy`         | `pp_yoffset_min` | `integer` | Minimum page y offset seen in the last ping period | No    | `5`     |
| `pp_may`         | `pp_yoffset_max` | `integer` | Maximum page y offset seen in the last ping period | No    | `200`   |

### Structured events

Structured events use these fields to capture event data:

| Payload property | Field name    | Type                      | Description                                                                                     | Reqd?  | Example                       |
| ---------------- | ------------- | ------------------------- | ----------------------------------------------------------------------------------------------- | ------ | ----------------------------- |
| `se_ca`          | `se_category` | `string`, max length 1000 | Category of event                                                                               | Yes \* | `ecomm`, `video`              |
| `se_ac`          | `se_action`   | `string`, max length 1000 | Action performed, or event name                                                                 | Yes \* | `add-to-basket`, `play-video` |
| `se_la`          | `se_label`    | `string`, max length 4096 | The object of the action e.g. the ID of the video played. or SKU of the product added to basket | No     | `pbz00123`                    |
| `se_pr`          | `se_property` | `string`, max length 1000 | A property associated with the object of the action                                             | No     | `HD`, `large`                 |
| `se_va`          | `se_value`    | `decimal`                 | A value associated with the event or action e.g. the value of goods added-to-basket             | No     | `9.99`                        |

\* These fields are only required for structured events.

## Self-describing event fields

For [self-describing events](/docs/fundamentals/events/#self-describing-events), the pipeline extracts data from the event schema and populates the following fields:

| Field name      | Type                      | Description                           | Reqd? | Example      | Source             | Web | Mobile |
| --------------- | ------------------------- | ------------------------------------- | ----- | ------------ | ------------------ | --- | ------ |
| `event_vendor`  | `string`, max length 1000 | Business that defined the event       | Yes   | `com.acme`   | Default enrichment | ✅   | ✅      |
| `event_name`    | `string`, max length 1000 | Event name                            | Yes   | `link_click` | Default enrichment | ✅   | ✅      |
| `event_format`  | `string`, max length 128  | Format for event; always `jsonschema` | Yes   | `jsonschema` | Default enrichment | ✅   | ✅      |
| `event_version` | `string`, max length 128  | Version of event schema               | Yes   | `1-0-2`      | Default enrichment | ✅   | ✅      |

> **Note:** Data in self-describing events or entities is added as additional columns, rather than in these standard atomic fields.
>
> Check out the [warehouse table structure](/docs/fundamentals/warehouse-tables/) page to learn more.

## Deprecated fields

We have deprecated the following fields. They're no longer populated by Snowplow trackers or enrichments, but they're still part of the Snowplow tracker protocol:

| Field name                 | Type                     | Description                                                                | Reqd? | Example      | Source     | Web | Mobile |
| -------------------------- | ------------------------ | -------------------------------------------------------------------------- | ----- | ------------ | ---------- | --- | ------ |
| `txn_id`                   | `integer`                | Transaction ID set client-side, used to de-dupe records                    | No    | `421828`     | Deprecated | ✅   | ✅      |
| `dvce_type`                | `string`, max length 50  | Type of device                                                             | No    | `Computer`   | Deprecated | ✅   | ✅      |
| `dvce_ismobile`            | `integer boolean`        | Is the device mobile?                                                      | No    | `1`          | Deprecated | ✅   | ✅      |
| `os_name`                  | `string`, max length 50  | Name of operating system                                                   | No    | `Android`    | Deprecated | ✅   | ✅      |
| `os_family`                | `string`, max length 50  | Operating system family                                                    | No    | `Linux`      | Deprecated | ✅   | ✅      |
| `os_manufacturer`          | `string`, max length 50  | Company responsible for OS                                                 | No    | `Apple`      | Deprecated | ✅   | ✅      |
| `user_fingerprint`         | `string`, max length 128 | A user fingerprint generated by looking at the individual browser features | No    | `2161814971` | Deprecated | ❌   | ❌      |
| `br_name`                  | `string`, max length 50  | Browser name                                                               | No    | `Firefox 12` | Deprecated | ❌   | ❌      |
| `br_version`               | `string`, max length 50  | Browser version                                                            | No    | `12.0`       | Deprecated | ❌   | ❌      |
| `br_family`                | `string`, max length 50  | Browser family                                                             | No    | `Firefox`    | Deprecated | ❌   | ❌      |
| `br_type`                  | `string`, max length 50  | Browser type                                                               | No    | `Browser`    | Deprecated | ❌   | ❌      |
| `br_renderengine`          | `string`, max length 50  | Browser rendering engine                                                   | No    | `GECKO`      | Deprecated | ❌   | ❌      |
| `br_features_pdf`          | `integer boolean`        | Whether the browser recognizes PDFs                                        | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_flash`        | `integer boolean`        | Whether Flash is installed                                                 | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_java`         | `integer boolean`        | Whether Java is installed                                                  | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_director`     | `integer boolean`        | Whether Adobe Shockwave is installed                                       | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_quicktime`    | `integer boolean`        | Whether QuickTime is installed                                             | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_realplayer`   | `integer boolean`        | Whether RealPlayer is installed                                            | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_windowsmedia` | `integer boolean`        | Whether mplayer2 is installed                                              | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_gears`        | `integer boolean`        | Whether Google Gears is installed                                          | No    | `1`          | Deprecated | ✅   | ❌      |
| `br_features_silverlight`  | `integer boolean`        | Whether Microsoft Silverlight is installed                                 | No    | `1`          | Deprecated | ✅   | ❌      |

### Legacy ecommerce fields

There are a large number of fields in the tracker protocol specifically for [legacy ecommerce](/docs/fundamentals/events/#baked-in-events) `transaction` and `transaction_item` events.

Fields that start `tr_` relate to the transaction as a whole. Fields that start `ti_` refer to the specific item included in the transaction, e.g. a product in the basket.

Single transactions typically span multiple lines of data: there will be a single line where `event` = `transaction`, where the `tr_` fields are set, and multiple lines (one for each product included) where `event` = `transaction_item` and the `ti_` fields are set.

| Field name            | Type                     | Description                                                | Reqd? | Example         |
| --------------------- | ------------------------ | ---------------------------------------------------------- | ----- | --------------- |
| `tr_orderid`          | `string`, max length 255 | Order ID                                                   | Yes   | `#134`          |
| `tr_affiliation`      | `string`, max length 255 | Transaction affiliation (e.g. store where sale took place) | No    | `web`           |
| `tr_total`            | `decimal`                | Total transaction value                                    | Yes   | `12.99`         |
| `tr_tax`              | `decimal`                | Total tax included in transaction value                    | No    | `3.00`          |
| `tr_shipping`         | `decimal`                | Delivery cost charged                                      | No    | `0.00`          |
| `tr_total_base` \*    | `decimal`                | Total in base currency                                     | No    | `12.99`         |
| `tr_tax_base` \*      | `decimal`                | Total tax in base currency                                 | No    | `3.00`          |
| `tr_shipping_base` \* | `decimal`                | Delivery cost in base currency                             | No    | `0.00`          |
| `tr_city`             | `string`, max length 255 | Delivery address, city                                     | No    | `London`        |
| `tr_state`            | `string`, max length 255 | Delivery address, state                                    | No    | `Washington`    |
| `tr_country`          | `string`, max length 255 | Delivery address, country                                  | No    | `France`        |
| `tr_currency`         | `string`, max length 3   | Currency                                                   | No    | `USD`           |
| `ti_orderid`          | `string`, max length 255 | Order ID                                                   | Yes   | `#134`          |
| `ti_sku`              | `string`, max length 255 | Product SKU                                                | Yes   | `pbz00123`      |
| `ti_name`             | `string`, max length 255 | Product name                                               | No    | `Cone pendulum` |
| `ti_category`         | `string`, max length 255 | Product category                                           | No    | `New Age`       |
| `ti_price`            | `decimal`                | Product unit price                                         | Yes   | `9.99`          |
| `ti_price_base` \*    | `decimal`                | Price in base currency                                     | No    | `9.99`          |
| `ti_quantity`         | `integer`                | Number of product in transaction                           | Yes   | `2`             |
| `ti_currency`         | `string`, max length 3   | Currency                                                   | No    | `EUR`           |
| `base_currency` \*    | `string`, max length 3   | Reporting currency                                         | No    | `GBP`           |

\* Set exclusively by the [Currency conversion enrichment](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/).

---

# Introduction to data destinations
> Different options for storing and forwarding Snowplow data
> Source: https://docs.snowplow.io/docs/fundamentals/destinations/

![](/assets/images/usecasearch-5b84782067868dc4be86b6012cf315ea.png)

Snowplow gives you multiple ways to get your data where it needs to go. You can load data into warehouses and lakes for deeper analysis, forward individual events to marketing platforms in real-time, or send aggregated segments to activate your insights.

The best approach depends on your use case, but most organizations benefit from combining approaches: storing data centrally in a warehouse while selectively forwarding specific events or segments to operational systems.

## Data warehouses and lakes

Snowplow loads your data into warehouses and lakes where you can combine it with other datasets for comprehensive analytics. You can choose from major data platforms including Redshift, BigQuery, Snowflake, Databricks, and Synapse Analytics, as well as cloud storage like S3, GCS, and ADLS through [our various loaders](/docs/destinations/warehouses-lakes/).

## Event forwarding

Send individual events to downstream platforms in real-time through [event forwarding](/docs/destinations/forwarding-events/). This approach works well for marketing and personalization platforms that can act on fresh event data, product analytics tools that require granular event-level data, and streaming platforms like Kafka, Kinesis, and Pub/Sub for internal consumption and real-time processing.

## Reverse ETL

Sync aggregated data and customer segments from your warehouse to operational systems through [reverse ETL](/docs/destinations/reverse-etl/). This approach lets you activate insights from your data by sending calculated metrics, audience segments, or enriched profiles to marketing and sales tools. Snowplow partners with Census to provide this functionality.

---

# Introduction to Snowplow entities
> Entities are a good way to deal with common fields across various events
> Source: https://docs.snowplow.io/docs/fundamentals/entities/

When you track an action or behavior, the information about the objects, users, and context in which the action occurred is just as important as the action itself. In Snowplow tracking, we use entities to capture this contextual data. They're reusable building blocks that make your tracking easier to implement, and your data easier to analyze.

> **Info:** What we now call "entity" or "entities" was previously called "context". You'll still find `context` or `contexts` used in many of the existing APIs, database column names, and documentation, especially to refer to a set of multiple entities.

For example, when tracking a "search" event, you might want to capture information about:

- The user who performed the search
- The web page where the search was made
- The products that were returned in the search results
- Any A/B test variations that were active on the page
- And so on

The event data might include fields such as the search term, number of results, and time taken to perform the search. These fields are specific to the "search" event.

The `user`, `webPage`, `A/B test variation`, and `products` data isn't specific to that event, and can be defined and tracked as entities:

![Diagram showing an example Snowplow event with atomic event data at the top, self-describing event data in the middle, and entities at the bottom including user, webPage, abTestID, and three product entities](/assets/images/example-event-entities-9ddd43c1ce9bd436594766b8067fdb6f.png)

The user who triggered the `search` event might go on to generate a `link_click` event. You might want to capture data on:

- The user who clicked the link
- The web page where the link was clicked

The same `user` and `webPage` entities can be reused for this event.

Every Snowplow event can have entities attached to it. They can be of the same or different types.

## How to track entities

Snowplow provides a number of entities out-of-the-box. You can configure your trackers to attach certain entities automatically to all events. For example, the web trackers add a [`webPage` entity](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) to all events by default.

Trackers add other out-of-the-box entities based on event type. For example, if you track a Snowplow media `play` event, the tracker will automatically add `media_player` and media `session` entities to it.

Check out the [out-of-the-box data](/docs/events/ootb-data/) section to find out more about the entities that come with Snowplow trackers.

Some [enrichments](/docs/pipeline/enrichments/available-enrichments/) add entities to events in stream. For example, the [cross-navigation enrichment](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/) adds a `cross_navigation` entity to events based on the cross-navigation querystring.

### Custom entities

Check out our [tracking plan guide](/docs/fundamentals/tracking-design-best-practice/) for best practice on designing entities.

Snowplow provides [tooling](/docs/event-studio/) to help you define your own custom entities. Read more about tracking custom data [here](/docs/events/custom-events/).

## Entities are defined by schemas

Like most Snowplow [events](/docs/fundamentals/events/#self-describing-events), entities are based on [self-describing JSON schemas](/docs/fundamentals/schemas/). As such, they can include arbitrarily complex data, and are versioned. You can evolve your entity definitions over time.

Each entity consists of two parts:

- `schema`: a reference to a [schema](/docs/fundamentals/schemas/) that describes the name, version and structure of the entity
- `data`: the entity data as a set of key-value properties in JSON format

Find out in the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page about how entities are structured in the data warehouse.

---

# Introduction to Snowplow events
> An event is a central concept in Snowplow that represents something that occurred at a particular point in time
> Source: https://docs.snowplow.io/docs/fundamentals/events/

Events are the primary way to capture data with Snowplow. Every time something happens that you want to track, you can send an event to your Snowplow pipeline. Each event is a JSON object that describes what happened. Events flow through your pipeline where they're validated, enriched, and loaded into your data warehouse or lake for analysis.

## Snowplow event types

All Snowplow events have the same underlying structure and [standard fields](/docs/fundamentals/canonical-event/). However, there are two types of events in Snowplow:

- Schema-less **baked-in** events
  - The baked-in event types are page views, page pings, and structured events

- **Self-describing** events that are based on a [JSON schema](/docs/fundamentals/schemas/)

  - All other events are self-describing events
  - This includes out-of-the-box events that come with Snowplow, and custom events that you define yourself

![Diagram showing the different types of Snowplow event](/assets/images/Snowplow-event-types-99d78b7b2932b01b0165b2990fb3085e.png)

Whether an event is baked-in or self-describing affects how you'll model the data.

All Snowplow events are by default tracked with schema-defined [entities](/docs/fundamentals/entities/) attached to them.

### Baked-in events

The following events are "baked in":

- [Page views](/docs/events/ootb-data/page-and-screen-view-events/)
- [Page pings](/docs/events/ootb-data/page-activity-tracking/)
- [Structured events](/docs/events/custom-events/#structured-events)

In the data warehouse, any event-specific information will be in standard columns in the Snowplow `events` table. You can find those listed [here](/docs/fundamentals/canonical-event/#baked-in-event-fields).

Find out more about how to track and model page view and page ping events [here](/docs/events/ootb-data/page-activity-tracking/).

> **Tip:** We recommend using self-describing events instead of structured events whenever possible. Structured event tracking is a legacy format used to track events that were not natively supported by Snowplow.

> **Info:** The legacy `transaction` and `transaction_item` events are also "baked-in". Several Snowplow trackers provide [ecommerce tracking](/docs/events/ootb-data/ecommerce-events/) APIs; we recommend using these instead, as they support current Snowplow best practise, and are compatible with the [Snowplow ecommerce dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/).

### Self-describing events

Self-describing events can include arbitrarily complex data, as defined by the event's [data structure](/docs/fundamentals/schemas/) or schema. We call them "self-describing" because these events include a reference to their underlying [JSON Schema](/docs/api-reference/json-schema-reference/) data structure.

Each self-describing event consists of two parts:

- `schema`: a reference to a [schema](/docs/fundamentals/schemas/) that describes the name, version and structure of the event
- `data`: the event data as a set of key-value properties in JSON format

Because the event references a specific version of its schema, it's always clear to downstream users and applications what each field in the event means, even if your definition of the event changes over time.

Snowplow provides a large number of self-describing events out-of-the-box, for example:

- [Link clicks](/docs/events/ootb-data/links-and-referrers/#link-clicks)
- [Form submissions](/docs/sources/web-trackers/tracking-events/form-tracking/)
- [Ecommerce transactions](/docs/events/ootb-data/ecommerce-events/)

Check out the full range of included events [here](/docs/events/ootb-data/).

You can also create [custom self-describing events](/docs/events/custom-events/) to match your business requirements. Snowplow provides [tooling](/docs/event-studio/) to help you define and track custom events.

Find out in the [warehouse tables fundamentals](/docs/fundamentals/warehouse-tables/) page about how self-describing events are structured in the data warehouse.

> **Info:** We originally called self-describing events "unstructured events", to distinguish them from structured events. This was misleading, because these events are actually more structured than structured events. The old term is deprecated, but you might still see it in some docs, APIs and database column names, such as `unstruct_event` or `ue`.

## How to track events

Track events in your applications using one of the [Snowplow tracking SDKs](/docs/sources/). Each SDK provides built-in APIs to track [different types of events](/docs/events/ootb-data/).

All the trackers also support custom tracking, so you can define the events that are relevant to your business.

You can also use [webhooks](/docs/sources/webhooks/) to track automated actions. The Snowplow [Collector endpoint](/docs/pipeline/collector/) accepts all [valid](/docs/fundamentals/canonical-event/) Snowplow event payloads, regardless of their source.

---

# Introduction to failed events
> Failed events represent data that did not pass validation or otherwise failed to be processed
> Source: https://docs.snowplow.io/docs/fundamentals/failed-events/

_Failed events_ is an umbrella term for events that the pipeline had some problem processing.

These problems can arise at various stages:

- Collection (e.g. invalid payload format)
- Validation (e.g. event does not match the schema)
- Enrichment (e.g. external API unavailable)
- Loading (very unlikely with modern versions of Snowplow)

> **Note:** Failed events are **not** written to your `atomic` events table, which only contains high quality data. See [below](#dealing-with-failed-events) for how to deal with them.

## Common failures

The two most common types of failed events are:

- **Validation failures**. These errors occur when an [event](/docs/fundamentals/events/) or an [entity](/docs/fundamentals/entities/) does not match its [schema](/docs/fundamentals/schemas/). The reason usually is incorrect tracking code. For example, if you have a schema with a property defined as an `enum` accepting values `a`, `b`, and `c`, but the incoming event which references that schema includes the property with a value of `d`, that will be a `ValidationError`. The error's description will include that information -- e.g. `$property does not match enum list`. Validation can also fail if the schema is not available, such as when you forgot to add it to the production schema registry before putting the tracking code live. The pipeline will first search for a schema in the associated [Iglu](/docs/fundamentals/schemas/#iglu-schema-repository) repository, and if it cannot find it, it will search in [Iglu Central](https://iglucentral.com), the public repository of self-describing schemas available to everyone. If the schema is not found in either location, the event will be labeled as a `ResolutionError`.

- **Enrichment failures**. These can be due to an [API enrichment](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/) reaching out to an external API that’s down. Another cause is a failure in the custom code in a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/).

> **Tip:** In many cases, you will be able to fix the underlying problem directly, e.g. by altering your tracking code, by providing the correct schema, or by changing your enrichment configuration.

## Other failures

Other failures generally fall into 3 categories:

- **Bots or malicious activity**. Bots, vulnerability scans, and so on, can send completely invalid events to the Collector. The format might be wrong, or the payload size might be extraordinarily large.

- **Pipeline misconfiguration**. For example, a loader could be reading from the wrong stream (with events in the wrong format). This is quite rare, unless you're self-hosting Snowplow, as all relevant pipeline configuration is automatic.

- **Temporary infrastructure issue**. This is again rare. One example would be Iglu Server (schema registry) not being available.

> **Tip:** All of these are internal failures you typically can’t address upstream.

## Dealing with failed events

Snowplow provides a dashboard and alerts for failed events. See [Monitoring failed events](/docs/monitoring/).

***

For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.

> **Note:** This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.

The events will include a special column with the details of the failure, and any invalid columns will be set to `null`. Otherwise, the format is [the same as for your atomic events](/docs/fundamentals/canonical-event/).

See [Exploring failed events](/docs/monitoring/exploring-failed-events/) for more details and setup instructions.

***

Finally, on AWS and GCP all failed events are backed up in object storage (S3 and GCS respectively). Sometimes, but not in all cases (e.g. not if the original events exceeded size limits), it’s possible to recover them by replaying them through the pipeline. This is a complicated process mainly reserved for internal failures and outages. Refer to [Recovering failed events](/docs/monitoring/recovering-failed-events/).

***

You can find a full list of failed event types [in the API reference section](/docs/api-reference/failed-events/).

---

# Snowplow fundamentals
> An overview of what a Snowplow pipeline does and what steps it includes
> Source: https://docs.snowplow.io/docs/fundamentals/

![Pipeline architecture diagram](/assets/images/architecture-16e4cd694721410bb8a1ff943a4c5255.png)

The diagram above illustrates a typical Snowplow pipeline with data flowing left to right.

- [**Trackers**](/docs/sources/) generate [event](/docs/fundamentals/events/) data and send this to your Collector. We have trackers covering web, mobile, desktop, server and IoT. Additionally, [**webhooks**](/docs/sources/webhooks/) allow third-party software to send their own internal event streams to your Collector for further processing.
- Events hit the **Collector** application and it saves the raw events to storage (S3 on AWS and Google Cloud Storage on GCP) and then sends them down the pipeline to Enrich.
- The **Enrich** application cleanses the data and validates each event against its [schema](/docs/fundamentals/schemas/) to ensure it meets the criteria you have designed and set. When an event fails to validate, it will feed into a bad data stream which contains all of your [**failed events**](/docs/fundamentals/failed-events/). This way, the Snowplow pipeline is non-lossy as failed events [can be reprocessed](/docs/monitoring/recovering-failed-events/).
- Once validated, each event is enriched by the [**Enrichments**](/docs/pipeline/enrichments/available-enrichments/) you have configured for your pipeline.
- The enriched events are sent to real-time streams (Kinesis on AWS, PubSub on GCP). At this point, data can be [**forwarded**](/docs/destinations/forwarding-events/) to other applications to power real-time use cases.
- [**Loaders**](/docs/destinations/warehouses-lakes/) then load your data off the real-time streams into the various destinations you have set up for your pipeline. Typically these include a data warehouse (Redshift, Snowflake, BigQuery, Databricks) or a data lake (S3, GCS, ADLS / OneLake). Note that you can also connect a data warehouse to a data lake (e.g. Synapse Analytics + ADLS) instead of loading to a warehouse directly.
- Your event-level data is now in your chosen destinations. From here we recommend this raw data is [**modelled**](/docs/modeling-your-data/) into smaller, cleaner tables that are easier to perform analysis on. At this stage it can also be joined with other data sets. We have out-of-the-box data models for web and mobile to help you get started.

## Why do we use this architecture?

Snowplow's distinctive architecture has been informed by a set of key design principles:

1. **Extreme scalability** - Snowplow should be able to scale to tracking billions of customer events without affecting the performance of your client (e.g. website) or making it difficult to subsequently analyze all of those events
2. **Permanent event history** - Snowplow events should be stored in a simple, non-relational, immutable data store. Your raw event data is never compacted, overwritten or otherwise corrupted by Snowplow
3. **Direct access to individual events** - you should have direct access to your raw Snowplow event data at the atomic level. Unlike with other solutions, the data is not intermediated by a third-party vendor, or a slow API, or an interface offering aggregates only
4. **Separation of concerns** - event tracking and event analysis should be two separate systems, only loosely-coupled. As a result, Snowplow can be used to feed whatever analytics process you want
5. **Support any analysis** - Snowplow should make it easy for business analysts, data scientists and engineers to answer any business question they want, using as wide a range of analytical tools as possible, including Machine Learning

---

# Introduction to structuring your data with schemas
> Schemas are a powerful feature that ensures your data is clean and descriptive
> Source: https://docs.snowplow.io/docs/fundamentals/schemas/

**Schemas** define the structure of the data that you collect. Each schema defines what fields are recorded with each [event](/docs/fundamentals/events/), and provides validation criteria for each field. Schemas are also used to describe the structure of [entities that are attached to events](/docs/fundamentals/entities/).

With schemas, you can:

- Build meaning into your data by clearly defining what each field represents.
- Define your own data structures to capture data in a way that works for your business. For example, a two-sided marketplace will be tracking very different events from a gaming application.
- Ensure your data presents a validated record of what has happened.

Schemas describe how you structure your data. When data is [processed through your Snowplow pipeline](/docs/fundamentals/), each event is validated against its schema and only valid events are allowed to pass through. [Failed events](/docs/fundamentals/failed-events/) are sent to a separate location.

By describing how the data should be structured as part of your schema definition, you ensure clean and consistent data landing in your data warehouse or other destinations.

As you evolve your website, mobile app, or server-side application, you can update your schemas to reflect the changes. Snowplow automatically evolves your table definition to accommodate both old and new data safely.

## Data structures are schemas plus metadata

We often use the terms "schema" and "data structure" interchangeably, but they're not exactly the same thing. A **schema** is the JSON Schema definition that describes the structure of your data.

A **data structure** is a higher-level data management concept that includes the schema along with additional metadata, including:

- Whether the schema is to be used for an event or an entity
- Whether it's available to your pipeline, or is still in draft
- Optional change notes associated with schema versions

[Data structures](/docs/event-studio/data-structures/) wrap schemas with this additional metadata to help you manage your data definitions. They allow you to group related schemas within [tracking plans](/docs/event-studio/tracking-plans/) (previously known as data products), or keep track of [which schemas are used where](/docs/event-studio/tracking-plans/event-specifications/).

> **Tip:** Check out the documentation for [managing](/docs/event-studio/data-structures/) and [versioning](/docs/event-studio/data-structures/versioning/) data structures.

## Self-describing JSON schema anatomy

> **Info:** Snowplow CDI customers can create custom schemas using the [data structures builder](/docs/event-studio/data-structures/), without worrying about how it works under the hood.

Snowplow schemas are based on the [JSON Schema](https://json-schema.org/) standard ([draft 4](https://datatracker.ietf.org/doc/html/draft-fge-json-schema-validation-00)). For a comprehensive guide to all Snowplow supported validation options, see the [Snowplow JSON Schema reference](/docs/api-reference/json-schema-reference/).

Snowplow specifically uses **self-describing JSON schemas**, meaning that they contain metadata about themselves within the schema definition.

Let's take a look at an example schema to talk about its constituent parts:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for an example event",
  "self": {
    "vendor": "com.snowplowanalytics",
    "name": "example_event",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "maxLength": 128
    },
    "job_role": {
      "description": "",
      "type": [
        "string",
        "null"
      ],
      "maxLength": 128
    },
    "promo_code": {
      "description": "",
      "type": [
        "string",
        "null"
      ],
      "minLength": 8,
      "maxLength": 20
    }
  },
  "additionalProperties": false,
  "required": [
    "name"
  ]
}
```

`$schema`: this argument instructs the Snowplow pipeline on how to handle this schema and in most circumstances should be left as shown in the example.

`description`: this argument is where you should put detailed information on the purpose of this schema. This will be particularly helpful for others who are trying to understand the meaning of particular data or when they want to know if a schema already exists for something they want to track.

`self`: this section of arguments contains metadata which makes the schema "self-describing".

- `vendor`: this usually refers to the company who has authored the schema. Most times this will be your company’s name. This could also be for organizing schemas from different groups in your organization if you have multiple teams working on different events and entities (e.g. `com.acme.android`, `com.acme.marketing`). Snowplow uses the reversed company internet domain for vendor names (e.g. `com.snowplowanalytics`).
- `name`: this is the name you want to give your schema. Much like the description above, this is a good chance to help others like data analysts who might be consuming this data know exactly what your schema is meant to capture.
- `format`: this field simply states the format of the schema which will always be `jsonschema`.
- `version`: Snowplow allows you to [increment versions of a schema](/docs/event-studio/data-structures/versioning/) as your tracking needs to evolve and this argument stores the current version.

After the self section, the remainder of the schema is where you will begin describing the event or entity fields that you will be collecting.

`type`: type should always be set as `object`.

`properties`: here is where you will describe the fields you intend on collecting. Each field is given a name and a number of arguments, these are important as they feed directly into the validation process.

- `description`: similar to the description field for the schema, this argument is where you should put detailed information on what this field represents to avoid any misunderstanding or misinterpretation during analysis.
- `type`: this denotes the type of data that is collected through this field. The most common types of data collected are `string`, `number`, `integer`, `object`, `array`, `boolean` and `null`. A single field can allow multiple types as shown in the field `job role` in the example schema which allows both `string` and `null`
- Validation arguments can then be passed into the field such as `minLength`, `maxLength` and `enum` for strings and `minimum` and `maximum` for integers.

This example doesn't show the optional `$supersedes` and `$supersededBy` fields. See [marking schemas as superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded).

## Iglu schema repository

> **Info:** Snowplow CDI customers don't need to create or manage their own Iglu repositories. A private Iglu repository is included in your pipeline.

**Iglu** is a set of tools for hosting and managing JSON schemas. We use it to store all the schemas associated with events and entities. It's called a schema repository or schema registry.

You'll notice that schema URIs use the `iglu:` protocol, for example: `iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0`. This is because all Snowplow schemas are stored in an Iglu repository.

There is a [central Iglu repository](https://iglucentral.com/) that holds public schemas for use with Snowplow, including ones for some of the [out-of-the-box self-describing events](/docs/fundamentals/events/#self-describing-events) and [out-of-the-box entities](/docs/fundamentals/entities/#how-to-track-entities).

For Snowplow Self-Hosted users, you'll need to run your own Iglu schema repository to host schemas for your custom [events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities). Use [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) (recommended), or a [static repository](/docs/api-reference/iglu/iglu-repositories/static-repo/).

## Self-describing JSON data in the event payload

Snowplow trackers can send self-describing JSON event and entity data as is, or as Base64-encoded strings. Base-64 strings are URL-safe and ensure that no data is lost or corrupted. The downside is that the data will be bigger and less readable.

Here's an example showing a self-describing JSON object for a product view event:

```json
{
  "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",
  "data": {
    "product_id": "ASO01043",
    "price": 49.95
  }
}
```

Before adding this data to the event payload, the tracker will wrap this event self-describing JSON in an outer self-describing JSON. The wrapper specifies that this is a self-describing event, using the historical `unstruct_event` naming:

```json
{
  // Tells Snowplow this is an self-describing event
  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
  "data": {

    // Tells Snowplow this is a viewed_product event
    "schema": "iglu:com.my_company/viewed_product/jsonschema/1-0-0",
    "data": {

      // The event data itself
      "product_id": "ASO01043",
      "price": 49.95
    }
  }
}
```

> **Info:** The event payload is itself a self-describing JSON object. See how this works in practise in these [example tracker HTTP requests](/docs/events/http-requests/).

The parameter name used for this wrapped structure depends on whether you've chosen to use Base-64 encoding or not:

| Payload property | Type                | Example values                                                 |
| ---------------- | ------------------- | -------------------------------------------------------------- |
| `ue_px`          | Base64-encoded JSON | `eyAicHJvZHVjdF9pZCI6ICJBU08wMTA0MyIsICJwcmljZSI6IDQ5Ljk1IH0=` |
| `ue_pr`          | JSON                | `{ "product_id": "ASO01043", "price": 49.95 }`                 |

Trackers process entities in a similar way. The differences are:

- They're wrapped in a `contexts` self-describing JSON rather than `unstruct_event`
- Payload parameter options are `co` for JSON or `cx` for Base64-encoded JSON

See the [schema translation reference](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) page to learn how schema data lands in your warehouse.

---

# Versioning, patching, and marking schemas as superseded
> Fix mistakes in existing schema versions by amending them, with guidelines on when amendments are appropriate and how to handle production schemas.
> Source: https://docs.snowplow.io/docs/fundamentals/schemas/versioning/

Every schema has a version, starting from `1-0-0`. As your schema evolves, you will need to create new versions of it. All previous versions of a schema remain available to ensure backwards-compatibility.

Versioning has an important role in telling Snowplow Loaders how to handle the changes when loading into your data warehouse. For example, there may be a need to create new columns, update columns, or even create new tables.

It's important to understand when your change is breaking, and [version correctly](/docs/event-studio/data-structures/versioning/).

There are two kinds of schema changes:

- **Non-breaking** - a non-breaking change is backward compatible with historical data and increments the `patch` number i.e. `1-0-0` -> `1-0-1`, or the middle digit i.e. `1-0-0` -> `1-1-0`.
- **Breaking** - a breaking change is not backwards compatible with historical data and increments the `model` number i.e. `1-0-0` -> `2-0-0`.

Different data warehouses handle schema evolution slightly differently. Use the table below as a guide for incrementing the schema version appropriately.

|                                              | Redshift     | Snowflake, BigQuery, Databricks |
| -------------------------------------------- | ------------ | ------------------------------- |
| **Add / remove / rename an optional field**  | Non-breaking | Non-breaking                    |
| **Add / remove / rename a required field**   | Breaking     | Breaking                        |
| **Change a field from optional to required** | Breaking     | Breaking                        |
| **Change a field from required to optional** | Breaking     | Non-breaking                    |
| **Change the type of an existing field**     | Breaking     | Breaking                        |
| **Change the size of an existing field**     | Non-breaking | Non-breaking                    |

> **Warning:** In Redshift and Databricks, changing _size_ may also mean _type_ change. For example, changing the `maximum` integer from `30000` to `100000`. See our documentation on [how schemas translate to database types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

Changing a schema without incrementing the version has several risks:

- Events that were previously valid could become invalid against the new changes
- Your warehouse Loader, which updates the table [according to the schema](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/#versioning), could get stuck if it’s not possible to cast the data in the existing table column to the new definition (e.g. if you change a field type from a string to a number)
- Similarly, data models or other applications consuming the data downstream might not be able to deal with the changes

Aim to treat each schema version as immutable. However, if you can't avoid making changes to an existing schema version, you have two options: patching the schema, or marking it as superseded.

## Patch a schema

Patching is available if your schema version isn't yet in production.

> **Danger:** For Snowplow CDI customers, patching is disabled for production pipelines.
>
> Patching in a production environment can break your loading, especially if your patch contains breaking changes.
>
> Also, never patch a schema version that exists in a production environment, even if you are doing the patching in a development environment. This will lead to problems later when you try to promote that schema to production.

If you are working on a new schema version in a development environment, patching is safer as it won't corrupt any production data.

Before:

```mermaid
flowchart LR
  style v102 fill:orange
  %% ~~~ is not working for some reason
  linkStyle default stroke:none,fill:none
  v100("1-0-0") --- v101("1-0-1") --- v102("1-0-2\n(incorrect)")
```

After:

```mermaid
flowchart LR
  style v102 fill:teal
  linkStyle default stroke:none,fill:none
  v100("1-0-0") --- v101("1-0-1") --- v102("1-0-2\n(corrected)")
```

Follow these [instructions](/docs/event-studio/data-structures/versioning/#patch-a-schema) to patch a schema.

## Mark a schema as superseded

If your events are failing in production because of an incorrect schema, you might not be able to instantly update the tracking code to use a new schema version. This is a common situation for mobile tracking, for example. You can resolve this by marking the old schema version as superseded by the new schema version.

> **Note:** You need to be on Enrich 3.8.0+ and Iglu Server 0.11.0+ to use this feature. Additionally, if you are using [Snowplow Mini](/docs/api-reference/snowplow-mini/) or [Snowplow Micro](/docs/testing/snowplow-micro/), you will need version 0.17.0+ or 1.7.1+ respectively.

Before:

```mermaid
flowchart RL
  style v102 fill:orange
  linkStyle default stroke:none,fill:none
  v102("1-0-2\n(incorrect)") --- v101("1-0-1") --- v100("1-0-0")
```

After:

```mermaid
flowchart RL
  style v102 fill:orange
  style v103 fill:teal
  v102("1-0-2\n(incorrect)") --- v101("1-0-1") --- v100("1-0-0")
  v103("1-0-3\n(corrected)") -->|supersedes| v102
  linkStyle 0,1 stroke:none,fill:none
```

Here's how this works, at a glance:

- Suppose schema `1-0-2` is wrong.
- Draft a new schema version correcting the issue.
- In the new schema, add the following field at the root: `"$supersedes": ["1-0-2"]`.
- [Set the version](/docs/event-studio/data-structures/versioning/) of the new schema as usual, i.e. `1-0-3` if there are no breaking changes or `2-0-0` if there are.
- [Add](/docs/event-studio/data-structures/) the new schema to your production environment.
- Events or entities that use schema `1-0-2` will now be automatically updated (in the Enrich application) to use version `1-0-3`, and will be validated against that version. (A special entity will be added to these events to record this fact.)

### Example

Let’s say we have a mobile application. We are sending certain events from this application, and these events contain entities with following schema:

**Geolocation 1-0-2**

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "description": "Schema for client geolocation contexts",
    "self": {
        "vendor": "com.acme",
        "name": "geolocation",
        "format": "jsonschema",
        "version": "1-0-2"
    },
    "type": "object",
    "properties": {
        "latitude": {
            "type": "number",
        },
        "longitude": {
            "type": "number",
        }
    },
    "additionalProperties": false
}
```

Later, we realize that when implementing tracking, we have mistakenly included an `altitude` field in the entity objects:

**Wrong tracking code (iOS)**

```swift
let event = ScreenView(name: "Screen")
event.entities.add(
    SelfDescribingJson(schema: "iglu:com.acme/geolocation/jsonschema/1-0-2",
        andDictionary: [
            "latitude": 38.7223,
            "longitude": 9.1393,
            "altitude": 20 // extra field not defined in the schema
        ])!)
tracker.track(event)
```

Since `additionalProperties` is set to `false`, all events with the `altitude` field end up as [failed events](/docs/fundamentals/failed-events/).

We can create a new schema with version `1-0-3` that contains the `altitude` field and then use this schema in the next version of the application. This would make the events valid. However, users will not update their application to the new version all at once. Events from the older version will continue to come, therefore there will still be failed events until all users start to use a newer version.

To solve this problem, we simply add the `$supersedes` definition to the new schema.

**Geolocation 1-0-3 with $supersedes**

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "$supersedes": ["1-0-2"],
    "description": "Schema for client geolocation contexts",
    "self": {
        "vendor": "com.acme",
        "name": "geolocation",
        "format": "jsonschema",
        "version": "1-0-3"
    },
    "type": "object",
    "properties": {
        "latitude": {
            "type": "number",
        },
        "longitude": {
            "type": "number",
        },
        "altitude": {
            "type": "number",
        }
    },
    "additionalProperties": false
}
```

Now, when we receive events from the mobile application that use schema `1-0-2`, these events will be updated to use schema `1-0-3` and will be validated against that schema. Therefore, these events will be valid.

To record this fact, an extra entity will be added to all such events:

```json
{
    "schema": "iglu:com.snowplowanalytics.iglu/validation_info/jsonschema/1-0-0",
    "data": {
        "originalSchema": "iglu:com.acme/geolocation/jsonschema/1-0-2",
        "validatedWith": "1-0-3"
    }
}
```

Finally, if we [browse](/docs/event-studio/data-structures/) schema version `1-0-2`, we will see that Iglu Server automatically keeps track of which schema supersedes which. Specifically, it will now contain a `$supersededBy` definition:

**Geolocation 1-0-2 with $supersededBy**

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "$supersededBy": "1-0-3",
    "description": "Schema for client geolocation contexts",
    "self": {
        "vendor": "com.acme",
        "name": "geolocation",
        "format": "jsonschema",
        "version": "1-0-2"
    },
    "type": "object",
    "properties": {
        "latitude": {
            "type": "number",
        },
        "longitude": {
            "type": "number",
        }
    },
    "additionalProperties": false
}
```

### Usage

The `$supersedes` field states that the schema version defined in the `self` part supersedes the schema versions listed in the `$supersedes` field (one or more). Its value must be an array of strings (even if it only includes one item). For example:

```json
...
"$supersedes": ["1-0-2", "1-0-3"],
...
```

> **Note:** Once you’ve defined the `$supersedes` field for a schema version, you can’t update it — even in the development environment where patching is allowed. However, you can change which schema version supersedes which by creating new schema versions.
>
> For example, if version `1-0-2` is defined to supersede version `1-0-1`, and you create version `1-0-3` which also supersedes `1-0-1`, then `1-0-1` will be superseded by the newest version, i.e. `1-0-3`. See diagrams below for more information on how this is determined.

> **Tip:** Note that various pipeline components, most importantly Enrich (including Enrich embedded in Snowplow Mini and Snowplow Micro), cache schemas to improve performance. The default caching time is 10 minutes (it’s controlled by the [Iglu Resolver configuration](/docs/api-reference/iglu/iglu-resolver/)). This means that the effect of superseding a schema will not be immediate.

### Rules

#### A schema version can only supersede previous versions

For example, `1-0-2` can supersede `1-0-1`, but _can’t_ supersede `1-0-3`, `1-1-0`, or `2-0-0`. Iglu Server will reject a schema with a definition that breaks this rule.

|                                    ✅ OK                                   |                                 ❌ Invalid                                 |
| :-----------------------------------------------------------------------: | :-----------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v102("1-0-2") -->|supersedes| v101("1-0-1")
``` | ```mermaid
flowchart LR
  v102("1-0-2") -->|supersedes| v200("2-0-0")
``` |

#### A schema version can supersede multiple previous versions at once

Events referencing either of those previous versions will be treated as explained above.

|                                                                                ✅ OK                                                                                |
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v102("1-0-2") --- v101("1-0-1")
  v103("1-0-3") -->|supersedes| v101
  v103 -->|supersedes| v102
  linkStyle 0 stroke:none,fill:none
``` |

#### At any given moment, a schema version can only be superseded by a single schema version

Iglu Server automatically upholds this rule.

For example, if you specify that `1-0-3` supersedes `1-0-2` and (later) that `1-0-4` also supersedes `1-0-2`, the latest schema — `1-0-4` — will automatically become the one that supersedes `1-0-2`.

|                                                    Specified                                                    |                                                                      Becomes                                                                      |
| :-------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v103("1-0-3") -->|supersedes| v102("1-0-2")
  v104("1-0-4") --->|supersedes| v102
``` | ```mermaid
flowchart RL
  v104("1-0-4") --- v103("1-0-3") --- v102("1-0-2")
  v104 -->|supersedes| v102
  linkStyle 0,1 stroke:none,fill:none
``` |

The same happens if you specify “chains”, e.g. `1-0-3` supersedes `1-0-2` and `1-0-4` supersedes `1-0-3`. This will be automatically updated so that `1-0-4` supersedes `1-0-2` and `1-0-3`.

|                                                    Specified                                                   |                                                                               Becomes                                                                              |
| :------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| ```mermaid
flowchart RL
  v103("1-0-3") -->|supersedes| v102("1-0-2")
  v104("1-0-4") -->|supersedes| v103
``` | ```mermaid
flowchart RL
  v103("1-0-3") --- v102("1-0-2")
  v104("1-0-4") -->|supersedes| v102
  v104 -->|supersedes| v103
  linkStyle 0 stroke:none,fill:none
``` |

---

# Introduction to tracking design
> Learn how to design effective behavioral data tracking by analyzing use cases, defining entities and events, and creating comprehensive tracking plans.
> Source: https://docs.snowplow.io/docs/fundamentals/tracking-design-best-practice/

Good tracking design is essential for successful data collection. This guide will help you approach the design process systematically.

To use Snowplow successfully, you need to have a good idea of:

- What events you care about in your business
- What events occur in your website, mobile application, or server-side systems
- What decisions you make based on those events
- What you need to know about those events to make those decisions

The final outcome of your planning will be a set of tracking plans.

## Snowplow tracking concepts

![Tracking plan overview showing the relationship between tracking plans, event specifications, data structures](/assets/images/tracking-plan-overview-9f48fdeef0c5d4c593b416f9313c174c.png)

This diagram shows how tracking plans are conceptualized in Snowplow.

- **Tracking plans** are containers for related event specifications.
- **Event specification** represents a single business event. It contains all the relevant information about the event, including its purpose, origin, and associated data structures. Each has a single event data structure to define the event's properties, and can have multiple associated entity data structures.
- **Event and entity data structures** define the structure of the captured data to allow in JSON schemas for consistent data collection and analysis.
-

> **Info:** Snowplow customers can create [tracking plans](/docs/event-studio/tracking-plans/) and [event specifications](/docs/event-studio/tracking-plans/event-specifications/) directly in [Snowplow Console](https://console.snowplowanalytics.com).

For example, an `Ecommerce Checkout Flow` tracking plan may contain three event specifications:

- `Checkout Started` with a `checkout_started` event data structure and associated `cart` entity
- `Product Add To Cart` with an `add_to_cart` event data structure and associated `cart` and `product` entities
- `Order Completed` with an `order_completed` event data structure and associated `cart`, `product`, `order` and `payment` entities

Note that the `cart` entity data structure is reused across event specifications, this promotes consistency in your tracking design and makes analysis easier.

Additionally you can assign entities to [source applications](/docs/event-studio/source-applications/) to document which entities are expected for each event within that application, these are also called [global entities](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/). An example would be assigning a `user` entity to a mobile application to ensure that user information is always captured with events from that application.

## Naming conventions for tracking plans

A common standard for naming and structuring events, entities, and properties makes it easier to understand and extend your tracking design.

Here are our recommendations, but it's more important to be consistent across your tracking plans than to follow these guidelines:

For tracking plans:

- Use a descriptive name in title case that reflects the business domain, application, or use-case, e.g., `Ecommerce Checkout Flow`, `Mobile App User Engagement`, `SaaS Application Usage`

For Event Specifications:

- Use a verb-noun format in title case that clearly describes the action, e.g., `Add To Cart`, `User Signup`, `Purchase Completed`
- Be consistent with tense (e.g., all present or all past tense)
- Ensure each event specification represents a single event that has a clear purpose

For Event and Entity Data Structures:

- Use snake\_case for names, e.g., `add_to_cart`, `user_signup`, `purchase_completed`
- Use clear, descriptive names that reflect the purpose of the data structure

For Properties:

- Use snake\_case for property names, e.g., `user_id`, `product_name`,
- Be specific and descriptive to avoid ambiguity, e.g., use `purchase_amount` instead of just `amount`
- Do not repeat information contained in the data structure name, e.g., avoid `order_id` in an `order` entity

## Tracking plan best practices

Tracking plans are logical groupings of related business events with defined ownership. They help organize your tracking design and make it easier to manage and analyze your data.

When defining tracking plans, consider who will own the data and what business domain or use-case the events relate to. If you have multiple teams or departments, it can be helpful to align tracking plans with those organizational structures. Additionally, those teams can reflect implementation ownership or analysis ownership.

Examples of tracking plans include:

- `Ecommerce Checkout Flow`: contains events related to the checkout process in an e-commerce application
- `Mobile App User Engagement`: contains events related to user interactions within a mobile application
- `SaaS Application Usage`: contains events related to user actions within a SaaS platform

Bad examples of tracking plans would be overly broad or vague groupings, such as `All User Events` or `Miscellaneous Events`, which do not provide clear context or ownership. Another example would be overly specific groupings that limit reusability, such as `Product Page Views for Campaign X`.

When defining tracking plans, consider the following best practices:

- **Clear purpose**: each tracking plan should have a well-defined purpose and scope
- **Ownership**: assign clear ownership to each tracking plan to ensure accountability for data quality and governance
- **Logical grouping**: group related events that share a common business domain or use-case. Consider a group that reflects how the data will be used in analysis.
- **Reusability**: design tracking plans to promote reusability of event and entity data structures across different tracking plans

## Event Specification best practices

Event Specifications represent the key business events you are tracking. They contain a name, description, trigger conditions, and associated data structures. Event Specifications are designed to represent a single event to be implemented and analyzed. Each Event Specification should have one primary purpose.

For example, an `Add To Cart` Event Specification would represent the action of a user adding a product to their shopping cart. It would include:

- A clear name: `Add To Cart`
- A description of the event's purpose: "tracks when a user adds a product to their shopping cart"
- Trigger conditions: "fired when the user clicks the 'Add to Cart' button on a product page"
- Associated data structures: an `add_to_cart` event data structure and associated `product` and `cart` entity data structures

A bad example would be an `Ecommerce Action` Event Specification that tries to capture multiple actions like adding to cart, starting checkout, and completing a purchase in a single event. This can lead to confusion in implementation and complexity in analysis.

## Entity design best practices

**It is recommended that you adopt an "entity-first" approach to design**. This means starting by defining the key entities in your business domain before defining the events that interact with those entities. This approach helps ensure consistency and reusability across your tracking design.

If a piece of information is likely to be relevant to multiple different events, it belongs in an entity, not as a property of a single event. In fact, we often recommend not including any properties directly on event data structures, and instead placing all information in entities unless truly necessary.

Examples of common entities include:

- `user`: information about the user performing the action
- `product`: details about a product being viewed or purchased
- `cart`: information about a shopping cart
- `order`: details about an order being placed

## Event data structure granularity

A common challenge in defining event schemas is the choice of their granularity. We see customers struggle with this decision frequently in their tracking design.

### Approach 1: Group multiple actions into a single event schema

In some cases, it may be beneficial to group related actions into a single event schema. This means that a single event schema captures multiple types of actions, often distinguished by a property within the schema.

For example, you might define a single `ecommerce_action` event schema that includes a `type` property to distinguish between `view_product`, `add_to_cart`, `checkout_started`, and `purchase_completed` actions. Another example could be a `user_interaction` event schema that captures various user actions like `click`, `scroll`, and `form_submit`, with a `interaction_type` property to differentiate them.

This approach can be useful when:

- **Analysis**: the actions are closely related and often analyzed together
- **Simplicity**: you want to reduce the number of event schemas and columns in your data warehouse

Continuing the example from above, it is important to ensure the correct `type` property is set for each action and the allowed values are enforced through strong governance principles. This can be managed in Snowplow through Event Specifications with [property instructions](/docs/event-studio/tracking-plans/event-specifications/#properties). Tools like [Snowtype](/docs/event-studio/implement-tracking/snowtype/) can also help simplify this complexity during implementation.

### Approach 2: One event schema per action

Defining a separate event schema for each action is often the most straightforward approach. This means that each event schema corresponds to a single user action or system event.

For example, in an ecommerce application, you might have separate event schemas for:

- `view_product`
- `add_to_cart`
- `checkout_started`
- `purchase_completed`

This approach has several advantages:

- **Clarity**: each event schema has a clear purpose, making it easier to understand
- **Flexibility**: you can evolve each event schema independently, without affecting others

However, this approach may lead to a large number of event schemas if your application has many distinct actions and can make analysis more complex.

---

# Introduction to tracking plans
> Snowplow's tracking plans enable organizations to easily generate AI and BI-ready data that is reliable, clear, compliant, accurate, and predictable
> Source: https://docs.snowplow.io/docs/fundamentals/tracking-plans/

Tracking plans are Snowplow's solution to helping organizations more easily create and democratise behavioral data. By creating well-documented datasets, that are both human and machine readable, you can more easily collaborate around data and unlock self-service analytics at scale.

With tracking plans, you can:

- Set clear ownership for the data being created
- Make tracking implementation easier
- Deliver better governance around your data
- More easily communicate what the data means and how to use it
- Collaborate more effectively with the various teams involved in delivering value from your data
- Drive a self-serve culture around data across your organization

## What is a Tracking Plan

A tracking plan is a documented dataset. By documenting what data you are collecting, where, the meaning of that data and how to use it, you can break down the barriers that often exist between the many teams involved in the data value chain, i.e. from those implementing the tracking to those analysing the data.

At its core, a tracking plan has:

- An explicit owner; that is responsible for maintaining and updating the data over time
- Consumers; who use the data to deliver use cases and are impacted by upstream changes

Tracking plans at Snowplow are underpinned by the concept of a data contract. They act as a formal agreement between the producers of tracking plans and the consumers of tracking plans, and support better collaboration around the data being created.

Examples of tracking plans:

- Ecommerce Web
- Media Web

## Key elements of a Tracking Plan

**The Source Application/s it is part of**; a tracking plan is referencing the [Source Application/s](/docs/event-studio/source-applications/) that is spanning across.

![Example Tracking Plan view](/assets/images/example_tracking_plan_view-949245f8b596b73d8d03038828a465fd.png)

**Benefits:**

- Have a clear view in which application the tracking plan is implemented in, which domains it spans and the related application context information it will have available by default in the dataset.

**An owner**; tracking plans are typically split by domain with each tracking plan having an explicit owner that is responsible for the maintenance and evolution of that data.

**Benefits:**

- Know who to go to when new data is required for a particular domain or when there is an issue with the data, breaking down the barriers between data producers and data consumers

**Event specifications**; these describe each event that is collected as part of the tracking plan, on which applications they are triggered and where, the event data structure to validate against, and the entities to attach to each event (e.g. user, product etc).

![Example Event Specification](/assets/images/example_event_specification-e0b48ec28bbc53327add421f607ff25f.png)

**Benefits:**

- Provide implementation details to developers implementing tracking (see section on [Snowtype](/docs/event-studio/implement-tracking/snowtype/) for further details)
- Provide documentation around the semantics of the data that you are creating, to enable analysts, data scientists, analytics engineers with data discoverability

**Subscribers;** allow colleagues to express an interest in understanding changes that are made to the data within a Tracking Plan, usually because the data is being used in a downstream data model or Data Model Pack.

**Benefits:**

Break down the barriers that exist between data producers and data consumers, by providing:

- data producers, with visibility of how the data is being used downstream and the implications of any changes
- data consumers, with notifications when changes are made to a Tracking Plan so that they can make any necessary changes to their models, reports etc.

**Change history**; a complete audit log of the changes that have been made to the tracking plan, including changes to existing data and new data that has been added over time.

**Benefits:**

- Enhances accountability and transparency by providing a clear audit trail of all data modifications, fostering confidence in data integrity

**Volume metrics**; tracking plans can detect events ingested in your pipeline that match the configured event specifications. This allows your team to monitor occurrence-related metrics for events being tracked with specific [event specification IDs](/docs/event-studio/implement-tracking/snowtype/commands/#snowtype-patch).

**Benefits:**

You will be able to view several items in the UI that help detect anomalies or potential misconfigurations in trackers that are either not sending the expected events or are using incorrect [application IDs](/docs/event-studio/source-applications/#application-ids). This is particularly useful during the development phase when implementing tracking for a specific application using [Snowtype](/docs/event-studio/implement-tracking/snowtype/). These elements include:

- A counter for each event specification, showing the total number of events detected from the tracked application IDs in the last 30 days.

- A 'last seen' field for each event specification, indicating when the last event matching the event specification ID was detected.

- A list of application IDs from which events are being tracked, displayed for each event specification. For each application ID, a status will be shown with different colors:

  - **Green**: Event specifications are being tracked and identified with the specific application ID inherited from the configured [source applications](/docs/event-studio/source-applications/).
  - **Gray**: No event specifications are being tracked for an application ID inherited from the configured source applications.
  - **Yellow**: Event specifications are being tracked for an application ID that has not been configured or inherited from the source applications.

> **Note:** Some tracking plans, such as [**Base Web**](/docs/event-studio/tracking-plans/templates/#base-web) and [**Base Mobile**](/docs/event-studio/tracking-plans/templates/#base-mobile), contain standard events (e.g., _page pings_, _link clicks_, _screen view_, _application install_).
>
> For these tracking plans, the volume metrics will behave differently:
>
> - If no standard events are being tracked with an application ID different from those inherited from the source applications set up in the tracking plan, the behavior will be the same as for a normal tracking plan.
>
> - If standard events are being tracked with application IDs different from those inherited from the source applications set up in the tracking plan, a toggle will appear above the event specification list.
>
>   This toggle will be disabled by default, so the metrics displayed will relate only to the application IDs inherited from the source applications set up in the tracking plan.
>
>   ![](/assets/images/tracking_plan_metrics_default-4ce71a9e5107ab9668dada2032937b6e.png)
>
>   If the toggle is enabled, it will show the metrics for all the application IDs found for the standard events (not just the ones inherited from the tracking plan).
>
>   ![](/assets/images/tracking_plan_metrics_toggled-5f17d4d14eaf50490650ceecf5e47791.png)

## How tracking plans help with governance, data quality and data discoverability

The **data structures** that you attach to your event specification describe the [structure of the data](/docs/fundamentals/schemas/). They validate that the values of the properties contained within your events and entities are **valid** as they pass through your pipeline.

The **event specification** describes a specific implementation of an event. It is a narrower definition of your event than a data structure - not only do they describe the structure of the data (by attaching the relevant data structure to validate against), they also allow you to define the **right values** for fields when the event gets triggered and the entities that need to be attached.

_NB: the event specifications are not enforced by your pipeline (for example, we don’t yet validate that the correct entities are attached to an event)._

By adding screenshots, and descriptions to the event specification, you are also able to communicate the **semantics of the data** to those that want to analyse it. In this way you can ensure that the data is **represented accurately** when being used to derive insights and make decisions by the many teams using the data downstream.

A data structure can be used across event specifications, and across tracking plans. In doing so, you can ensure you consistently track business critical events and entities (for example, your "product" entity) across your organisation. Having the ability to use centralised event and entity schemas in this way, means that you are able to better govern the structure of the data across an organisation whilst also empowering teams to manage their own specific implementation of events via tracking plans.

To understand how to get started with tracking plans, see [Defining the Data to collect with Tracking Plans](/docs/event-studio/tracking-plans/) for further details.

---

# Introduction to the atomic events table
> A summary of the Snowplow events table and its fields, including custom events and entities
> Source: https://docs.snowplow.io/docs/fundamentals/warehouse-tables/

All Snowplow events have the same underlying structure and standard fields. All of these fields can be found in the `atomic.events` table, which is a "wide" (many columns) table.

Each line in the atomic events table represents a single event, be that a `page_view`, `add_to_basket`, `play_video`, etc.

Individual fields are stored in their own columns. Check out the [event properties reference](/docs/fundamentals/canonical-event/) for a full list of standard fields. [Self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) are stored as additional columns in the `atomic.events` table (except in Redshift, see below).

> **Tip:** The Snowplow data table is designed to be immutable: the data in each line should not change over time.
>
> Data points that you would expect to change over time e.g. what cohort a particular user belongs to, or how you classify a particular visitor, can be derived from Snowplow data. Our recommendation is that you define and calculate these derived fields at analysis time, store them in a separate table, and join to the `atomic.events` table when performing any analysis.

Learn about querying Snowplow data [here](/docs/destinations/warehouses-lakes/querying-data/).

## Redshift table structure

In Redshift, unlike other warehouses, [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/) are stored in their own dedicated tables.

These additional tables can be joined back to the core `atomic.events` table, by joining on the `root_id` field in the self-describing event or entity table with the `event_id` in the `atomic.events` table, and the `root_tstamp` and `collector_tstamp` field in the respective tables.

You can still query the data as if it were in a single wide table. This is because:

- The joins from the additional tables to the core `atomic.events` table are one-to-one
- The field joined on is the distribution key for both tables, so queries are as fast as if the data were in a single table

## Schemas in the warehouse

To understand how self-describing events and entities are translated into warehouse columns, check out this [reference page](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

---

# Get started with Console
> Use the Getting Started dashboard in Console to design tracking, implement it, deploy data models, and configure Signals.
> Source: https://docs.snowplow.io/docs/get-started/console/

The Getting Started dashboard in Console guides you through collecting and acting on your behavioral data. It organizes workflows into three sections: **Fundamentals**, **Analytics**, and **Real-time use cases**.

![The Getting Started dashboard in Console showing Fundamentals, Analytics, and Real-time use cases sections with guided workflow cards](/assets/images/console-getting-started-055d1e21167901b1ef5a52e76ab12cf8.png)

## How the dashboard is organized

Each section contains workflow cards for a specific task, from designing a tracking plan to configuring Signals. The workflows follow a prerequisite chain: create a tracking plan, implement tracking, then unlock downstream features like data models and Signals.

> **Note:** The Getting Started dashboard is available to Snowplow CDI customers with Console access.

## Design and implement tracking

The Fundamentals section covers the core data collection workflow.

### Create a tracking plan

The first card guides you through creating a [tracking plan](/docs/event-studio/tracking-plans/), which defines the [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) you want to collect.

1. Choose between built-in events or custom tracking
2. Create or select a [source application](/docs/event-studio/source-applications/)
3. Select tracking plan templates (Base Web, E-commerce Web, Media Web, and others)
4. Review and confirm your selections

Console then creates the tracking plan and associated data structures in your account.

### Implement tracking

With a tracking plan in place, the next card helps you [implement tracking](/docs/event-studio/implement-tracking/).

1. Select your source application
2. Configure tracker initialization with the generated code snippet
3. Select your tracking plan and view event-specific code snippets
4. Test with Snowplow Micro
5. Deploy to production

> **Tip:** Test your implementation with Snowplow Micro before deploying to production to verify that events are structured correctly and match your tracking plan.

## Deploy a data model

Once data is flowing, you can deploy a data model to transform raw events into structured tables for analysis. Console offers two types of data models:

- [Out-of-the-box data models](/docs/modeling-your-data/running-data-models-via-console/) for common use cases like web and mobile analytics
- [Automatically generated data models](/docs/modeling-your-data/automatically-generated-data-models/) tailored to your tracking plan

## Configure Signals for real-time use cases

[Signals](/docs/signals/get-started/) processes your event stream to compute user attributes in real time and trigger automated actions.

> **Note:** Signals is a paid add-on. Contact your account team to enable it.

### Design Signals attributes

Define [Signals attributes](/docs/signals/attributes/) that compute real-time properties about your users (for example, session count, cart value, or engagement score).

1. Choose a template or create a custom attribute
2. Create or edit an attribute group
3. Test and simulate against real data

### Retrieve Signals attributes

Create a [service](/docs/signals/attributes/services/) and follow the [retrieval steps](/docs/signals/connection/) to access attribute values from your front-end or back-end.

### Trigger interventions

Create [interventions](/docs/signals/interventions/) that automatically respond to user behavior — for example, displaying a promotion when a cart value exceeds a threshold or suppressing a campaign for users who have already converted.

---

# Creating your first custom events
> How to define your own events and entities
> Source: https://docs.snowplow.io/docs/get-started/custom-events/

While Snowplow provides various [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/) out of the box, you can define your own to match your business more closely.

The documentation on [self-describing events](/docs/fundamentals/events/#self-describing-events) provides a high-level view of how you might create and track one. For more guidance, check out [this tutorial](/tutorials/data-products-base-tracking/start/).

---

# Inspecting your first failed events
> How validation and failed events work
> Source: https://docs.snowplow.io/docs/get-started/failed-events/

If you have defined some custom events in the previous section, we recommend playing around with sending invalid data to see what happens.

For example, if your custom event has a numeric field, try sending a string. It will not reach your warehouse! Instead, a [failed event](/docs/fundamentals/failed-events/) will be generated.

> **Tip:** If you have not defined any custom events yet, you can use the following command-line snippet to send yourself an invalid event (it uses a nonexistent schema `com.my_company/product_view/jsonschema/1-0-0`):
>
> ```bash
> curl 'https://{{COLLECTOR_URL}}/com.snowplowanalytics.snowplow/tp2' \
>    -H 'Content-Type: application/json; charset=UTF-8' \
>    -H 'Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c' \
>    --data-raw '{"schema":"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4","data":[{"e":"ue","ue_px":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJzY2hlbWEiOiJpZ2x1OmNvbS5teV9jb21wYW55L3Byb2R1Y3Rfdmlldy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJpZCI6IjVOMFctUEwwVyIsImN1cnJlbnRfcHJpY2UiOjQ0Ljk5LCJkZXNjcmlwdGlvbiI6IlB1cnBsZSBTbm93cGxvdyBIb29kaWUifX19","tv":"js-2.17.2","tna":"spExample","aid":"docs-example","p":"web","tz":"Europe/London","lang":"en-GB","cs":"UTF-8","res":"3440x1440","cd":"24","cookie":"1","eid":"542a79d3-a3b8-421c-99d6-543ff140a56a","dtm":"1626182778193","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiI0YTU2ZjQyNy05MTk2LTQyZDEtOWE0YS03ZjRlNzk2OTM3ZmEifX1dfQ","vp":"863x1299","ds":"848x5315","vid":"3","sid":"87c18fc8-2055-4ec4-8ad6-fff64081c2f3","duid":"5f06dbb0-a893-472b-b61a-7844032ab3d6","refr":"https://docs.snowplow.io/","url":"/docs/","stm":"1626182778194"}]}'
> ```

Follow the [documentation](/docs/monitoring/exploring-failed-events/) to explore your failed events.

---

# Product features in Snowplow CDI and Self-Hosted
> Detailed comparison of features available in Snowplow CDI and Self-Hosted deployments.
> Source: https://docs.snowplow.io/docs/get-started/feature-comparison/

Here is a detailed list of product features, showing which are available as part of Snowplow [Customer Data Infrastructure](/docs/get-started/#customer-data-infrastructure) (CDI) or [Snowplow Self-Hosted](/docs/get-started/#self-hosted).

Check out the [Snowplow Product Directory](https://snowplow.io/snowplow-product-description) to learn more about which features are available for CDI Cloud or CDI Private Managed Cloud deployments.

| ### Data Pipeline                                                                                                 | CDI | Self-Hosted      |
| ----------------------------------------------------------------------------------------------------------------- | --- | ---------------- |
| [25+ trackers and webhooks](/docs/sources/)                                                                       | ✅   | ✅                |
| First party tracking                                                                                              | ✅   | ✅                |
| Anonymous data collection                                                                                         | ✅   | ✅                |
| [Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/)       | ✅   | ✅                |
| High availability and auto-scaling                                                                                | ✅   | ❌                |
| [Enrichments](/docs/pipeline/enrichments/available-enrichments/)                                                  | ✅   | ✅                |
| [Failed events](/docs/fundamentals/failed-events/)                                                                | ✅   | ✅                |
| [Data quality monitoring](/docs/monitoring/)                                                                      | ✅   | ❌                |
| Single Sign-On                                                                                                    | ✅   | ❌                |
| Pipeline observability                                                                                            | ✅   | do-it-yourself   |
| Surge protection                                                                                                  | ✅   | do-it-yourself   |
| **Warehouse / lake destinations**                                                                                 |     |                  |
| • Snowflake                                                                                                       | ✅   | ✅                |
| • Redshift                                                                                                        | ✅   | ✅                |
| • BigQuery                                                                                                        | ✅   | ✅                |
| • Databricks                                                                                                      | ✅   | ✅                |
| • Synapse Analytics                                                                                               | ✅   | ✅                |
| • Elasticsearch                                                                                                   | ✅   | ✅                |
| • S3                                                                                                              | ✅   | ✅                |
| • Google Cloud Storage                                                                                            | ✅   | ✅                |
| • Azure Data Lake Storage / OneLake                                                                               | ✅   | ✅                |
| **Real-time streams**                                                                                             |     |                  |
| • Kinesis                                                                                                         | ✅   | ✅                |
| • PubSub                                                                                                          | ✅   | ✅                |
| • Kafka / Azure Event Hubs / Confluent Cloud                                                                      | ✅   | ✅                |
| ### Event Studio                                                                                                  | CDI | Self-Hosted      |
| Advanced enrichments (PII, IP anonymization, JS, API, SQL enrichments)                                            | ✅   | ✅ (no UI or API) |
| [Data structures tooling and API](/docs/event-studio/data-structures/)                                            | ✅   | ❌                |
| [Tracking plans](/docs/event-studio/tracking-plans/)                                                              | ✅   | ❌                |
| [Data modeling management tooling](/docs/modeling-your-data/running-data-models-via-console/)                     | ✅   | ❌                |
| Jobs monitoring dashboard                                                                                         | ✅   | ❌                |
| Failed events alerting                                                                                            | ✅   | ❌                |
| Failed events in the warehouse                                                                                    | ✅   | ✅                |
| QA pipeline                                                                                                       | ✅   | do-it-yourself   |
| Fine-grained user permissions using access control lists                                                          | ✅   | ❌                |
| API key access                                                                                                    | ✅   | ❌                |
| ### [Data Model Packs](/docs/modeling-your-data/visualization/)                                                   | CDI | Self-Hosted      |
| **Digital Analytics**                                                                                             |     |                  |
| Funnel builder                                                                                                    | ✅   | ❌                |
| User and Marketing Analytics                                                                                      | ✅   | ❌                |
| Marketing Attribution                                                                                             | ✅   | ❌                |
| Video and Media                                                                                                   | ✅   | ❌                |
| [Unified Digital, Attribution, Media Player, Ecommerce, Normalize, Utils data models](/docs/modeling-your-data/)  | ✅   | ❌                |
| **Ecommerce Analytics**                                                                                           |     |                  |
| Ecommerce Analytics                                                                                               | ✅   | ❌                |
| [Ecommerce data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) | ✅   | ❌                |
| ### Signals                                                                                                       | CDI | Self-Hosted      |
| [Real-time personalization engine](/docs/signals/introduction/)                                                   | ✅   | ❌                |
| Profiles Store                                                                                                    | ✅   | ❌                |
| Interventions                                                                                                     | ✅   | ❌                |
| ### Extensions                                                                                                    | CDI | Self-Hosted      |
| Reverse ETL, powered by Census                                                                                    | ✅   | ❌                |
| Audience Hub, powered by Census                                                                                   | ✅   | ❌                |
| ### Performance and Resilience                                                                                    | CDI | Self-Hosted      |
| Outage Protection                                                                                                 | ✅   | ❌                |
| Global Availability                                                                                               | ✅   | ❌                |
| ### Infrastructure and Security                                                                                   | CDI | Self-Hosted      |
| **High**                                                                                                          |     |                  |
| HTTP access controls                                                                                              | ✅   | ❌                |
| VPC peering                                                                                                       | ✅   | ❌                |
| SSH access control                                                                                                | ✅   | ❌                |
| CVE reporting                                                                                                     | ✅   | ❌                |
| Static collector IPs                                                                                              | ✅   | ❌                |
| **Advanced**                                                                                                      |     |                  |
| Custom VPC integration                                                                                            | ✅   | ❌                |
| Custom IAM policy                                                                                                 | ✅   | ❌                |
| Custom security agents                                                                                            | ✅   | ❌                |
| ### SLAs                                                                                                          | CDI | Self-Hosted      |
| Collector uptime SLA                                                                                              | ✅   | ❌                |
| Warehouse loading latency SLA                                                                                     | ✅   | ❌                |

---

# First steps with Snowplow
> Compare Snowplow deployment options including CDI Private Managed Cloud, CDI Cloud, Community Edition, and Self-Hosted to choose the right platform for your organization.
> Source: https://docs.snowplow.io/docs/get-started/

Snowplow is the customer context layer that provides the data foundation for your organization. It transforms raw behavioral data into real-time customer context for AI agents and advanced analytics, using [Snowplow Signals](/docs/signals/) and the Snowplow Customer Data Infrastructure (CDI) platform.

- Start by [defining the data](/docs/fundamentals/tracking-plans/) you want to collect
- Choose from over 20 [tracker SDKs](/docs/sources/) to collect data from web, mobile, server-side, and other sources, using custom tracking or [over 100 built-in events and entities](/docs/events/ootb-data/)
- Enrich your events before loading with over 15 [enrichments](/docs/pipeline/enrichments/available-enrichments/)
- Monitor your [data quality](/docs/fundamentals/failed-events/) - your pipeline will only load [validated data](/docs/fundamentals/schemas/) into your [warehouse or lake](/docs/destinations/warehouses-lakes/)
- Send enriched events to [third-party platforms](/docs/destinations/forwarding-events/) in real-time
- Model your data with our [dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/)
- Enable real time personalization with [Signals](/docs/signals/introduction/)

![diagram of snowplow architecture](/img/snowplow-cdi-signals_vertical.svg)

Choose the Snowplow platform that works for your business. See the [feature comparison page](/docs/get-started/feature-comparison/) for more information.

We offer two fully featured Customer Data Infrastructure (CDI) platforms:

- **Snowplow CDI Private Managed Cloud**: hosted in your own cloud, managed by Snowplow
- **Snowplow CDI Cloud**: hosted and managed by Snowplow

For self-hosted deployments, we have:

- **Snowplow Community Edition**: not for production use, hosted and managed by you
- **Snowplow Self-Hosted**: for production use, for existing Snowplow users

## Customer Data Infrastructure

Snowplow CDI is our full infrastructure offering. Choose whether you'd like the **data plane** to be entirely hosted in your cloud account, or whether you'd prefer Snowplow to host the pipeline infrastructure for you.

The **control plane**, which includes a UI and an API for [defining your data](/docs/event-studio/tracking-plans/) and managing your infrastructure, is always hosted by Snowplow.

### CDI Private Managed Cloud

Private Managed Cloud is a version of [Snowplow](https://snowplow.io) hosted in your own cloud account, using your data warehouse or lake. Ongoing pipeline maintenance, such as upgrades and security patches, are managed by Snowplow.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | ✅                  |               |
| API endpoints                               | ✅                  |               |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

### CDI Cloud

Cloud is a hosted version of Snowplow designed to get your organization up and running and delivering value from behavioral data as quickly as possible. With Cloud, you don't need to set up any cloud infrastructure yourself.

All data processed and collected with Snowplow Cloud is undertaken within Snowplow's own cloud account.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | ✅                  |               |
| API endpoints                               | ✅                  |               |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) | ✅                  |               |
| Data destination (warehouse / lake)         |                    | ✅             |

## Self-Hosted

With Self-Hosted, you deploy and host everything. Many features, including the **control plane**, are not available in Self-Hosted.

### Community Edition

Snowplow [Community Edition](/docs/get-started/self-hosted/) is for **non-production** use cases. It's a starter template: use it to evaluate Snowplow for testing purposes.

Community Edition infrastructure is provided under the [SLULA license](/docs/licensing/).

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | N/A                | N/A           |
| API endpoints                               | N/A                | N/A           |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

### Production Self-Hosted license

If you have an existing Snowplow implementation, either Community Edition or a legacy deployment, you're eligible for Snowplow Self-Hosted. It's a production-use license.

|                                             | Hosted by Snowplow | Hosted by you |
| ------------------------------------------- | ------------------ | ------------- |
| **Control plane**                           |                    |               |
| Management Console                          | N/A                | N/A           |
| API endpoints                               | N/A                | N/A           |
| **Data plane**                              |                    |               |
| Pipeline infrastructure (AWS / Azure / GCP) |                    | ✅             |
| Data destination (warehouse / lake)         |                    | ✅             |

---

# Running your first data models
> Using a data model to aggregate your data
> Source: https://docs.snowplow.io/docs/get-started/modeling/

Querying the events table directly — as you would have done in the previous step — can be useful for exploring your events or building custom analytics. However, for many common use cases it's much easier to use our [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/), which provide a pre-aggregated view of your data.

We recommend [dbt](https://www.getdbt.com/) for data modeling. Refer to the [setup instructions](/docs/modeling-your-data/running-data-models-via-console/) to add and configure your models in [Snowplow Console](https://console.snowplowanalytics.com), so that they can be run automatically by Snowplow.

The [Unified Digital model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) is a good starting point for websites and/or mobile applications. It provides data about page and screen views, sessions, users, and more. You can also explore the [full list of available models](/docs/modeling-your-data/modeling-your-data-with-dbt/).

If you don't have access to Snowplow Console, you'll need to install dbt and run the models yourself. Check out our [Unified Digital Quick Start tutorial](/tutorials/unified-digital/intro/) for help getting started.

> **Tip:** To start using our models with dbt, you will need to [create a dbt project](https://docs.getdbt.com/reference/commands/init) and [add the respective packages](https://docs.getdbt.com/docs/build/packages).

---

# Setting up Snowplow CDI
> Get started with Snowplow Customer Data Infrastructure on Cloud or Private Managed Cloud with setup guides for AWS, Azure, and GCP.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/

To get started with Snowplow Customer Data Infrastructure, follow the **Getting Started** steps in [Snowplow Console](https://console.snowplowanalytics.com). You will receive an account as part of your onboarding.

## CDI Cloud

If you have a Snowplow [CDI Cloud](/docs/get-started/#cdi-cloud) account, we'll set up your infrastructure for you. Check out **Pipelines** in Console to see your new pipeline.

## CDI Private Managed Cloud

If you have a Snowplow [CDI Private Managed Cloud](/docs/get-started/#cdi-private-managed-cloud) account, the Getting Started steps will guide you through setting up your cloud environment. You can also find the instructions here:

- [AWS Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-aws/)
- [Azure Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-azure/)
- [GCP Setup Guide](/docs/get-started/private-managed-cloud/setup-guide-gcp/)

Once you've set up your cloud environment, go to **Pipelines** in Console to request your new pipeline.

---

# Private Managed Cloud on AWS
> Set up Snowplow CDI Private Managed Cloud on AWS with sub-account configuration, IAM roles, and cross-account access.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-aws/

To set up Snowplow, follow the ['Getting Started' steps in the Snowplow Console](https://console.snowplowanalytics.com). You will receive an account as part of your onboarding.

## What are the steps

The first setup steps are designed to get your infrastructure in place and have you sending data as quickly as possible. The initial steps include:

- providing the right cloud environment for Snowplow to be installed
- setting up your first development environment
- selecting which warehouse, if any, you want to load your data into

Completing the forms for these initial steps should take you around 30 minutes.

## What will I need

To set up your cloud environment as required you will need:

- to be able to set up a sub-account and appropriate permissions on AWS
- to know which AWS region you’d like us to install your Snowplow pipeline into
- to know whether or not you will need to use an existing VPC or require VPC peering (note: VPC peering and using a custom VPC are additional bolt-ons)

We often find our point of contact requires support from their DevOps or Networking colleagues to complete the cloud setup step; in Console you can [easily create accounts for colleagues](/docs/account-management/managing-users/) who can complete this step for you.

## Preparing your AWS sub-account

These instructions are also provided as part of the setup flow in Console.

### Create sub-account

1. From your main AWS account, set up an Organization if you haven't done so already.
2. Create a member account (the sub-account) in that organization.
3. Sign out and sign into the new sub-account. Everything Snowplow-related will take place within this account from here in.

### Set up Role and IAM permissions

> **Note:** Before you begin, raise the [Managed policies per role](https://us-east-1.console.aws.amazon.com/servicequotas/home/services/iam/quotas/L-0DA4ABF3) attached to an IAM role Service Quota in IAM (region: us-east-1) from the default of 10 to 25. This is typically auto-approved within seconds of requesting.

1. In the AWS sub-account, open IAM from the console
2. Navigate to `Access management` > `Roles` and click `Create role`
3. Under trusted entity type, select `AWS account`, then choose `Another AWS account`

- Account ID: `793733611312`
- Leave `Require MFA` unchecked — Snowplow uses Okta for MFA, which handles authentication before role assumption

4. Search for and attach each of the following AWS managed policies:

```text
arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
arn:aws:iam::aws:policy/AmazonEC2FullAccess
arn:aws:iam::aws:policy/AmazonECS_FullAccess
arn:aws:iam::aws:policy/AmazonElastiCacheFullAccess
arn:aws:iam::aws:policy/AmazonEMRFullAccessPolicy_v2
arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess
arn:aws:iam::aws:policy/AmazonKinesisFullAccess
arn:aws:iam::aws:policy/AmazonMSKFullAccess
arn:aws:iam::aws:policy/AmazonOpenSearchServiceFullAccess
arn:aws:iam::aws:policy/AmazonRDSFullAccess
arn:aws:iam::aws:policy/AmazonRedshiftFullAccess
arn:aws:iam::aws:policy/AmazonRoute53FullAccess
arn:aws:iam::aws:policy/AmazonRoute53ResolverFullAccess
arn:aws:iam::aws:policy/AmazonS3FullAccess
arn:aws:iam::aws:policy/AmazonSNSFullAccess
arn:aws:iam::aws:policy/AmazonSQSFullAccess
arn:aws:iam::aws:policy/AmazonSSMFullAccess
arn:aws:iam::aws:policy/AutoScalingFullAccess
arn:aws:iam::aws:policy/AWSCertificateManagerFullAccess
arn:aws:iam::aws:policy/AWSLambda_FullAccess
arn:aws:iam::aws:policy/CloudWatchFullAccess
arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
arn:aws:iam::aws:policy/IAMFullAccess
```

5. Name the role `SnowplowAdmin` — this exact name is required
6. Open the role you just created, go to the Permissions tab and click `Add permissions` > `Create inline policy`. Switch to the JSON editor, paste the following, and save the policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EKSClusterManagement",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "eks:AccessKubernetesApi",
        "eks:AssociateAccessPolicy",
        "eks:AssociateIdentityProviderConfig",
        "eks:CreateAccessEntry",
        "eks:CreateAddon",
        "eks:CreateCluster",
        "eks:CreateFargateProfile",
        "eks:CreateNodegroup",
        "eks:DeleteAccessEntry",
        "eks:DeleteAddon",
        "eks:DeleteCluster",
        "eks:DeleteFargateProfile",
        "eks:DeleteNodegroup",
        "eks:DeregisterCluster",
        "eks:DescribeAccessEntry",
        "eks:DescribeAddon",
        "eks:DescribeAddonConfiguration",
        "eks:DescribeAddonVersions",
        "eks:DescribeCluster",
        "eks:DescribeFargateProfile",
        "eks:DescribeIdentityProviderConfig",
        "eks:DescribeNodegroup",
        "eks:DescribeUpdate",
        "eks:DisassociateAccessPolicy",
        "eks:DisassociateIdentityProviderConfig",
        "eks:ListAccessEntries",
        "eks:ListAccessPolicies",
        "eks:ListAddons",
        "eks:ListAssociatedAccessPolicies",
        "eks:ListClusters",
        "eks:ListFargateProfiles",
        "eks:ListIdentityProviderConfigs",
        "eks:ListNodegroups",
        "eks:ListTagsForResource",
        "eks:ListUpdates",
        "eks:RegisterCluster",
        "eks:TagResource",
        "eks:UntagResource",
        "eks:UpdateAccessEntry",
        "eks:UpdateAddon",
        "eks:UpdateClusterConfig",
        "eks:UpdateClusterVersion",
        "eks:UpdateNodegroupConfig",
        "eks:UpdateNodegroupVersion"
      ]
    },
    {
      "Sid": "KMS",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "kms:DescribeKey",
        "kms:List*"
      ]
    },
    {
      "Sid": "SecretsManager",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue",
        "secretsmanager:TagResource"
      ]
    },
    {
      "Sid": "GlobalAccelerator",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "globalaccelerator:AddEndpoints",
        "globalaccelerator:Create*",
        "globalaccelerator:Delete*",
        "globalaccelerator:Describe*",
        "globalaccelerator:List*",
        "globalaccelerator:TagResource",
        "globalaccelerator:UntagResource",
        "globalaccelerator:Update*"
      ]
    },
    {
      "Sid": "ServiceQuotas",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "servicequotas:Associate*",
        "servicequotas:Delete*",
        "servicequotas:Disassociate*",
        "servicequotas:Get*",
        "servicequotas:List*",
        "servicequotas:Put*",
        "servicequotas:Request*",
        "servicequotas:TagResource",
        "servicequotas:UntagResource"
      ]
    },
    {
      "Sid": "WAFv2ReadOnly",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "wafv2:Describe*",
        "wafv2:Get*",
        "wafv2:List*"
      ]
    },
    {
      "Sid": "Support",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "support:Add*",
        "support:Create*",
        "support:Describe*",
        "support:Refresh*",
        "support:ResolveCase"
      ]
    }
  ]
}
```

7. Go to the Trust relationships tab and click **Edit trust policy**. Replace the generated JSON with the following and click **Update policy**:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::793733611312:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "false"
        },
        "StringLike": {
          "aws:PrincipalArn": "arn:aws:iam::793733611312:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_sre-*"
        }
      }
    }
  ]
}
```

You will need to share this role with us as part of filling out the setup form in Snowplow Console.

For complete documentation from Amazon go [here](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts.html).

### Set up Role and IAM permissions with **CloudFormation**

We also provide a CloudFormation template that will create a role named SnowplowAdmin with the full permission set [here](https://snowplow-hosted-assets.s3-eu-west-1.amazonaws.com/common/iam/SnowplowAdminRole_CF.yml). The IAM quota will also need updating for this template to apply successfully.

1. Access the CloudFormation service within the sub-account
2. Go to `Stacks` select `Create stack` > `With new resources (standard)`
3. Select `Template is ready` within the Prepare template block
4. Specify an Amazon S3 URL with the full path to the SnowplowAdmin CloudFormation template and proceed
5. Provide the stack with a meaningful name such as SnowplowAdmin stack
6. Now proceed through the remainder of the prompts and choose `Create stack`

For complete documentation from Amazon go [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html).

### Set up the Snowplow deployment role

The last step is to set up the Snowplow deployment role. This is a role assumed by the machine user to make changes with Terraform.

1. In the AWS sub-account, open IAM from the console
2. Navigate to `Access management` > `Roles` and click `Create role`
3. Under trusted entity type, select `AWS account`, then choose `Another AWS account`

- Account ID: `793733611312`
- Leave `Require MFA` unchecked — Snowplow uses Okta for MFA, which handles authentication before role assumption

4. Attach the `IAMFullAccess` policy. If a Permission Boundary was set on the admin role, add it in the bottom section of the permissions page.
5. Name the role `SnowplowDeployment` — this exact name is required. Optionally add a description: "Allows the Snowplow team to programmatically deploy to this account". Create the role.
6. Open the role you just created, go to the Trust relationships tab and click **Edit trust policy**. Replace the generated JSON with the following and click **Update policy**:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::793733611312:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::793733611312:role/aws-reserved/sso.amazonaws.com/*/AWSReservedSSO_sre-*",
            "arn:aws:iam::793733611312:user/nomad-handler"
          ]
        }
      }
    }
  ]
}
```

7. Copy the Snowplow deployment role ARN. You will need to share this role with us as part of filling out the setup form in Snowplow Console.

### Provide a CIDR range for VPC peering or using a custom VPC (optional)

For VPC peering, you will need to select a peering-compatible /21 or /22 CIDR range (2048 or 1024 addresses).

If you require Snowplow to be deployed into a custom VPC, you will need to specify a /18 CIDR range (16,384 addresses) for that VPC so that we can create /20 and /23 subnets. This ensures that we have an ample room for scaling, as well as deploying additional services.

Note: VPC peering and using a custom VPC are additional bolt-ons.

### Determine if Snowplow requires a specific IAM Permission Boundary (optional)

If you require Snowplow to be deployed using a specific IAM Permission Boundary, this should be provided at the same time.

### Final checklist

If you are sending a request to our team to set up your account for you, please ensure you provide the following information:

1. SnowplowAdmin role ARN
2. SnowplowDeployment role ARN
3. AWS region to deploy into
4. VPC CIDR requirements for VPC peering or a custom VPC (optional)
5. The IAM Permission Boundary ARN (optional)

---

# Private Managed Cloud on Azure
> Set up Snowplow CDI Private Managed Cloud on Azure with subscription configuration, service principal permissions, and custom roles.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-azure/

To set up Snowplow, log in to Snowplow [Console](https://console.snowplowanalytics.com) where you'll be able to follow a step-by-step guide to getting started, including the steps below.

## Setting up your Azure account

The following steps explain how to set up your cloud environment ready for your Snowplow pipeline to be installed.

### Create a new subscription

We require a new subscription which does not contain anything else that you have running in Azure. If you don't have an Azure account, then you will need to create a new Azure account and tenant, along with a subscription for Snowplow to use. For the latest documentation on setting up a new account and tenant please refer to the [Azure documentation](https://azure.microsoft.com/).

### Enable billing for the account

Enable billing in the tenant by creating a subscription. Otherwise, the pipeline will fail to deploy. For details on enabling billing for your tenant, please refer to the Azure documentation for [subscriptions](https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription).

### Set up access and required permissions

Snowplow deploys into your tenant using a verified [application service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser#service-principal-object) (Enterprise application). We require a custom role to be assigned to the application service principal. This will allow us to create custom pipeline roles needed for deploying and managing different components of your infrastructure.

#### Consent to Snowplow Private Managed Cloud Deployment application

You will need to grant our verified application service principal the access into your Azure tenant. Once that’s done, you should see the application service principal under _Microsoft Entra ID_ → _Enterprise Applications_.

1. Grab your Azure tenant ID by navigating to Microsoft Entra ID
2. Complete the consent URL where `<TenantID>` is your tenant ID. The `client_id` set is for “Snowplow BDP Enterprise Deployment” application service principal. Go to this URL in a browser window
   ```text
   https://login.microsoftonline.com/<TenantID>/oauth2/authorize?client_id=0581feb4-b614-42c7-b8e7-b4e7fba9153a&response_type=code
   ```
3. A consent window will appear detailing that an Enterprise application is being set up in your tenant. It needs to be accepted by your Azure tenant admin for the organization (there is a tick box that must be ticked). After accepting, Microsoft redirects you to a page unrelated to the Azure Portal, so close this window
4. Verify the trust has been established by viewing “Snowplow Private Managed Cloud Deployment” application in the Enterprise Applications section of Entra ID

#### Create and assign role to application service principal

Create a custom role and assign it the “Snowplow BDP Enterprise Deployment” application service principal under your subscription. This grants the permission to create distinct roles for deploying and managing infrastructure resources that make up your pipeline.

1. Navigate to your newly created subscription
2. Click into “Access Control (IAM)”
3. Click “Add custom role”
4. Create the following role, which allows Snowplow to create various roles that are specific to components that make up your pipeline. The `customer_subscription_id` should be the subscription ID that Snowplow will deploy into. The role should be single scoped to the subscription. It is recommended to have a custom role per environment to give clear separation between environments.
   ```json
   {
       "properties": {
           "roleName": "Snowplow-Deployment-Role-Creator-Role",
           "description": "Custom policy to allow creation of individual Azure stack related roles",
           "assignableScopes": [
               "/subscriptions/<customer_subscription_id>"
           ],
           "permissions": [
               {
                   "actions": [
                       "Microsoft.Authorization/roleAssignments/write",
                       "Microsoft.Authorization/roleAssignments/read",
                       "Microsoft.Authorization/roleAssignments/delete",
                       "Microsoft.Authorization/roleDefinitions/write",
                       "Microsoft.Authorization/roleDefinitions/read",
                       "Microsoft.Authorization/roleDefinitions/delete",
                       "Microsoft.Resources/subscriptions/resourcegroups/read"
                   ],
                   "notActions": [],
                   "dataActions": [],
                   "notDataActions": []
               }
           ]
       }
   }
   ```
5. Within “Access Control (IAM)”, click “Add role assignment”
6. Assign the `Snowplow-Deployment-Role-Creator-Role` to service principal “Snowplow BDP Enterprise Deployment”. The role can be found under the “Privileged administrator roles” tab. The Conditions tab should be selected and you must select the third option. This is required to enable the [Lighthouse Offer to be created as detailed in Microsoft documentation](https://learn.microsoft.com/en-us/azure/lighthouse/how-to/deploy-policy-remediation#create-a-user-who-can-assign-roles-to-a-managed-identity-in-the-customer-tenant:~:text=To%20allow%20a,Administrator%20or%20Owner) ![IAM role assignment conditions](/assets/images/azure_role_assignment_conditions-d9c19eb3283061a7a93b07595eb33f38.png)

### Determine if Snowplow requires a specific VPC (optional)

If you require Snowplow to be deployed into a specific VPC CIDR range, this should be provided at the same time as credentials. We need a /16 provided for the VPC so that we can create /20 subnets (note: [VPC peering and using a custom VPC is an additional bolt-on](https://snowplow.io/snowplow-behavioral-data-platform-product-description/#h-vpc-peering-aws-gcp))

### Check subscription resource availability

You must check that the following infrastructure is available in the subscription before deployment starts. If the required instances are unavailable, it will result in delays to getting started.

Snowplow requires the below instances and databases. We have provided the Azure CLI commands that can be used to check their availability. Please note that they are required in all three availability zones in the region for redundancy. You may need to raise a quota ticket with Azure or contact your Azure Technical Account Manager to ensure the instance types are fully available.

Instances D4s\_v5 (preferred), or D4s\_v4 / D4\_v5 / D4\_v4 series:

```bash
az vm list-skus --location <REGION> --size Standard_D4s_v5 --all --output table
az vm list-skus --location <REGION> --size Standard_D4s_v4 --all --output table
az vm list-skus --location <REGION> --size Standard_D4_v5 --all --output table
az vm list-skus --location <REGION> --size Standard_D4_v4 --all --output table
```

Database B\_Standard\_B2s:

```bash
az vm list-skus --location <REGION> --size Standard_B2s --all --output table
```

Please note that the above is for a minimal deployment. If you will be sending over one billion events per month through the pipeline, please speak to us for advice on what instance sizes are recommended.

### Final checklist

If you are sending a request to our team to set up deployment into your Azure account, please ensure you provide the following information:

1. The tenant ID
2. The subscription ID
3. Azure region to deploy into
4. The ID of the `Snowplow-Deployment-Role-Creator-Role`
5. The specific VPC CIDR range that the pipeline should be deployed in (optional).

---

# Private Managed Cloud on GCP
> Set up Snowplow CDI Private Managed Cloud on GCP with project configuration, IAM roles, and service account permissions.
> Source: https://docs.snowplow.io/docs/get-started/private-managed-cloud/setup-guide-gcp/

To set up Snowplow, log in to Snowplow [Console](https://console.snowplowanalytics.com) where you'll be able to follow a step-by-step guide to getting started, including the steps below.

## Setting up your Google project

The following steps explain how to set up your cloud environment ready for your Snowplow pipeline to be installed.

### Create a new project

Please create a new project segregated from anything else you have running in GCP. For the latest documentation on setting up a new project please refer to Google Cloud's documentation [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects).

### Set up required permissions

Please add **<techops-cloud-admin@snowplowanalytics.com>** to your project. This account will be used to install and maintain your pipeline in GCP. The following roles are required:

- [`editor`](https://cloud.google.com/iam/docs/roles-overview#basic-definitions)
- [`Roles/resourcemanager.projectIamAdmin`](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin)
- [`Roles/errorreporting.admin`](https://cloud.google.com/iam/docs/understanding-roles#errorreporting.admin)
- [`Roles/logging.admin`](https://cloud.google.com/iam/docs/understanding-roles#logging.admin)
- [`Roles/monitoring.admin`](https://cloud.google.com/iam/docs/understanding-roles#monitoring.admin)
- [`Roles/iam.roleAdmin`](https://cloud.google.com/iam/docs/understanding-roles#iam.roleAdmin)
- [`Roles/iam.serviceAccountAdmin`](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountAdmin)
- [`Roles/container.admin`](https://cloud.google.com/iam/docs/roles-permissions/container#container.admin)
- [`Roles/run.admin`](https://cloud.google.com/run/docs/reference/iam/roles#run.admin)
- [`Roles/datastore.owner`](https://docs.cloud.google.com/iam/docs/roles-permissions/firestore#datastore.owner)

The following roles are also required if using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) within GCP:

- [`Roles/secretmanager.secretAccessor`](https://cloud.google.com/iam/docs/understanding-roles#secretmanager.secretAccessor)
- [`Roles/secretmanager.secretVersionAdder`](https://cloud.google.com/iam/docs/understanding-roles#secretmanager.secretVersionAdder)

### Enable billing for the project

Please enable billing on the project as failing to do so will prevent the pipeline from being installed. For details on enabling billing for your project, please refer to the GCP documentation for [APIs and billing](https://support.google.com/googleapi/answer/6158867).

### Ensure that the following constraints are turned off

The following is a list of Organizational Policy constraints that we require to be turned off, otherwise we are unable to deploy the pipeline. For some, more specific configuration would enable the constraint to stay and the pipeline to deploy. This list will be updated as we discover more constraints that are not compatible with the pipeline deployment.

- `constraints/compute.requireShieldedVm`
- `constraints/gcp.resourceLocations`
- `constraints/iam.disableServiceAccountCreation`
- `constraints/iam.disableServiceAccountKeyCreation`
- `constraints/sql.restrictPublicIp`
- `constraints/sql.restrictAuthorizedNetworks`
- `constraints/sql.restrictNoncompliantResourceCreation`

---

# Querying your first events
> Inspecting the events you tracked
> Source: https://docs.snowplow.io/docs/get-started/querying/

Once you've tracked some events, you'll want to look at them in your data warehouse, database, or lake. The exact steps will depend on your choice of storage and your Snowplow platform.

For **Snowplow CDI** Private Managed Cloud or Cloud customers, you can find your connection details in [Snowplow Console](https://console.snowplowanalytics.com/destinations), under the destination you've selected.

Follow [our querying guide](/docs/destinations/warehouses-lakes/querying-data/) for advice on querying your data.

## Snowplow Self-Hosted

If you don't have access to Snowplow Console, follow these instructions to connect to your Snowplow data:

**Postgres:**

Your database will be named according to the `postgres_db_name` Terraform variable. It will contain two schemas:

- `atomic` — the validated events
- `atomic_bad` — the [failed events](/docs/fundamentals/failed-events/)

You can connect to the database using the credentials you provided for the loader in the Terraform variables (`postgres_db_username` and `postgres_db_password`), along with the `postgres_db_address` and `postgres_db_port` Terraform outputs.

> **Tip:** If you need to reset your username or password, you can follow [these steps](https://aws.amazon.com/premiumsupport/knowledge-center/reset-master-user-password-rds/).

See [the AWS RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html) for more details on how to connect.

> **Note:** If you opted for the `secure` option, you will first need to create a tunnel into your VPC to be able to connect to your RDS instance and be able to query the data. A common solution to this issue is to configure a bastion host as described [here](https://repost.aws/knowledge-center/rds-connect-using-bastion-host-linux).

**Redshift:**

The database name and the schema name will be defined by the `redshift_database` and `redshift_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`redshift_loader_user` and `redshift_loader_password`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

To connect, you can use the Redshift UI or something like [`psql`](https://www.postgresql.org/docs/current/app-psql.html).

**BigQuery:**

The database will be called `<prefix>_snowplow_db`, where `<prefix>` is the prefix you picked in your Terraform variables file. It will contain an `atomic` schema with your validated events.

You can access the database via the [BigQuery UI](https://console.cloud.google.com/bigquery).

**Snowflake:**

The database name and the schema name will be defined by the `snowflake_database` and `snowflake_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`snowflake_loader_user` and `snowflake_loader_password`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

To connect, you can use either Snowflake dashboard or [SnowSQL](https://docs.snowflake.com/en/user-guide/snowsql.html).

**Databricks:**

> **Info:** On Azure, you have created an external table in the [last step of the guide](/docs/get-started/self-hosted/quick-start/#configure-the-destination). Use this table and ignore the text below.

The database name and the schema name will be defined by the `databricks_database` and `databricks_schema` variables in Terraform.

There are two different ways to login to the database:

- The first option is to use the credentials you configured for the loader in the Terraform variables (`databricks_loader_user` and `databricks_loader_password`, or alternatively the `databricks_auth_token`)
- The second option is to grant `SELECT` permissions on the schema to an existing user

See the [Databricks tutorial](https://docs.databricks.com/getting-started/quick-start.html) for more details on how to connect. The documentation on [Unity Catalog](https://docs.databricks.com/data-governance/unity-catalog/queries.html) is also useful.

**Synapse Analytics:**

In Synapse Analytics, you can connect directly to the data residing in ADLS. You will need to know the names of the storage account (set in the `storage_account_name` Terraform variable) and the storage container (it’s a fixed value: `lake-container`).

Follow [the Synapse documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format) and use the `OPENROWSET` function. If you created a data source in the [last step](/docs/get-started/self-hosted/quick-start/#configure-the-destination) of the quick start guide, your queries will be a bit simpler.

> **Tip:** If you created a OneLake shortcut in the [last step](/docs/get-started/self-hosted/quick-start/#configure-the-destination) of the quick start guide, you will be able to explore Snowplow data in Fabric, for example, using Spark SQL.

***

---

# Snowplow Self-Hosted quick start: frequently asked questions
> Common questions about managing, upgrading, and troubleshooting Snowplow Self-Hosted deployments with Terraform.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/faq/

## How do I shut down the pipeline?

If you would like to shut down your pipeline then you can easily do so by running `terraform destroy`.

Note that if you want to delete your S3 bucket or Postgres database, you would need to do that from within the AWS, GCP or Azure console. If you want to maintain these then you can - just be aware that next time you spin up your pipeline you might see some errors.

## How do I upgrade the version of the application that I am using?

We release new versions of our pipeline components very frequently; however the versions used within the terraform modules are updated a few times a year to the most stable and recommended versions of our components. [Sign up to get the latest updates](https://go.snowplowanalytics.com/get-snowplow-technology-updates) on platform releases and new features.

When a new version of a module is released, follow these instructions to upgrade:

- Update the module version in your terraform
- Run `terraform plan` to check for what changes will be made

## Which enrichments are enabled by default?

The following enrichments are enabled by default within the Enrich module:

- [UA parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)
- [Campaign Attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)
- [Event fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)
- [Referer parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)

Other available enrichments enrichments and the configurations can be found [here](/docs/pipeline/enrichments/available-enrichments/).

Follow the [documentation](/docs/pipeline/enrichments/managing-enrichments/terraform/) if you want to enable or disable certain enrichments.

## Troubleshooting Terraform Errors

The following are some common errors that you might encounter when running `terraform plan` or `terraform apply`.

AWS:

**Error: Invalid provider configuration**

`Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider documentation.`

**Solution:** Double check that your AWS Access Key ID and AWS Secret Access Key are set up correctly (with no typos) using the `aws configure` command.

**Error:** **"x.x.x.x" is not a valid CIDR block**

`"x.x.x.x" is not a valid CIDR block: invalid CIDR address: x.x.x.x with module.iglu_server.aws_security_group_rule.ingress_tcp_22, on .terraform/modules/iglu_server/main.tf line 143, in resource "aws_security_group_rule" "ingress_tcp_22": 143: cidr_blocks = var.ssh_ip_allowlist`

**Solution**: Add a mask to the IP in your terraform.tvars file, e.g. `x.x.x.x/32`.

**Error creating Application Load Balancer**

`ValidationError: At least two subnets in two different Availability Zones must be specified status code: 400`

**Solution**: Add subnets to cover at least 2 availability zones. See this [AWS guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-public-private-vpc.html) on how to set this up.

After this step, add the two freshly created subnets to your `terraform.tfvars` file like this: `public_subnet_ids = ["subnet-00000000", "subnet-00000001"]` and run `terraform apply` again.

**Error creating DB Subnet Group**

`DBSubnetGroupDoesNotCoverEnoughAZs: DB Subnet Group doesn't meet availability zone coverage requirement. Please add subnets to cover at least 2 availability zones. Current coverage: 1 status code: 400`

**Solution**: Add subnets to cover at least 2 availability zones. See this [AWS guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-public-private-vpc.html) on how to set this up.

After this step, add the two freshly created subnets to your `terraform.tfvars` file like this: `public_subnet_ids = ["subnet-00000000", "subnet-00000001"]` and run `terraform apply` again.

**Error creating DB Instance: InvalidParameterValue**

`The parameter MasterUserPassword is not a valid password. Only printable ASCII characters besides '/', '@', '"', ' ' may be used.`

**Solution**: Modify the `iglu_db_password` in your `terraform.tfvars` file so that it does not contain any forbidden characters like '\_', '\*'. Make sure the password is not longer than 13 characters.

---

# Set up Snowplow Self-Hosted
> Deploy Snowplow Community Edition using Terraform modules on AWS, GCP, or Azure for non-production testing and evaluation.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/

This page is an introduction for the [Self-Hosted](/docs/get-started/#self-hosted) [Quick Start guide](/docs/get-started/self-hosted/quick-start/). Follow the Quick Start to set up a self-hosted deployment.

We have built a set of [terraform](https://www.terraform.io/docs/language/modules/develop/index.html) modules, which automates the setup and deployment of the required infrastructure and applications for an operational Snowplow Community Edition pipeline, with just a handful of input variables required on your side.

After following this guide, you will be able to: 

- Collect granular, well-structured data with our suite of web, mobile and server side [SDKs](/docs/sources/)
- Create your own [custom events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities)
- Easily enable and disable our suite of [out-of-the-box enrichments](/docs/pipeline/enrichments/available-enrichments/)
- Consume your rich data from the data warehouse, database, lake and/or real-time stream

> **Warning:** Please read the terms of the [Snowplow Limited Use License Agreement](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)) which apply to Community Edition.
>
> In short, Community Edition is meant for testing and evaluating Snowplow and must not be deployed in production.

## Required time

If you are proficient with Terraform and cloud tooling, 1 hour should be sufficient. Otherwise, expect to spend a few hours.

## Cost

Assuming around 100 events per second, the pipeline will cost around $200 per month on AWS and $240 per month on GCP.

> **Tip:** To reduce the costs, you can tweak the configuration (e.g. use smaller instances), or shut down the pipeline when not in use.

## Scale

Out of the box, the deployed pipeline will handle up to \~100 events per second (\~9 million events per day).

## Getting help

Check out our [Community](https://community.snowplow.io/). If you run into any problems or have any questions, we are here to help.

---

# Quick start guide for Snowplow Self-Hosted
> Deploy Snowplow Community Edition on AWS, GCP, or Azure using Terraform modules with step-by-step instructions.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/quick-start/

This guide will take you through how to spin up a Snowplow Community Edition pipeline using the [Snowplow Terraform modules](https://registry.terraform.io/namespaces/snowplow-devops). _(Not familiar with Terraform? Take a look at [Infrastructure as code with Terraform](https://learn.hashicorp.com/tutorials/terraform/infrastructure-as-code?in=terraform/aws-get-started).)_

## Prerequisites

[Sign up](https://snowplow.io/pricing/) for Snowplow Community Edition and follow the link in the email to get a copy of the repository containing the Terraform code.

> **Warning:** Please read the terms of the [Snowplow Limited Use License Agreement](/limited-use-license-1.1/) ([FAQ](/docs/licensing/limited-use-license-faq/)) which apply to Community Edition.
>
> In short, Community Edition is meant for testing and evaluating Snowplow and must not be deployed in production.

Install [Terraform 1.0.0](https://www.terraform.io/downloads.html) or higher. Follow the instructions to make sure the `terraform` binary is available on your `PATH`. You can also use [tfenv](https://github.com/tfutils/tfenv) to manage your Terraform installation.

**AWS:**

Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) version 2.

[Configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) the CLI against a role that has the `AdminstratorAccess` policy attached.

> **Warning:** `AdminstratorAccess` allows all actions on all AWS services and shouldn't be used in production

Details on how to configure the AWS Terraform Provider can be found [on the registry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables).

**GCP:**

Install [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).

Make sure the following APIs are active in your GCP account (this list might not be exhaustive and is subject to change as GCP APIs evolve):

- [Compute Engine API](https://console.cloud.google.com/apis/api/compute.googleapis.com)
- [Cloud Resource Manager API](https://console.cloud.google.com/apis/api/cloudresourcemanager.googleapis.com)
- [Identity and Access Management (IAM) API](https://console.cloud.google.com/apis/api/iam.googleapis.com)
- [Cloud Pub/Sub API](https://console.cloud.google.com/apis/api/pubsub.googleapis.com)
- [Cloud SQL Admin API](https://console.cloud.google.com/apis/api/sqladmin.googleapis.com)

Configure a Google Cloud service account. See [details on using the service account with the Cloud SDK](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable). You will need to:

- Navigate to your service account on Google Cloud Console
- Create a new JSON Key and store it locally
- Create the environment variable by running `export GOOGLE_APPLICATION_CREDENTIALS="KEY PATH"` in your terminal

**Azure:**

Install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).

If your organisation has an existing Azure account, make sure your user has been granted the following roles on a valid Azure Subscription:

- [Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor)
- [User Access Administrator](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#user-access-administrator)
- [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor)

> **Warning:** `User Access Administrator` allows the user to modify, create and delete permissions across Azure resources, and shouldn’t be used in production. Instead, you can use a custom role with the following permissions:
>
> - `Microsoft.Authorization/roleAssignments/write` to deploy the stacks below
> - `Microsoft.Authorization/roleAssignments/delete` to destroy them

If you don’t have an Azure account yet, you can get started with a new [pay-as-you-go account](https://azure.microsoft.com/free/).

Details on how to configure the Azure Terraform Provider can be found [on the registry](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/azure_cli).

***

## Storage options

The sections below will guide you through setting up your destination to receive Snowplow data, but for now here is an overview.

| Warehouse         | AWS | GCP | Azure |
| ----------------- | --- | --- | ----- |
| Snowflake         | ✅   | ❌   | ❌     |
| Databricks        | ✅   | ❌   | ✅     |
| Redshift          | ✅   | —   | —     |
| BigQuery          | —   | ✅   | —     |
| Synapse Analytics | —   | —   | ✅     |

## Real-time streaming options

As part of the deployment, your data will be available in real-time streams corresponding to the cloud provider you have chosen. You can consume data directly from these streams, either in addition to or instead of the data warehouse.

| Stream    | AWS | GCP | Azure |
| --------- | --- | --- | ----- |
| Kinesis   | ✅   | ❌   | ❌     |
| Pub/Sub   | ❌   | ✅   | ❌     |
| EventHubs | ❌   | ❌   | ✅     |

For an out-of-the-box solution to accessing this data in real-time streams, you can [check out our Snowbridge tool](/docs/api-reference/snowbridge/). Alternatively, if you want to develop a custom consumer, you can [leverage our Analytics SDKs](/docs/api-reference/analytics-sdk/) to parse the event formats more easily.

> **Note:** EventHubs topics are deployed in a Kafka-compatible model, so you can consume from them using standard Kafka connector libraries.

**AWS:**

There are three main storage options for you to select: Redshift, Snowflake and Databricks. Additionally, there is an S3 option, which is primarily used to archive enriched events and to store [failed events](/docs/fundamentals/failed-events/).

We recommend to only load data into a single destination, but nothing prevents you from loading into multiple destinations with the same pipeline (e.g. for testing purposes).

**GCP:**

Only one storage option is provided: BigQuery.

**Azure:**

Only a data lake (ADLS) storage option is provided. This enables querying data from Databricks and Synapse Analytics.

***

## Set up a VPC to deploy into

**AWS:**

AWS provides a [default VPC](https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html#view-default-vpc) in every region for your sub-account. Take a note of the identifiers of this VPC and the associated subnets for later parts of the deployment.

**GCP:**

GCP provides a [default VPC](https://cloud.google.com/vpc/docs/vpc#default-network) for your project. In the steps below, it is sufficient to set `network = default` and leave subnetworks empty, and Terraform will discover the correct network to deploy into.

**Azure:**

Azure does not provide a default VPC or resource group, so we have added a helper module to create a working network we can deploy into.

To use our out-of-the-box network, you will need to navigate to the `terraform/azure/base` directory in the code repository and update the input variables in `terraform.tfvars`.

Once that’s done, you can use Terraform to create your base network.

```bash
terraform init
terraform plan
terraform apply
```

After the deployment completes, you should get an output like this:

```text
...
vnet_subnets_name_id = {
  "collector-agw1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/collector-agw1"
  "iglu-agw1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu-agw1"
  "iglu-vmss1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu-vmss1"
  "iglu1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/iglu1"
  "pipeline1" = "/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/pipeline1"
}
```

These are the subnet identifiers, e.g. `"/subscriptions/<...>/resourceGroups/<...>/providers/Microsoft.Network/virtualNetworks/<...>/subnets/pipeline1"` is the identifier of the `pipeline1` subnet. Take note of these four identifiers, as you will need them in the following steps.

***

## Set up Iglu Server

The first step is to set up the [Iglu Server](/docs/api-reference/iglu/) stack required by the rest of your pipeline.

This will allow you to create and evolve your own [custom events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/#custom-entities). Iglu Server stores the [schemas](/docs/fundamentals/schemas/) for your events and entities and fetches them as your events are processed by the pipeline.

### Step 1: Update the `iglu_server` input variables

Once you have cloned the code repository, you will need to navigate to the `iglu_server` directory to update the input variables in `terraform.tfvars`.

**AWS:**

```bash
cd terraform/aws/iglu_server/default
nano terraform.tfvars # or other text editor of your choosing
```

**GCP:**

```bash
cd terraform/gcp/iglu_server/default
nano terraform.tfvars # or other text editor of your choosing
```

**Azure:**

```bash
cd terraform/azure/iglu_server
nano terraform.tfvars # or other text editor of your choosing
```

If you [used our `base` module](#set-up-a-vpc-to-deploy-into), you will need to set these variables as follows:

- `resource_group_name`: use the same value as you supplied in `base`
- `subnet_id_database`: use the identifier of the `iglu1` subnet from `base`
- `subnet_id_servers`: use the identifier of the `iglu-vmss1` subnet from `base`

***

To update your input variables, you’ll need to know a few things:

- Your IP Address. [Help](https://duckduckgo.com/?q=ip+address\&t=ffab\&ia=answer).
- A UUIDv4 to be used as the Iglu Server’s API Key. [Help](https://duckduckgo.com/?t=ffab\&q=uuid\&ia=answer).
- How to generate an SSH Key.

> **Tip:** On most systems, you can generate an SSH Key with: `ssh-keygen -t rsa -b 4096`. This will output where you public key is stored, for example: `~/.ssh/id_rsa.pub`. You can get the value with `cat ~/.ssh/id_rsa.pub`.

**Telemetry notice**

By default, Snowplow collects telemetry data for each of the Quick Start Terraform modules. Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `user_provided_id` variable.

If you wish to disable telemetry, you can do so by setting `telemetry_enabled` to `false`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.

### Step 2: Run the `iglu_server` Terraform script

You can now use Terraform to create your Iglu Server stack.

**AWS:**

You will be asked to select a region, you can find more information about [available AWS regions here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).

```bash
terraform init
terraform plan
terraform apply
```

The deployment will take roughly 15 minutes.

**GCP:**

```bash
terraform init
terraform plan
terraform apply
```

**Azure:**

```bash
terraform init
terraform plan
terraform apply
```

***

Once the deployment is done, it will output `iglu_server_dns_name`. Make a note of this, you’ll need it when setting up your pipeline. If you have attached a custom SSL certificate and set up your own DNS records, then you don’t need this value.

## Prepare the destination

Depending on the destination(s) you’ve choosen, you might need to perform a few extra steps to prepare for loading data there.

> **Tip:** Feel free to go ahead with these while your Iglu Server stack is deploying.

**Redshift:**

Assuming you already have an active Redshift cluster, execute the following SQL (replace the `${...}` variables with your desired values). You will need the permissions to create databases, users and schemas in the cluster.

```sql
-- 1. (Optional) Create a new database - you can also use an existing one if you prefer
CREATE DATABASE ${redshift_database};
-- Log back into Redshift with the new database:
-- psql --host <host> --port <port> --username <admin> --dbname ${redshift_database}

-- 2. Create a schema within the database
CREATE SCHEMA IF NOT EXISTS ${redshift_schema};

-- 3. Create the loader user
CREATE USER ${redshift_loader_user} WITH PASSWORD '${redshift_password}';

-- 4. Ensure the schema is owned by the loader user
ALTER SCHEMA ${redshift_schema} OWNER TO ${redshift_loader_user};
```

> **Note:** You will need to ensure that the loader can access the Redshift cluster over whatever port is configured for the cluster (usually, `5439`).

**BigQuery:**

No extra steps needed.

**Snowflake:**

You will need to generate a key pair following the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth#generate-the-private-key). Make sure to enter an empty passphrase, as the terraform module below does not support keys with passphrases (for simplicity).

Execute the following SQL (replace the `${...}` variables with your desired values). You will need access to both `SYSADMIN` and `SECURITYADMIN` level roles to action this:

```sql
-- 1. (Optional) Create a new database - you can also use an existing one if you prefer
CREATE DATABASE IF NOT EXISTS ${snowflake_database};

-- 2. Create a schema within the database
CREATE SCHEMA IF NOT EXISTS ${snowflake_database}.${snowflake_schema};

-- 3. Create a role that will be used for loading data
CREATE ROLE IF NOT EXISTS ${snowflake_loader_role};
GRANT USAGE ON DATABASE ${snowflake_database} TO ROLE ${snowflake_loader_role};
GRANT ALL ON SCHEMA ${snowflake_database}.${snowflake_schema} TO ROLE ${snowflake_loader_role};

-- 4. Create a user that can be used for loading data
CREATE USER IF NOT EXISTS ${snowflake_loader_user}
  RSA_PUBLIC_KEY='MIIBIj...'
  TYPE = SERVICE
  DEFAULT_ROLE = ${snowflake_loader_role}
  EMAIL = 'loader@acme.com';
GRANT ROLE ${snowflake_loader_role} TO USER ${snowflake_loader_user};

-- 5. (Optional) Grant this role to SYSADMIN to make debugging easier from admin users
GRANT ROLE ${snowflake_loader_role} TO ROLE SYSADMIN;
```

**Databricks:**

> **Info:** On Azure, we currently support loading data into Databricks via a data lake. You can still follow Step 1 below to create the cluster, however you should skip the rest of these steps. Instead, proceed with [deploying the pipeline](#set-up-the-pipeline) — we will return to configuring Databricks [at the end of this guide](#configure-the-destination).

#### Step 1: Create a cluster

> **Note:** The cluster spec described below should be sufficient for a monthly event volume of up to 10 million events. If your event volume is greater, then you may need to increase the size of the cluster.

Create a new cluster, following the [Databricks documentation](https://docs.databricks.com/clusters/create.html), with the following settings:

- the runtime version must be 13.0 or greater (but not 13.1 or 13.2)
- single node cluster
- "smallest" size node type
- auto-terminate after 30 minutes.

**Advanced cluster configuration (optional)**

You might want to configure cluster-level permissions, by following [the Databricks instructions on cluster access control](https://docs.databricks.com/security/access-control/cluster-acl.html). Snowplow's RDB Loader must be able to restart the cluster if it is terminated.

If you use AWS Glue Data Catalog as your metastore, [follow these Databricks instructions](https://docs.databricks.com/data/metastores/aws-glue-metastore.html) for the relevant spark configurations. You will need to set `spark.databricks.hive.metastore.glueCatalog.enabled true` and `spark.hadoop.hive.metastore.glue.catalogid <aws-account-id-for-glue-catalog>` in the spark configuration.

You can configure your cluster with [an instance profile](https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html) if it needs extra permissions to access resources. For example, if the S3 bucket holding the delta lake is in a different AWS account.

#### Step 2: Note the JDBC connection details

1. In the Databricks UI, click on "Compute" in the sidebar
2. Click on the _RDB Loader cluster_ and navigate to "Advanced options"
3. Click on the "JDBC/ODBC" tab
4. Note down the JDBC connection URL, specifically the `host`, the `port` and the `http_path`

#### Step 3: Create an access token for the loader

> **Warning:** The access token must not have a specified lifetime. Otherwise, the loader will stop working when the token expires.

1. Navigate to the user settings in your Databricks workspace

   - For Databricks hosted on AWS, the "Settings" link is in the lower left corner in the side panel
   - For Databricks hosted on Azure, "User Settings" is an option in the drop-down menu in the top right corner.

2. Go to the "Access Tokens" tab

3. Click the "Generate New Token" button

4. Optionally enter a description (comment). Leave the expiration period empty

5. Click the "Generate" button

6. Copy the generated token and store it in a secure location

#### Step 4: Create the catalog and the schema

Execute the following SQL (replace the `${...}` variables with your desired values). The default catalog is called `hive_metastore` and is what you should use in the loader unless you specify your own.

```sql
-- USE CATALOG ${catalog_name}; -- Uncomment if your want to use a custom Unity catalog and replace with your own value.

CREATE SCHEMA IF NOT EXISTS ${schema_name}
-- LOCATION s3://<custom_location>/ -- Uncomment if you want tables created by Snowplow to be located in a non-default bucket or directory.
;
```

**Advanced security configuration (optional)**

The security principal used by the loader needs a `Databricks SQL access` permission, which can be enabled in the _Admin Console_.

Databricks does not have table access enabled by default. Enable it with an initialization script:

```scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
|#!/bin/bash
|
|cat << 'EOF' > /databricks/driver/conf/00-custom-table-access.conf
|[driver] {
|  "spark.databricks.acl.sqlOnly" = "true"
|}
|EOF
""".stripMargin, true)
```

After adding the script, you need to restart the cluster. Verify that changes took effect by evaluating `spark.conf.get("spark.databricks.acl.sqlOnly")`, which should return `true`.

Follow the rest of the quick start guide so that the loader creates the required tables. Afterwards, reconfigure the permissions:

```sql
-- Clean initial permissions from tables
REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.events FROM `<principal>`;
REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.manifest FROM `<principal>`;
REVOKE ALL PRIVILEGES ON TABLE <catalog>.<schema>.rdb_folder_monitoring FROM `<principal>`;

-- Clean initial permissions from schema
REVOKE ALL PRIVILEGES ON SCHEMA <catalog>.<schema> FROM `<principal>`;

-- Loader will run CREATE TABLE IF NOT EXISTS statements, so USAGE and CREATE and both required.
GRANT USAGE, CREATE ON SCHEMA <catalog>.<schema> TO `<principal>`;

-- COPY TO statement requires ANY FILE and MODIFY for the receiving table
GRANT SELECT ON ANY FILE TO `<principal>`;
GRANT MODIFY  ON TABLE  <catalog>.<schema>.events TO `<principal>`;

-- These tables are used to store internal loader state
GRANT MODIFY, SELECT ON TABLE  <catalog>.<schema>.manifest TO `<principal>`;
GRANT MODIFY, SELECT ON TABLE  <catalog>.<schema>.rdb_folder_monitoring TO `<principal>`;
```

**Synapse Analytics:**

No extra steps needed. Proceed with [deploying the pipeline](#set-up-the-pipeline) — we will return to configuring Synapse [at the end of this guide](#configure-the-destination).

***

## Set up the pipeline

In this section, you will update the input variables for the Terraform module, and then run the Terraform script to set up your pipeline. At the end you will have a working Snowplow pipeline ready to receive web, mobile or server-side data.

### Step 1: Update the `pipeline` input variables

Navigate to the `pipeline` directory in the code repository and update the input variables in `terraform.tfvars`.

**AWS:**

```bash
cd terraform/aws/pipeline/default
nano terraform.tfvars # or other text editor of your choosing
```

**GCP:**

```bash
cd terraform/gcp/pipeline/default
nano terraform.tfvars # or other text editor of your choosing
```

**Azure:**

```bash
cd terraform/azure/pipeline
nano terraform.tfvars # or other text editor of your choosing
```

If you [used our `base` module](#set-up-a-vpc-to-deploy-into), you will need to set these variables as follows:

- `resource_group_name`: use the same value as you supplied in `base`
- `subnet_id_lb`: use the identifier of the `collector-agw1` subnet from `base`
- `subnet_id_servers`: use the identifier of the `pipeline1` subnet from `base`

**Confluent Cloud**

If you already use Confluent Cloud, you can opt to create the necessary message topics there, instead of relying on Azure Event Hubs. This way, you will also benefit from features like [Stream Lineage](https://docs.confluent.io/cloud/current/stream-governance/stream-lineage.html).

To do this, you will need to:

- Set the `stream_type` variable to `confluent_cloud`
- Create 3 or 4 topics manually in your Confluent cluster and add their names in the respective variables (`confluent_cloud_..._topic_name`)
- Create an API key and fill the relevant fields (`confluent_cloud_api_key`, `confluent_cloud_api_secret`)
- Add a bootstrap server in `confluent_cloud_bootstrap_server`

> **Tip:** If you need to stay within the free tier for your Confluent cluster, make sure to select no more than 2 partitions for each topic.

***

To update your input variables, you’ll need to know a few things:

- Your IP Address. [Help](https://duckduckgo.com/?q=ip+address\&t=ffab\&ia=answer).
- Your Iglu Server’s domain name from the [previous step](#step-2-run-the-iglu_server-terraform-script)
- Your Iglu Server’s API Key from the [previous step](#step-1-update-the-iglu_server-input-variables)
- How to generate an SSH Key.

> **Tip:** On most systems, you can generate an SSH Key with: `ssh-keygen -t rsa -b 4096`. This will output where you public key is stored, for example: `~/.ssh/id_rsa.pub`. You can get the value with `cat ~/.ssh/id_rsa.pub`.

#### Destination-specific variables

**AWS:**

As mentioned [above](#storage-options), there are several options for the pipeline’s destination database. For each destination you’d like to configure, set the `<destination>_enabled` variable (e.g. `redshift_enabled`) to `true` and fill all the relevant configuration options (starting with `<destination>_`).

When in doubt, refer back to the [destination setup](#prepare-the-destination) section where you have picked values for many of the variables.

**Snowflake**

If you are using Snowflake, you will need to provide a private key you’ve generated during [destination setup](#prepare-the-destination).

Here’s how to do it:

```text
snowflake_streaming_loader_private_key = <<EOT
-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCd2dEYSUp3hdyK
5hWwpkNGG56hLFWDK47oMf/Niu+Yh+8Wm4p9TlPje+UuKOnK5N4nAbM4hlhKyEJv
...
99Xil8uas3v7o2OSe7FfLA==
-----END PRIVATE KEY-----
EOT
```

> **Warning:** For all active destinations, change any `_password` setting to a value that _only you_ know.

**GCP:**

Set `bigquery_db_enabled` to `true` to enable loading to BigQuery.

**Azure:**

Set `lake_enabled` to `true` to enable loading to a data lake.

***

### Step 2: Run the `pipeline` Terraform script

**AWS:**

You will be asked to select a region, you can find more information about [available AWS regions here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).

```bash
terraform init
terraform plan
terraform apply
```

This will output your `collector_dns_name`.

**GCP:**

```bash
terraform init
terraform plan
terraform apply
```

This will output your `collector_ip_address`, `bigquery_db_dataset_id` and `bq_loader_bad_rows_topic_name`.

**Azure:**

```bash
terraform init
terraform plan
terraform apply
```

This will output your `collector_lb_ip_address` and `collector_lb_fqdn`.

***

Make a note of the outputs: you'll need them when sending events and (in some cases) connecting to your data.

> **Tip:** Depending on your cloud and chosen destination, some of these outputs might be empty — you can ignore those.

If you have attached a custom SSL certificate and set up your own DNS records, then you don't need `collector_dns_name`, as you will use your own DNS record to send events from the Snowplow trackers.

> **Tip:** For solutions to some common Terraform errors that you might encounter when running `terraform plan` or `terraform apply`, see the [FAQs section](/docs/get-started/self-hosted/faq/#troubleshooting-terraform-errors).

## Configure the destination

**Redshift:**

No extra steps needed.

**BigQuery:**

No extra steps needed.

**Snowflake:**

No extra steps needed.

**Databricks:**

> **Info:** On Azure, we currently support loading data into Databricks via a data lake. To complete the setup, you will need to configure Databricks to access your data on ADLS.
>
> First, follow the [Databricks documentation](https://docs.databricks.com/en/storage/azure-storage.html) to set up authentication using either Azure service principal, shared access signature tokens or account keys. _(The latter mechanism is not recommended, but is arguably the easiest for testing purposes.)_
>
> You will need to know a couple of things:
>
> - Storage account name — this is the value of the `storage_account_name` variable in the pipeline `terraform.tvars` file
> - Storage container name — `lake-container`
>
> Once authentication is set up, you can create an external table using Spark SQL (replace `<storage-account-name>` with the corredponding value):
>
> ```sql
> CREATE TABLE events
> LOCATION 'abfss://lake-container@<storage-account-name>.dfs.core.windows.net/events/';
> ```

**Synapse Analytics:**

Your data is loaded into ADLS. To access it, follow [the Synapse documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format) and use the `OPENROWSET` function.

You will need to know a couple of things:

- Storage account name — this is the value of the `storage_account_name` variable in the pipeline `terraform.tvars` file
- Storage container name — `lake-container`

**Example query**

```sql
SELECT TOP 10 *
FROM OPENROWSET(
    BULK 'https://<storage-account-name>.blob.core.windows.net/lake-container/events/',
    FORMAT = 'delta'
) AS events;
```

We recommend [creating a data source](https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/query-delta-lake-format#data-source-usage), which simplifies future queries (note that unlike the previous URL, this one does not end with `/events/`):

```sql
CREATE EXTERNAL DATA SOURCE SnowplowData
WITH (LOCATION = 'https://<storage-account-name>.blob.core.windows.net/lake-container/');
```

**Example query with data source**

```sql
SELECT TOP 10 *
FROM OPENROWSET(
    BULK 'events',
    DATA_SOURCE = 'SnowplowData',
    FORMAT = 'delta'
) AS events;
```

> **Tip:** You can also consume your ADLS data via Fabric and OneLake:
>
> - First, [create a Lakehouse](https://learn.microsoft.com/en-us/fabric/onelake/create-lakehouse-onelake#create-a-lakehouse) or use an existing one.
> - Next, [create a OneLake shortcut](https://learn.microsoft.com/en-us/fabric/onelake/create-adls-shortcut) to your storage account. In the URL field, specify `https://<storage-account-name>.blob.core.windows.net/lake-container/events/`.
> - You can now [use Spark notebooks](https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-notebook-explore) to explore your Snowplow data.
>
> Do note that currently not all Fabric services support nested fields present in the Snowplow data.

***

## Configure HTTPS (optional)

Now that you have a working pipeline, you can _optionally_ configure your Collector and Iglu Server to have an HTTPS-enabled endpoint. This might be required in some cases to track events on strictly SSL-only websites, as well as to enable first-party tracking (by putting the Collector endpoint on the same sub-domain as your website).

**AWS:**

1. Navigate to Amazon Certificate Manager (ACM) in the AWS Console
2. Request a public certificate from the ACM portal for the domain you want to host these endpoints under (e.g. for the Collector this might be `c.acme.com`) - make sure you are in the _same_ region as your pipeline

- `Fully qualified domain name` will be something like `c.acme.com`
- `Validation method` is whatever works best for you - generally `DNS validation` is going to be easiest to action
- `Key algorithm` should be left as `RSA 2048`

3. Once you have requested the certificate, it should show up in the ACM certificate list as `Pending validation` - complete the DNS / Email validation steps and wait until the status changes to `Issued`
4. Copy the issued certificate’s `ARN` and paste it into your `terraform.tfvars` file under `ssl_information.certificate_arn`
5. Change `ssl_information.enabled` to `true`
6. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer
7. Add a `CNAME` DNS record for your requested domain pointing to the AWS Load balancer (e.g. `c.acme.com` -> `<lb-identifier>.<region>.elb.amazonaws.com`)

- If you are using `Route53` for DNS record management, you can instead [setup an Alias route](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html) which can help circumvent certain `CNAME` cloaking tracking protections

You should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

**GCP:**

1. Navigate to Google Certificate Manager in the GCP Console
2. Select the `Classic Certificates` tab and then press `Create SSL Certificate`

- `Name` should be something unique for the certificate you are creating
- `Description` describe the service you are creating the certificate for
- `Create mode` allows you to either supply your own certificate or create a `Google-managed` one. We generally recommend the latter for ease of implementation and updates
- `Domains` add the domain you want to host the service under (e.g. for the Collector this might be `c.acme.com`)

3. Add an `A` DNS record for the requested domain pointing to the GCP Load balancer (e.g. `c.acme.com` -> `<lb-ipv4-address>`)
4. Copy the Certificate ID into your `terraform.tfvars` file under `ssl_information.certificate_id`

- The ID takes the form `projects/{{project}}/global/sslCertificates/{{name}}`

5. Change `ssl_information.enabled` to `true`
6. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer

Once the certificate is issued, you should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

> **Info:** It’s worth noting that GCP Managed Certificates can take up to 24 hours to be provisioned successfully, so it might take a while before you can access your service over HTTPS.

**Azure:**

Unlike AWS or GCP, Azure doesn't have a Certificate Management layer. So to follow this guide, you will need to have bought your own valid SSL certificate covering the domain(s) you want to bind to.

1. [Convert your SSL Certificate](https://github.com/snowplow-devops/terraform-azurerm-lb?tab=readme-ov-file#adding-a-custom-certificate) into the correct format (`pkcs12`) needed for the Azure Application Load Balancer
2. Copy the `pkcs12` certificate password into your `terraform.tfvars` file under `ssl_information.password`
3. Copy the `pkcs12` certificate into your `terraform.tfvars` file under `ssl_information.data`

- Ensure the certificate is `base64`-encoded (e.g. `cat cert.p12 | base64`)

4. Change `ssl_information.enabled` to `true`
5. Apply the `iglu_server` / `pipeline` module as you have done previously to attach the certificate to the Load Balancer
6. Add an `A` DNS record for your requested domain pointing to the Azure Load Balancer (e.g. `c.acme.com` -> `<lb-ipv4-address>`)

You should now be able to access your service over HTTPS. Verify this by going to your newly set up endpoint in a browser — you should get a valid response with a valid SSL certificate.

***

***

If you are curious, here’s [what has been deployed](/docs/get-started/self-hosted/what-is-deployed/). Now it’s time to [send your first events to your pipeline](/docs/get-started/tracking/)!

---

# Telemetry principles in Snowplow Self-Hosted
> Learn about telemetry data collection in Snowplow Self-Hosted applications, what data is collected, and how to disable it.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/telemetry/

Telemetry helps us better understand how our applications are used:

- Which applications, clouds and warehouses are more popular than others?
- What are the most common pipeline topologies?
- Are users successful in running our stack over long periods of time?
- And so on.

This data is important for us when deciding where to invest our efforts to build a better product for our users (including you!).

## What data is collected?

In general, we track:

- Heartbeat events that tell us Snowplow applications are alive.
- Events regarding installation, startup, shutdown, etc, of our Terraform modules and applications.
- Metadata such as application version, cloud and region.

You can always disable telemetry if you prefer.

## Our principles

### Privacy

We do not automatically collect any personally identifiable information (PII) other than the IP address of the computer where a Snowplow application or a terraform module is running. This IP address is subsequently pseudonymised using SHA-256 with the [Snowplow PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/).

### Minimalism

We only ever collect what is required at any given point in time. We do not pre-empt future requirements or collect anything “just in case”. We also make sure telemetry does not affect application performance in any way.

### Transparency

Not only is our telemetry code open source (e.g. this [terraform module](https://github.com/snowplow-devops/terraform-snowplow-telemetry)), you can also inspect the schema we use for our telemetry events [here](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

## How can I help?

It helps our product development immensely if you keep telemetry enabled. We promise to keep it anonymous and as minimal as possible!

We also appreciate if you provide your email (or just a UUID) in the `user_provided_id` (or `userProvidedId`) setting. This allows us to tie events together across resources and offers a more complete picture of how the pipeline has been orchestrated. If you do provide an email address, we will only ever contact you with exciting Product & Engineering updates and Research studies. You can always exercise your right to be forgotten by [contacting us](https://snowplow.io/contact-us/).

## Which components have telemetry?

At the moment, opt-out telemetry is present in the following:

- Terraform modules for the [quick start guide](/docs/get-started/self-hosted/quick-start/).
- [Collector](/docs/api-reference/stream-collector/setup/).
- Enrich ([Enrich Kinesis](/docs/api-reference/enrichment-components/enrich-kinesis/), [Enrich PubSub](/docs/api-reference/enrichment-components/enrich-pubsub/), [Enrich Kafka](/docs/api-reference/enrichment-components/enrich-kafka/).
- RDB Loader ([Transformer Kinesis](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-kinesis/), [Transformer PubSub](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/stream-transformer/transformer-pubsub/), [Redshift Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/redshift-loader/), [Snowflake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/snowflake-loader/), [Databricks Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/loading-transformed-data/databricks-loader/)).
- Snowplow Mini for [AWS](/docs/api-reference/snowplow-mini/setup-guide-for-aws/) and [GCP](/docs/api-reference/snowplow-mini/setup-guide-for-gcp/).
- [Snowbridge](/docs/api-reference/snowbridge/getting-started/).
- [Lake loader](/docs/api-reference/loaders-storage-targets/lake-loader/).

See the telemetry notice for each component linked above for more details.

---

# Snowplow Self-Hosted upgrade guide
> Step-by-step guide to upgrading Snowplow Self-Hosted Community Edition components and Terraform modules to newer versions.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/upgrade-guide/

Now that you have successfully spun up a Snowplow pipeline, you might want to upgrade your components when there are updates available.

The Community Edition Quick Start is updated in line with the Snowplow Community Edition Distributions and here we describe the steps you need to take to upgrade between versions.

> **Note:** You may have deviated from the original Quick Start example Terraform files. If this is the case then the concepts described here still apply. However, you will want to compare the differences between Quick Start releases to see what you need to update.

## Updating your repository

To upgrade to a new Quick Start version, you first need to update the git repository to the next release. We recommend upgrading one release at a time to ease the transition rather than trying to leap from an older version to the latest, although this should also be possible with some caution.

You'll first need to commit or stash any changes you haven't commited. If you've updated your `.tfvars` then you should stash these first (use `git stash`) and once you've updated your repository, you can retrieve your changes from the stash (use `git stash pop`).

To jump to the latest release, you should update your local `main` branch to latest:

```bash
git checkout main
git pull
```

If you only want to move to a specific version then you should checkout that tag into a new branch.

```bash
git checkout -b 21.08 21.08 // To checkout 21.08
```

## Migrations

> **Info:** This information is currently specific to AWS.

Before updating your stack, you should run any required manual migrations. These are listed below.

If the specific upgrade path isn't mentioned below, then you should be able to upgrade without any manual interventation.

Clicking the links below will show all the Terraform differences between each version.

### 22.01 to 22.01-Patch.1

#### Changing the name of the `pipeline_db*` variables

Since we have two destination database options (`postgres` and `snowflake`) starting with this release, we've replaced 'pipeline\_db' part in the variable names with 'postgres\_db'. You need to change your variable names accordingly.

Also, new 'pipeline\_db' variable is introduced. This variable specifies the database that will be used as destination in the pipeline. It can be either 'postgres' or 'snowflake'. Since all existing pipelines are using Postgres if they aren't modified, this variable should be set to 'postgres' in those pipelines.

### 21.04 to 21.08

#### Changing some of the column types

The database schemas created with a 21.04 Quick Start do not work well with events enriched with pii pseudoanonymization. Therefore, a Postgres migration is required to change types of some of the columns. You need to apply [this migration script](https://github.com/snowplow-incubator/snowplow-postgres-loader/blob/master/migrations/0-3-0.sql) to your database (You may need to change the `public` schema to `atomic` for this to run successfully with the standard Quick Start installation).

#### Cloudwatch logs are on by default

21.08 Quick Start has enabled Cloud Watch logs by default. We recommend you leave this enabled as the cloud watch logs can be beneficial when diagnosing any issues with your pipeline components.

## Updating Snowplow components

If you are using a standard Quick Start deployment, you can follow the steps below to upgrade your Iglu Server and Pipeline. If you have deviated from the example, then you should compare the differences between the Snowplow Community Edition Distributions in the examples and update your usage of the Terraform modules accordingly.

> **Note:** You will experience some brief downtime when running these upgrades, as there is no redudency in the Community Edition.

#### Iglu Server

Now you are ready to update your Iglu Server. The example below is for the `default` configuration on `aws`. You should change to the correct folder for your preferred installation.

```bash
cd terraform/aws/iglu_server/default
terraform init
terraform plan
```

At this stage, you should see the changes that will be made to your infrastructure. If everything looks ok and there are no errors you can apply the update.

```bash
terraform apply
```

#### Pipeline

Now you are ready to update your pipeline components. The example below is for the `default` configuration on `aws`. You should change to the correct folder for your preferred installation.

```bash
cd terraform/aws/pipeline/default
terraform init
terraform plan
```

At this stage, you should see the changes that will be made to your infrastructure. If everything looks ok and there are no errors you can apply the update.

```bash
terraform apply
```

---

# What is deployed in the Self-Hosted Quick Start?
> Overview of Snowplow pipeline infrastructure components deployed by Terraform including collector, enrichment, and loader modules.
> Source: https://docs.snowplow.io/docs/get-started/self-hosted/what-is-deployed/

Let’s take a look at what is deployed when you follow the quick start guide.

> **Tip:** You can very easily edit the script or run each of the Terraform modules independently, giving you the flexibility to design the topology of your pipeline according to your needs.

## Overview

**AWS:**

**Redshift:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Redshift Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Redshift)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Redshift Loader

For more information about the Redshift Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?warehouse=redshift).

**Snowflake:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Snowflake Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Snowflake)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Snowflake Loader

For more information about the Snowflake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/).

**Databricks:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(EC2)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kinesis app</i>
(EC2)"}}
iglu{{"<b>Iglu Server</b>
(EC2)"}}
igludb[("<b>Iglu Database</b>
(RDS)")]
bad[["<b>Bad Stream</b>
(Kinesis)"]]
loader("<b>Databricks Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(Databricks)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kinesis)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kinesis)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Archival and failed events

```mermaid
flowchart LR
      good[["<b>Enriched Stream</b>
(Kinesis)"]]
        blobloadergood{{"<b>S3 Loader Good</b>
(EC2)"}}
        blobgood[("<b>Enriched Events</b>
(S3)")]
        good-->blobloadergood-->blobgood
        bad[["<b>Bad Stream</b>
(Kinesis)"]]
        blobloaderbad{{"<b>S3 Loader Bad</b>
(EC2)"}}
        blobbad[("<b>Failed Events</b>
(S3)")]
        bad-->blobloaderbad-->blobbad
```

#### Databricks Loader

For more information about the Databricks Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?warehouse=databricks).

***

**GCP:**

**BigQuery:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(CE)"}}
enrich{{"<b>Enrich</b>
<i>enrich-pub/sub app</i>
(CE)"}}
iglu{{"<b>Iglu Server</b>
(CE)"}}
igludb[("<b>Iglu Database</b>
(CloudSQL)")]
bad[["<b>Bad Stream</b>
(Pub/Sub)"]]
loader("<b>BigQuery Loader</b>
<i>(see below)</i>")
atomic[("<b>Events</b>
(BigQuery)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Pub/Sub)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Pub/Sub)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### BigQuery Loader

For more information about the BigQuery Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/bigquery-loader/).

***

**Azure:**

**Databricks:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(VMSS)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kafka app</i>
(VMSS)"}}
iglu{{"<b>Iglu Server</b>
(VMSS)"}}
igludb[("<b>Iglu Database</b>
(Postgres)")]
bad[["<b>Bad Stream</b>
(Kafka)"]]
loader{{"<b>Data Lake Loader</b>
(VMSS)"}}
atomic[("<b>Events</b>
(ADLS Gen2)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kafka)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kafka)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Data Lake Loader

For more information about the Data Lake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/lake-loader/).

**Synapse Analytics:**

#### The main components of the pipeline

```mermaid
flowchart LR
collect{{"<b>Collector</b>
<i>stream-collector app</i>
(VMSS)"}}
enrich{{"<b>Enrich</b>
<i>enrich-kafka app</i>
(VMSS)"}}
iglu{{"<b>Iglu Server</b>
(VMSS)"}}
igludb[("<b>Iglu Database</b>
(Postgres)")]
bad[["<b>Bad Stream</b>
(Kafka)"]]
loader{{"<b>Data Lake Loader</b>
(VMSS)"}}
atomic[("<b>Events</b>
(ADLS Gen2)")]
collect---iglu
%% above line is an invisible link for alignment
enrich-.-oiglu<-.->igludb
collect-->|"<b>Raw Stream</b><br/>(Kafka)"| enrich
enrich-->|"<b>Enriched Stream</b><br/>(Kafka)"| loader-->atomic
collect & enrich -.->|<i>events that<br/>fail validation</i>| bad
loader -.->|<i>events that<br/>fail loading</i>| bad
linkStyle 0 stroke:none,fill:none
```

#### Data Lake Loader

For more information about the Data Lake Loader, see the [loader documentation](/docs/api-reference/loaders-storage-targets/lake-loader/).

***

***

## Collector load balancer

This is an application load balancer for your inbound HTTP(S) traffic. Traffic is routed from the load balancer to the Collector instances.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/alb/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/lb/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/lb/azurerm/latest)).

## Collector

This is an application that receives raw Snowplow events over HTTP(S), serializes them, and then writes them to Kinesis (on AWS), Pub/Sub (on GCP) or Kafka / Event Hubs (on Azure). More details can be found [here](/docs/api-reference/stream-collector/).

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/collector-kinesis-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/collector-pubsub-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/collector-event-hub-vmss/azurerm/latest)).

## Enrich

This is an application that reads the raw Snowplow events, validates them (including validation against [schemas](/docs/fundamentals/schemas/)), [enriches](/docs/pipeline/enrichments/) them and writes the enriched events to another stream. More details can be found [here](/docs/api-reference/enrichment-components/).

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/enrich-kinesis-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/enrich-pubsub-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/enrich-event-hub-vmss/azurerm/latest)).

## Iglu

The Iglu stack allows you to manage [schemas](/docs/fundamentals/schemas/).

### Iglu load balancer

This load balances the inbound traffic and routes traffic to the Iglu Server.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/alb/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/lb/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/lb/azurerm/latest)).

### Iglu Server

The [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) serves requests for Iglu schemas stored in your schema registry.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ec2/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/iglu-server-ce/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/iglu-server-vmss/azurerm/latest)).

### Iglu database

This is the Iglu Server database (RDS on AWS, CloudSQL on GCP and PostgreSQL on Azure) where the Iglu [schemas](/docs/fundamentals/schemas/) themselves are stored.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/rds/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/cloud-sql/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/postgresql-server/azurerm/latest)).

## Streams

The various streams (Kinesis on AWS, Pub/Sub on GCP and Kafka / Event Hubs on Azure) are a key component of ensuring a non-lossy pipeline, providing crucial back-up, as well as serving as a mechanism to drive real time use cases from the enriched stream.

For further details on the resources, default and required input variables, and outputs, see the Terraform module ([AWS](https://registry.terraform.io/modules/snowplow-devops/kinesis-stream/aws/latest), [GCP](https://registry.terraform.io/modules/snowplow-devops/pubsub-topic/google/latest), [Azure](https://registry.terraform.io/modules/snowplow-devops/event-hub-namespace/azurerm/latest)).

**AWS only — DynamoDB**

On the first run of each of the applications consuming from Kinesis (e.g. Enrich), the Kinesis Connectors Library creates a DynamoDB table to keep track of what they have consumed from the stream so far. Each Kinesis consumer maintains its own checkpoint information.

The DynamoDB autoscaling module enables autoscaling for a target DynamoDB table. Note that there is a `kcl_write_max_capacity` variable which can be set to your expected RPS, but setting it high will of course incur more cost.

You can find further details in the [DynamoDB Terraform module](https://registry.terraform.io/modules/snowplow-devops/dynamodb-autoscaling/aws/latest).

### Raw stream

Collector payloads are written to the raw stream, before being picked up by the Enrich application.

### Enriched stream

Events that have been validated and enriched by the Enrich application are written to the enriched stream. Depending on your cloud and destination, different loaders pick up the data from this stream, as shown on the diagram [above](#overview).

> **Note:** The S3 loader (enriched) also reads from this enriched stream and writes to the enriched folder on S3.

### Bad 1 stream

This bad stream is for [failed events](/docs/fundamentals/failed-events/), which get created when the Collector, Enrich or various loader applications fail to process the data.

### Other streams

**AWS:**

The _Bad 2 stream_ is for failed events generated by the S3 loader as it tries to write from the Bad 1 stream to the bad folder on S3.

If you selected Redshift, Snowflake or Databricks as your destination, the loader will use an extra SQS stream internally as explained in the [reference documentation](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/?cloud=aws).

**GCP:**

If you selected BigQuery as your destination, the _Bad Rows_ stream will contain events that could not be inserted into BigQuery by the loader. This includes data that is not valid against its schema or that is somehow corrupted in a way that the loader cannot handle. In addition, the loader users a few streams internally, as explained in the [reference documentation](/docs/api-reference/loaders-storage-targets/bigquery-loader/previous-versions/bigquery-loader-1.x/).

**Azure:**

No other streams.

***

## Archival and failed events

**AWS:**

Aside from your main destination, the data is written to S3 by the S3 Loader applications for archival and to deal with [failed events](/docs/fundamentals/failed-events/).

See the [S3 Loader](https://registry.terraform.io/modules/snowplow-devops/s3-loader-kinesis-ec2/aws/latest) and [S3](https://registry.terraform.io/modules/snowplow-devops/s3-bucket/aws/latest) Terraform modules for further details on the resources, default and required input variables, and outputs.

The following loaders and folders are available:

- Enriched loader, `enriched/`: enriched events, in GZipped blobs of [enriched TSV](/docs/pipeline/enriched-tsv-format/). Historically, this has been used as the staging ground for loading into data warehouses via the [Batch transformer](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/spark-transformer/) application. However, it’s no longer used in the quick start examples.
- Bad loader, `bad/`: [failed events](/docs/fundamentals/failed-events/). You can [query them using Athena](/docs/monitoring/exploring-failed-events/file-storage/).

**GCP:**

To store all failed events, you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) application.

**Azure:**

Currently, [failed events](/docs/fundamentals/failed-events/) are only available in the `bad` Kafka / Event Hubs topic.

***

## Loaders

**Redshift:**

[RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is a set of applications that loads enriched events into Redshift.

See the following Terraform modules for further details on the resources, default and required input variables, and outputs:

- [Transformer Kinesis](https://registry.terraform.io/modules/snowplow-devops/transformer-kinesis-ec2/aws/latest)
- [Redshift Loader](https://registry.terraform.io/modules/snowplow-devops/redshift-loader-ec2/aws/latest)

**BigQuery:**

The [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) is a set of applications that loads enriched events into BigQuery.

See the [Terraform module](https://registry.terraform.io/modules/snowplow-devops/bigquery-loader-pubsub-ce/google/latest) for further details on the resources, default and required input variables, and outputs.

There will be a new dataset available in BigQuery with the suffix `_snowplow_db`. Within this dataset, there will be a table called `events` — all of your collected events will be available here generally within a few seconds after they are sent into the pipeline.

**Snowflake:**

[Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) is an application that loads data into Snowflake.

See the following Terraform module for further details on the resources, default and required input variables, and outputs:

- Snowflake Streaming Loader ([AWS](https://registry.terraform.io/modules/snowplow-devops/snowflake-streaming-loader-ec2/aws/latest))

**Databricks (direct):**

[RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is a set of applications that loads enriched events into Databricks.

See the following Terraform modules for further details on the resources, default and required input variables, and outputs:

- [Transformer Kinesis](https://registry.terraform.io/modules/snowplow-devops/transformer-kinesis-ec2/aws/latest)
- [Databricks Loader](https://registry.terraform.io/modules/snowplow-devops/databricks-loader-ec2/aws/latest)

**Databricks (via lake):**

[Lake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is an application that loads enriched events into a data lake so that they can be queried via Databricks (or other means).

See the Lake Loader [Terraform module](https://registry.terraform.io/modules/snowplow-devops/lake-loader-vmss/azurerm/latest) for further details on the resources, default and required input variables, and outputs.

The Terraform stack for the pipeline will deploy a storage account and a storage container where the loader will write the data.

**Synapse Analytics:**

[Lake Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) is an application that loads enriched events into a data lake so that they can be queried via Synapse Analytics (or Fabric, OneLake, etc).

See the Lake Loader [Terraform module](https://registry.terraform.io/modules/snowplow-devops/lake-loader-vmss/azurerm/latest) for further details on the resources, default and required input variables, and outputs.

The Terraform stack for the pipeline will deploy a storage account and a storage container where the loader will write the data.

***

---

# Strong cookies and ad-blockers
> Configure first-party cookies and ad-blocker resilient tracking with Cookie Extension service and custom collector domains.
> Source: https://docs.snowplow.io/docs/get-started/tracking/cookies-and-ad-blockers/

This page details a few recommendations to set up your Snowplow tracking. Their goal is to make sure that:

1. Your Web app has strong cookies to support persisting the Snowplow user and session identifiers.
2. The Snowplow tracking is resilient to ad-blockers.

## 1. Strong cookies for user and session identifiers

**Strategy 1: First party cookies**

As browsers [cut down on third-party cookies](https://snowplow.io/blog/privacy-updates-ad-blockers-and-first-party-tracking/), they are becoming less and less useful for storing user and session identifiers. The alternative is to use first-party cookies. Although first-party cookies don't allow sharing user and session identifiers across domains (they do let you share them across sub-domains), there are other alternative solutions for [cross-domain tracking](/docs/sources/web-trackers/cross-domain-tracking/) that can be used in conjunction with first-party cookies.

Using Snowplow, you can set up your collector to be on the same domain as the website, which is the requirement for the use of first-party cookies. To learn how to set up the collector domain, [visit this page](/docs/sources/first-party-tracking/).

**Strategy 2: Snowplow Cookie Lifetime Extension service for Safari ITP**

As of Safari 16.4 released in April 2023, Safari sets the [lifetime of server-set cookies](https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense) to a maximum of 7 days under some circumstances even if they are first-party cookies. This greatly limits the effectiveness of tracking a customer journey where users are not regularly returning to your website. In particular, it affects the `network_userid` identifier in Snowplow events.

Snowplow provides the Cookie Lifetime Extension service solution that fully mitigates the impact of this change. Visit the [documentation for the Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) to learn more.

## 2. Mitigating the impact of ad-blockers

**Strategy 1: Custom POST path on the collector**

The recommended method for sending events to the Snowplow collector is using POST requests. These requests are by default made to the `/com.snowplowanalytics.snowplow/tp2` path on the collector. As many ad blockers will block this path, it is recommended to change the path.

To change the path, you will need to take the following steps (in this order):

1. Update the [Collector configuration](/docs/pipeline/collector/) to support the new path _before_ you send events to it with this setting.
2. Provide the new path in the [`customPostPath` configuration in the tracker](/docs/sources/web-trackers/configuring-how-events-sent/#custom-post-path).

**Strategy 2: Rename sp.js file**

In case you are using the Snowplow JavaScript tracker using the `sp.js` file that you include using an HTML tag, it is advisable to rename this file to reduce the chance of ad-blockers preventing the script from loading. You may rename the file to a random 8 character string, see the [recommendations here](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/self-hosting-the-javascript-tracker-aws/).

In case you have a JavaScript app (e.g., built using React, Angular, Vue.js), consider installing the tracker using the NPM package. This approach is more resilient against ad-blockers and advisable for single page apps. See the [installation instructions here](/docs/sources/web-trackers/tracker-setup/).

**Strategy 3: Server-side tracking**

Finally, the most robust solution against the impact of ad-blockers and other browser privacy measures is the use of server-side tracking. Through server-side tracking, you will track events directly from your server app (e.g., in Python, Node.js, Ruby). Server-side tracking works within first party context allowing you to build on top of the first-party cookies for user identification.

Snowplow provides tracker SDKs for all commonly used programming languages used in server-side apps (see the [full list here](/docs/sources/)). To learn more about server-side tracking using Snowplow, [visit this blog post](https://snowplow.io/blog/server-side-tracking-vs-client-side-tracking/).

---

# Tracking your first events
> Tracking your first Snowplow events
> Source: https://docs.snowplow.io/docs/get-started/tracking/

Once your pipeline is set up, you will want to send some events to it. Here's an overview of the different options.

> **Tip:** Regardless of how you send the events, it might take a few minutes for them to reach your destination (e.g. data warehouse). This depends on which [destination and loader](/docs/destinations/warehouses-lakes/) you have configured.

## A quick test

If you are eager to look at some Snowplow events or just test your pipeline, you can use the box below to send some data (including [failed events](/docs/fundamentals/failed-events/)) to your Collector.

> **Tip:** Note that to use this tool, you need a Collector URL that starts with `https://`, not `http://`. This is because web browsers block traffic from HTTPS-enabled sites (such as `https://docs.snowplow.io` — this site — to non-HTTPS resources).

**CDI Private Managed Cloud:**

You can find the Collector URL (Collector endpoint) in [Console](https://console.snowplowanalytics.com/environments).

**CDI Cloud:**

You can find the Collector URL (Collector endpoint) in [Console](https://console.snowplowanalytics.com/environments).

**Snowplow Self-Hosted:**

Input the Collector URL you chose when deploying your Snowplow Self-Hosted pipeline.

If you have not yet configured an SSL certificate and a custom domain name for your Collector, you can use `http://<collector_dns_name>` (`http`, not `https`), where `collector_dns_name` is the output of the pipeline Terraform module.

***

**Details**

Which events are sent?

We use the following tracking code:

```javascript
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.snowplowanalytics.snowplow/add_to_cart/jsonschema/1-0-0',
      data: { sku: 'ASO01043', unitPrice: 49.95, quantity: 1000 }
    }
  }
)
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
      data: {
        targetUrl: 'https://snowplow.io'
      }
    }
  }
)
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.snowplowanalytics.mobile/deep_link/jsonschema/1-0-0',
      data: {
        url: 'https://snowplowanalytics.com'
      }
    }
  }
)
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
      data: {
        targetUrl: 'https://snowplow.io'
      }
    },
    context: [
      {
        schema: 'iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0',
        data: {
          id: '6a2a8c7c-8e7a-4e9f-9b4c-0f4f9a3f35a9'
        }
      }
    ]
  }
)
window.snowplow('trackPageView', undefined)
window.snowplow('trackPageView', { title: 'My custom page title' })

// Bad Events
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.fake_event/event/jsonschema/1-0-0',
      data: { foo: 'bar' }
    }
  }
)
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: '',
      data: {}
    }
  }
)
window.snowplow('trackSelfDescribingEvent',
  {
    event: {
      schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
      data: {
        invalidProperty: "This schema doesn't have this property"
      }
    }
  }
)
```

> **Tip:** An application ID is sent along with each Snowplow event. You can pick a value that will make it easy to filter out these test events from your data later, e.g. like this:
>
> ```sql
> ...
> WHERE app_id != 'test'
> ...
> ```

Now, let’s take a look at how to set up actual event tracking.

## Using the JavaScript tracker

The [JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/) is our most commonly used tracker. It’s a good choice for websites, and the installation process is similar to other tools like Google Analytics.

To use the JavaScript tracker on your site, you will need to obtain a code snippet first.

**CDI Private Managed Cloud:**

CDI Private Managed Cloud can automatically generate the snippet for you. Go to the [tag generator](https://console.snowplowanalytics.com/tag-generator) screen, fill in the necessary parameters, and copy the snippet at the bottom.

**CDI Cloud:**

You can find the pre-generated snippet in the [Getting started](https://console.snowplowanalytics.com/) section.

**Snowplow Self-Hosted:**

Take note of the Collector URL you’ve chosen when deploying your Snowplow Self-Hosted pipeline.

If you have not yet configured an SSL certificate and a custom domain name for your Collector, you can use `http://<collector_dns_name>` (`http`, not `https`), where `collector_dns_name` is the output of the pipeline Terraform module.

Then, follow the JavaScript tracker [quick start guide](/docs/sources/web-trackers/quick-start-guide/) to create your snippet.

***

Once you have the snippet, there are two common ways to deploy it.

**Editing your website directly:**

If you have access to the source code of your site, paste the snippet into the `<head>` HTML section and deploy the changes.

**Using Google Tag Manager:**

If you are already using Google Tag Manager to add various code snippets to your site, you can add your Snowplow snippet there.

![google-tag-manager-setup](/assets/images/gtm-01e0ff314a61e447de24b8aa69cf7ee7.gif)

- Navigate to the Google Tag Manager account you wish to add tracking to
- Create a new Custom HTML tag and paste the Javascript snippet into the tag
- Set it to fire on 'All Pages' or a trigger of your choosing
- You can preview your tag to send some events before publishing it

***

The JavaScript tracker captures many events (e.g. page views) automatically, so you should start accumulating your first events as soon as the changes are rolled out to your users.

> **Tip:** To make sure that your tracking is well configured for strong first-party cookies and resilient against the impact of ad-blockers, [visit this page with several useful recommendations](/docs/get-started/tracking/cookies-and-ad-blockers/).

## Using other trackers

We have many different trackers in different programming languages, in case the JavaScript tracker is not a fit for you. For example, see the [mobile native trackers](/docs/sources/mobile-trackers/) or the [full list](/docs/sources/) of what’s available.

> **Tip:** For quick testing, you might be tempted to send data to your Collector URL using a basic command-line tool like `cURL`. However, you would need to ensure that the data format follows our [tracker protocol](/docs/events/). Instead, take a look at the [command-line tracker](/docs/sources/snowplow-tracking-cli/) that will do this for you.

---

# Glossary
> Comprehensive glossary of Snowplow terms covering components, concepts, legal terms, and product offerings.
> Source: https://docs.snowplow.io/docs/glossary/

This page provides a glossary of terms used by Snowplow.

To help clarify, we've categorized the terms:

- **Component**: a pipeline application, package, SDK, or other code-based tool or set of tools, e.g. [Snowplow Micro](/docs/testing/snowplow-micro/)
- **Concept**: names for Snowplow things and ideas, e.g. [tracking plan](/docs/fundamentals/tracking-plans/)
- **Legal**: about licensing, e.g. [SLULA](/docs/licensing/)
- **Offering**: a Snowplow product that you can buy, e.g. [Snowplow CDI Cloud](/docs/get-started/) or [Snowplow Signals](/docs/signals/introduction/)

| Term                                   | Category  | Description                                                                                                                                                           | More information                                                                           |
| -------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **Atomic events table**                | Concept   | Core warehouse table containing all Snowplow events                                                                                                                   | [Atomic events table](/docs/fundamentals/warehouse-tables/)                                |
| **Atomic fields**                      | Concept   | Core event properties defined by the Snowplow atomic schema                                                                                                           | [Out-of-the-box fields](/docs/fundamentals/canonical-event/#common-fields)                 |
| **Attribute**                          | Concept   | Individual data point or characteristic calculated by Signals                                                                                                         | [Attributes](/docs/signals/concepts/#types-of-attribute)                                   |
| **Attribute group**                    | Concept   | Versioned collection of related attributes that defines how Signals should calculate behavioral data                                                                  | [Attribute groups](/docs/signals/concepts/#attribute-groups)                               |
| **Attribute key**                      | Concept   | Event field that defines the context in which attributes are calculated by Signals                                                                                    | [Attribute groups](/docs/signals/concepts/#attribute-keys)                                 |
| **Audience Hub**                       | Offering  | Customer segmentation in warehouse, powered by Census                                                                                                                 |                                                                                            |
| **Collection**                         | Concept   | Generation of event data and sending to an event Collector endpoint                                                                                                   | [Sources](/docs/sources/)                                                                  |
| **Collector**                          | Component | Application that receives and saves raw events, and forwards them to Enrich                                                                                           | [Collector](/docs/pipeline/collector/)                                                     |
| **Context**                            | Concept   | Deprecated name for entity                                                                                                                                            | [Entities](/docs/fundamentals/entities/)                                                   |
| **Cookie Extension service**           | Concept   | Process for extending cookie lifetimes                                                                                                                                | [Cookie Extension](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) |
| **Data contract**                      | Concept   | Event definitions; set of tracking plans                                                                                                                              | [Event Studio](/docs/event-studio/)                                                        |
| **Data model**                         | Component | dbt SQL package for processing events in the warehouse                                                                                                                | [Modeling your data](/docs/modeling-your-data/)                                            |
| **Data pipeline**                      | Concept   | Set of applications for collecting, processing, and storing events                                                                                                    | [Fundamentals](/docs/fundamentals/)                                                        |
| **Data product**                       | Concept   | Deprecated name for tracking plan                                                                                                                                     | [Tracking plans](/docs/event-studio/tracking-plans/)                                       |
| **Data Product Studio**                | Offering  | Former name for Event Studio                                                                                                                                          | [Event Studio](/docs/event-studio/)                                                        |
| **Data quality**                       | Concept   | Tooling and reporting related to failed events and QA                                                                                                                 | [Data quality](/docs/testing/)                                                             |
| **Data structure**                     | Concept   | Schema, plus metadata                                                                                                                                                 | [Data structures](/docs/event-studio/data-structures/)                                     |
| **Derived events table**               | Concept   | Warehouse table of events generated by a data model                                                                                                                   | [Modeling your data with dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/)       |
| **Destination**                        | Concept   | Third-party data store or data activation platform that can receive Snowplow event data                                                                               | [Destinations](/docs/fundamentals/destinations/)                                           |
| **Enrich**                             | Component | Application that validates and enriches events                                                                                                                        | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Enrichment**                         | Component | A process, run by Enrich, that adds to or modifies event data                                                                                                         | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Entity**                             | Concept   | Supplemental or additional data included in an event; a schema or data structure describing additional data                                                           | [Entities](/docs/fundamentals/entities/)                                                   |
| **Event**                              | Concept   | A payload, processed by a data pipeline, describing something that happened; the "main" schema or data structure (i.e. not an entity) in an event                     | [Events](/docs/fundamentals/events/)                                                       |
| **Event specification**                | Concept   | Set of data structures, including event and entities, that an event payload should match                                                                              | [Event specifications](/docs/event-studio/tracking-plans/event-specifications/)            |
| **Event Studio**                       | Offering  | Tracking and data contracts tooling                                                                                                                                   | [Event Studio](/docs/event-studio/)                                                        |
| **Event, canonical**                   | Concept   | One of the original Snowplow events - page view, page ping, or the legacy ecommerce transaction events - that has contains data for some atomic fields, but no schema | [Canonical events](/docs/fundamentals/canonical-event/)                                    |
| **Event, enriched**                    | Concept   | An event payload that's valid and has been successfully enriched                                                                                                      | [Enrichments](/docs/pipeline/enrichments/)                                                 |
| **Event, failed**                      | Concept   | An event payload that the pipeline had some problem processing                                                                                                        | [Failed events](/docs/fundamentals/failed-events/)                                         |
| **Event, self-describing**             | Concept   | Custom event, based on a custom schema                                                                                                                                | [Self-describing events](/docs/fundamentals/events/#self-describing-events)                |
| **Event, structured**                  | Concept   | Legacy, proto-custom event                                                                                                                                            | [Structured events](/docs/fundamentals/canonical-event/#structured-events)                 |
| **Event, unstructured**                | Concept   | Deprecated name for self-describing event                                                                                                                             | [Self-describing events](/docs/fundamentals/events/#self-describing-events)                |
| **Event, valid**                       | Concept   | An event payload that matches its schema definitions; an event that the pipeline has collected, validated, and enriched without failure (i.e. not a failed event)     | [Failed events](/docs/fundamentals/failed-events/)                                         |
| **Extension**                          | Offering  | Third-party integration                                                                                                                                               |                                                                                            |
| **ID service**                         | Concept   | Deprecated name for Cookie Extension service                                                                                                                          | [Cookie Extension](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) |
| **Iglu**                               | Component | Schema registry and associated tooling                                                                                                                                | [Iglu](/docs/api-reference/iglu/)                                                          |
| **Intervention**                       | Concept   | Signals user behavior trigger                                                                                                                                         | [Interventions](/docs/signals/concepts/#interventions)                                     |
| **License, Community**                 | Legal     | Deprecated license for certain Snowplow components; SCL                                                                                                               | [Copyright licenses](/docs/licensing/)                                                     |
| **License, Limited Use**               | Legal     | Non-production use license for certain Snowplow components; SLULA                                                                                                     | [Copyright licenses](/docs/licensing/)                                                     |
| **License, Personal and Academic**     | Legal     | Personal and academic license for certain Snowplow components; SPAL                                                                                                   | [Copyright licenses](/docs/licensing/)                                                     |
| **Loader**                             | Component | Application that loads data into warehouse or lake storage                                                                                                            | [Warehouses and lakes](/docs/destinations/warehouses-lakes/)                               |
| **Organization**                       | Concept   | Your organization, in the context of account management and SSO                                                                                                       | [Account management](/docs/account-management/)                                            |
| **Out-of-the-box**                     | Concept   | Non-custom events, tracking plans, schemas, etc. that are included with Snowplow                                                                                      |                                                                                            |
| **Profiles Store**                     | Component | Centralized Signals database that stores calculated attribute values                                                                                                  | [Profiles Store](/docs/signals/concepts/#profiles-store)                                   |
| **Registry**                           | Concept   | Server that stores schemas (schema repository)                                                                                                                        | [Iglu repositories](/docs/api-reference/iglu/iglu-repositories/)                           |
| **Reverse ETL**                        | Offering  | Sync insights from warehouse to other tools, powered by Census                                                                                                        | [Reverse ETL](/docs/destinations/reverse-etl/)                                             |
| **Schema**                             | Concept   | JSON Schema that describes how data should look                                                                                                                       | [Schemas](/docs/fundamentals/schemas/)                                                     |
| **Schema respository**                 | Concept   | Server that stores schemas (schema registry)                                                                                                                          | [Iglu repositories](/docs/api-reference/iglu/iglu-repositories/)                           |
| **Service**                            | Concept   | Attribute consumption interface layer for Signals                                                                                                                     | [Services](/docs/signals/concepts/#services)                                               |
| **Signals**                            | Offering  | Real-time customer intelligence platform                                                                                                                              | [Signals](/docs/signals/)                                                                  |
| **SLULA**                              | Legal     | Snowplow Limited Use License Agreement, a non-production use license                                                                                                  | [Copyright licenses](/docs/licensing/)                                                     |
| **Snowbridge**                         | Component | Application that sends data streams into external destinations                                                                                                        | [Snowbridge](/docs/api-reference/snowbridge/)                                              |
| **Snowplow BDP**                       | Offering  | Deprecated name for Snowplow CDI                                                                                                                                      | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow BDP Enterprise**            | Offering  | Deprecated name for Snowplow CDI Private Managed Cloud                                                                                                                | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI**                       | Offering  | Full Snowplow product offering                                                                                                                                        | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI Cloud**                 | Offering  | Snowplow running in a cloud hosted by Snowplow                                                                                                                        | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CDI Private Managed Cloud** | Offering  | Snowplow running in your own cloud                                                                                                                                    | [Snowplow CDI](/docs/get-started/#customer-data-infrastructure)                            |
| **Snowplow CLI**                       | Component | Tool for command-line management of tracking plans and data structures                                                                                                | [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/)                   |
| **Snowplow Community Edition**         | Offering  | Limited license (SLULA) Snowplow self-hosted product offering                                                                                                         | [Snowplow Self-Hosted](/docs/get-started/#self-hosted)                                     |
| **Snowplow Console**                   | Offering  | Web UI for Snowplow data management                                                                                                                                   | [Snowplow Console](https://console.snowplowanalytics.com)                                  |
| **Snowplow Inspector**                 | Component | Browser extension for testing and validation of web tracking                                                                                                          | [Snowplow Inspector](/docs/testing/snowplow-inspector/)                                    |
| **Snowplow Micro**                     | Component | Testing and QA pipeline                                                                                                                                               | [Snowplow Micro](/docs/testing/snowplow-micro/)                                            |
| **Snowplow Mini**                      | Component | Legacy testing and QA pipeline (see Snowplow Micro)                                                                                                                   | [Snowplow Mini](/docs/api-reference/snowplow-mini/)                                        |
| **Snowplow Self-Hosted**               | Offering  | Production self-hosted Snowplow product offering                                                                                                                      | [Snowplow Self-Hosted](/docs/get-started/#self-hosted)                                     |
| **Snowtype**                           | Component | Tool for custom tracking code generation                                                                                                                              | [Snowtype](/docs/event-studio/implement-tracking/snowtype/)                                |
| **Snowtype CLI**                       | Component | Tool for command-line management of Snowtype code generation                                                                                                          | [Snowtype CLI](/docs/event-studio/implement-tracking/snowtype/using-the-cli/)              |
| **Source**                             | Concept   | Tracker SDK or webhook that sends events to a Collector endpoint                                                                                                      | [Sources](/docs/sources/)                                                                  |
| **Source application**                 | Concept   | Your application where you have implemented tracking                                                                                                                  | [Source applications](/docs/event-studio/source-applications/)                             |
| **Tracker**                            | Component | SDK that sends events to an event Collector endpoint                                                                                                                  | [Trackers](/docs/sources/)                                                                 |
| **Tracking catalog**                   | Concept   | Deprecated list of received event types in Snowplow Console                                                                                                           |                                                                                            |
| **Tracking plan**                      | Concept   | Set of event specifications, plus metadata                                                                                                                            | [Tracking plans](/docs/event-studio/tracking-plans/)                                       |
| **TSV**                                | Concept   | Tab-separated values format, used for events within a Snowplow pipeline                                                                                               | [Enriched TSV format](/docs/pipeline/enriched-tsv-format/#tsv-header)                      |
| **Validation**                         | Concept   | Pipeline process of checking events against their schemas                                                                                                             |                                                                                            |
| **Visualization**                      | Offering  | Accompanying dashboard for a data model                                                                                                                               | [Visualizations](/docs/modeling-your-data/visualization/)                                  |
| **Webhook**                            | Component | API that allows you to send external data to Snowplow pipelines, usually third-party                                                                                  | [Webhooks](/docs/sources/webhooks/)                                                        |
| **Workspace**                          | Concept   | Isolated environment containing one production pipeline, and potentially multiple QA or development pipelines                                                         |                                                                                            |

---

# Snowplow Documentation
> Snowplow CDI delivers validated, enriched event-level data to your warehouse, lake, or stream in real time. Define custom events with flexible schemas and distinguish AI agent traffic from human visitors.
> Source: https://docs.snowplow.io/docs/

**Deliver validated, enriched event data to your warehouse, lake, or stream in real time with Snowplow**.

[Get started with Snowplow CDI](/docs/get-started/)

[Your data foundation, deployed in your cloud or ours](/docs/get-started/)

[Snowplow Signals](/docs/signals/introduction/)

[Real-time customer context for AI agents and personalized experiences](/docs/signals/introduction/)

## Explore Snowplow Customer Data Infrastructure

Fundamentals

Learn about core Snowplow concepts

[Events](/docs/fundamentals/events/) [Entities](/docs/fundamentals/entities/) [Schemas](/docs/fundamentals/schemas/) [Atomic event fields reference](/docs/fundamentals/canonical-event/)

[Learn more →](/docs/fundamentals/)

Event collection

Manage your tracking

[Tracking plans](/docs/event-studio/tracking-plans/) [Source applications](/docs/event-studio/source-applications/) [Create a data structure](/docs/event-studio/data-structures/) [Snowtype](/docs/event-studio/implement-tracking/snowtype/)

[Learn more →](/docs/event-studio/)

Destinations

Send data to warehouses, lakes, and third-party tools

[Warehouse and lake destinations](/docs/destinations/warehouses-lakes/) [Event forwarding](/docs/destinations/forwarding-events/) [How to query Snowplow data](/docs/destinations/warehouses-lakes/querying-data/) [Google Tag Manager Server-Side](/docs/destinations/forwarding-events/google-tag-manager-server-side/)

[Learn more →](/docs/destinations/)

Trackers

Instrument 20+ SDKs for web, mobile, and servers

[Get started on web](/docs/sources/web-trackers/tracker-setup/) [Track page views on web](/docs/sources/web-trackers/tracking-events/page-views/) [Get started on mobile](/docs/sources/mobile-trackers/installation-and-set-up/) [Track screen views on mobile](/docs/sources/mobile-trackers/tracking-events/screen-tracking/)

[Learn more →](/docs/sources/)

Pipeline

Configure your Collector and enrichments

[Collector](/docs/pipeline/collector/) [Enrichments](/docs/pipeline/enrichments/) [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)

[Learn more →](/docs/pipeline/)

Data modeling

Transform raw events into analytics-ready tables

[Unified Digital dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) [Attribution dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) [Incremental processing logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) [Identity stitching](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/)

[Learn more →](/docs/modeling-your-data/)

## Build with Snowplow

> **Tip:** Visit our [Developer Hub](https://snowplow.io/developer-hub/) for demo videos, blog posts, workshops, and more.

[Implement an abandoned browse system with composable CDP](/tutorials/abandoned-browse-ccdp/introduction/)

[Engage with customers that show intent to purchase but leave without buying](/tutorials/abandoned-browse-ccdp/introduction/)

[Set up the Unified Digital dbt package](/tutorials/unified-digital/intro/)

[Transform your raw web and mobile event data into derived tables](/tutorials/unified-digital/intro/)

[Use the Snowplow CLI MCP tool](/tutorials/snowplow-cli-mcp/introduction/)

[Let Claude or other MCP clients help with your data structure management](/tutorials/snowplow-cli-mcp/introduction/)

[See all tutorials ->](/tutorials/)

---

# Snowplow Community License FAQ
> Frequently asked questions about the Snowplow Community License terms, usage rights, and restrictions.
> Source: https://docs.snowplow.io/docs/licensing/community-license-faq/

> **Info:** This FAQ is not a substitute for reading [the license text](/community-license-1.0/).

This page explains some of the most common questions about the Snowplow Community License.

## What is the Snowplow Community License?

The Snowplow Community License is a source-available license based on the Confluent Community License, with changes related to rights and restrictions.

## How does the Snowplow Community License work in practice?

Under the Snowplow Community License, you can access the source code and modify it; but you cannot distribute it or use it to make a competing SaaS or on-premises offering. In the license, the things that you are not licensed to do are defined as an Excluded Purpose.

**The exact language**

For purposes of this Agreement, “Excluded Purpose” means making available any on-premises or distributed software product, software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or another similar online service, that competes with any Snowplow products or services that Snowplow or any of its affiliates provide using the Software.

For example, this does not allow the hosting of any Community License version of Snowplow’s Snowbridge product, or other software licensed under the Snowplow Community License, as part of an online service or distribution of on-premises offerings, that compete with Snowplow products or services.

If you are not doing what is excluded, this license change will not affect your current use.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## Can my company or I provide support to others who are running software under the Snowplow Community License?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the Snowplow Community License. Clients can engage you to do this work for them as long as they themselves are complying with our licenses.

## Can my company or I embed software under the Snowplow Community License in the software I distribute?

Only the MIT and Apache 2 licensed software from Snowplow can be embedded and distributed. The Snowplow Community License does not grant you rights to distribute software licensed under it. You do not have the right to distribute components licensed under the Snowplow Community License as part of an on-premises deployed package, and you do not have the right to use them as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), or Infrastructure-as-a-Service (IaaS).

## I have commercially licensed software from Snowplow. Does this impact me?

No, if you have entered into a separate commercial licensing with Snowplow, for example, buying a Snowplow CDI commercial product, then the commercial license terms you have agreed to will continue to govern your use of the software.

## How can I contact Snowplow in case of doubts?

For any further questions on licensing of Snowplow software, please send us an email at <ip-portfolio@snowplow.io>.

---

# Snowplow Contributor License Agreement
> Understand the Snowplow Contributor License Agreement requirements for contributing code to Snowplow projects.
> Source: https://docs.snowplow.io/docs/licensing/contributor-license-agreement/

As the project sponsor, Snowplow Analytics Ltd would like to ensure the long-term viability of Snowplow and its community. The Contributor License Agreement helps ensure everyone can enjoy Snowplow with confidence that Snowplow is here to stay.

Specifically, our Contributor License Agreements (CLAs) grant the contributor and Snowplow Analytics Ltd joint copyright interest in contributed code. Further, it provides assurance from the contributor that contributions are original work that does not violate any third-party license agreement. The agreement between contributors and project is explicit, so Snowplow users can be confident in the legal status of the source code and their right to use it.

## Our CLAs

For all contributions to Snowplow (be they software, bug fixes, configuration changes, documentation, or any other materials), we ask that contributors complete and sign a Contributor License Agreement.

We have two different CLAs, depending on whether you are contributing to Snowplow in a personal or professional capacity:

1. [Individual Contributor License Agreement v1.0](https://docs.google.com/forms/d/1J1FNYq9538ndzzcBdlCbxPo1yFiOY4mwalhDTSl1pgg/viewform)
2. [Software Grant and Corporate Contributor License Agreement v1.0](https://docs.google.com/forms/d/1ZUzz7lQJhs7oZqbkBL1bp0r048hAi7uIN6aLWCyZWWs/viewform)

## For lawyers

Our individual and corporate CLAs are based on the Contributor License Agreements published by the [Apache Software Foundation](http://www.apache.org/), with modifications:

1. Individual Contributor License Agreement ("Agreement") V2.0 ([cached here](/assets/apache-cla-2.0.pdf), or see the [newer V2.2 here](https://apache.org/licenses/icla.pdf))
2. [Software Grant and Corporate Contributor License Agreement ("Agreement") v r190612](http://www.apache.org/licenses/cla-corporate.txt)

If you have questions about Snowplow's own CLAs, please contact us at <legal@snowplowanalytics.com>.

---

# Snowplow licenses
> Information about the different licenses used in Snowplow components and projects, including the Community License, Limited Use License, and Personal and Academic License.
> Source: https://docs.snowplow.io/docs/licensing/

This section provides information about the different licenses used in Snowplow components and projects.

## Source-available components

> **Tip:** Since some of the below licenses are not approved by [OSI](https://opensource.org/licenses/), we do not refer to them as _Open Source_ licenses. However, the source code is still available.

| [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0)                                  | [Community License](/community-license-1.0/)                                                            | [Limited Use License](/limited-use-license-1.1/)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | [Personal & Academic License](/personal-and-academic-license-1.0/)                                          |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| _Usage and distribution for any purpose_                                                  | _Usage in non-competing products, no distribution_ ([FAQ](/docs/licensing/community-license-faq/))      | _Usage in non-production environments only, no distribution_ ([FAQ](/docs/licensing/limited-use-license-faq/))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | _Usage for personal and academic purposes only_ ([FAQ](/docs/licensing/personal-and-academic-license-faq/)) |
| - All tracker SDKs¹
- All Google Tag Manager templates
- Legacy data models (web, mobile) | * Core libraries (schema-ddl, common-streams, etc)
* Developer tools (Chrome Inspector, `igluctl`, etc) | All pipeline apps:* [Collector](/docs/api-reference/stream-collector/) _(3.0.0+)_
* [Enrich](/docs/api-reference/enrichment-components/) _(4.0.0+)_
* [Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) _(0.12.0+)_
* [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) _(6.0.0+)_
* [Snowflake Streaming Loader](/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/)
* [BigQuery Loader](/docs/api-reference/loaders-storage-targets/bigquery-loader/) _(2.0.0+)_
* [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) _(0.2.0+)_
* Other loaders
* [Snowbridge](/docs/api-reference/snowbridge/) _(2.4.0+)_
* Community Edition terraform modules _(2024+)_ | All data models (except web and mobile)                                                                     |

¹As an exception, [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow-javascript-tracker) is distributed under [BSD 3 Clause](https://opensource.org/licenses/BSD-3-Clause). This tracker was originally based on Anthon Pang’s `piwik.js`, and has maintained the same license for distribution.

> **Tip:** When in doubt, consult each component’s GitHub repository for the LICENSE file.

## Proprietary components

[Snowplow CDI](https://snowplow.io/) is built upon the above components, but adds a vast set of proprietary, closed source ones (UI, API, highly available deployment logic, and so on). These are only available under a commercial license for Snowplow customers.

---

# Snowplow Limited Use License FAQ
> Frequently asked questions about the Snowplow Limited Use License Agreement terms and permitted uses.
> Source: https://docs.snowplow.io/docs/licensing/limited-use-license-faq/

> **Info:** This FAQ is not a substitute for reading the license text:
>
> - [version 1.0](/limited-use-license-1.0/)
> - [version 1.1](/limited-use-license-1.1/)

This page explains some of the most common questions about the Snowplow Limited Use License Agreement.

## Why was the Snowplow Limited Use License Agreement introduced?

Previously, many of our Snowplow core pipeline components and dbt models were licensed as open source under the Apache 2.0 license. For more than a decade, our team has invested significant time and resources into researching, developing, and maintaining this code in an open source manner.

Looking forward, we have an exciting technology roadmap that will deliver additional value to all of our core pipeline users. It includes plans to reduce the cost of running the pipeline at scale, improve processing speed, provide more flexibility for custom in-stream processing steps, and support non-HTTP protocols for more efficient data ingestion from server-side and mobile devices. In order to fund these developments, we are asking Snowplow users who are running the Snowplow behavioral data pipeline in production to pay for the value they receive in return.

## What is the Snowplow Limited Use License Agreement?

Snowplow is moving new versions of the Snowplow pipeline components to a source-available license. This license is based on the [Snowplow Community License](/docs/licensing/community-license-faq/), with additional deployment and provisioning restrictions. Notably, this license includes restrictions that prohibit deploying the Snowplow software for production workloads.

## How does the Snowplow Limited Use License Agreement work in practice?

Under the Snowplow Limited Use License Agreement, you can access the source code and modify it, but you cannot distribute it or use it to make a competing SaaS or on-premises offering. Nor can you deploy the Snowplow software in production. Non-production use of the software, such as with development and quality assurance pipelines, are allowed.

**Excerpt: the exact language in the license**

Licensee is not granted the right to, and Licensee shall not, exercise the License for any Competing Use, and Licensee may exercise the License only for Non-Production Use or Non-Commercial Use.

- “Competing Use” is making available any on-premises or distributed software product, or any software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.

- “Non-Production Use” means any use of the Software to process test or synthetic data to evaluate the sufficiency of the Software for use by Licensee.

- “Non-Commercial Use” is only: (a) personal use for research, experiment, personal study, or hobby projects, without any anticipated commercial application, or (b) use for teaching purposes by lecturers of a school or university.

If you are not doing what is excluded, this license change will not affect your current use.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## When do these license changes take effect?

The Snowplow Limited Use License Agreement (SLULA) version 1.0 was rolled out in January, 2024. Version 1.1 is rolled out in December, 2024.

## What changed in version 1.1?

The Snowplow Limited Use License v1.1 (SLULA v1.1) simplifies the terms of acceptable use. It allows you to run SLULA-licensed software in test environments or for academic or research purposes. It does not allow you to run in production or commercial environments. To run in these environments you must obtain a commercial license from Snowplow.

This change does not take away your rights under SLULA 1.0, for versions that were released under that license.

### Summary of changes

Here is a summary of the specific changes in SLULA v1.1 over the v1.0 license:

- Section 1.1 was updated to clarify acceptable use as “Non-Production” or “Non-Commercial” use only
- Section 1.2 and its subsections were replaced completely. We have removed the “Highly Available” provision in the v1.0 license. If you are running the software in production in either a non-highly available or highly available manner, you must obtain a commercial license
- Section 1.2.2 now defines the definitions of “Non-Production Use”. You may use the software on test or synthetic data in a non-commercial manner
- Section 1.2.3, a new section, now defines “Non-Commercial Use”, which includes research, experimentation, personal study or hobby projects with no anticipated commercial application

## What components of Snowplow are offered under this new license?

See the [licensing overview](/docs/licensing/) page.

## What does this mean for me?

If you currently run Snowplow’s open source pipeline code in production (on live data) or in a competitive manner, and wish to use versions of Snowplow released under the SLULA, please contact the team [here](https://snowplow.io/snowplow-oss-license-change/).

If you are a current user of Snowplow open source software but do not run the core pipeline code in production or in a competitive manner, you may continue to use new Snowplow Limited Use License Agreement versions of Snowplow.

If you are a commercial Snowplow CDI Private Managed Cloud or CDI Cloud customer, this license change does not impact your use of Snowplow.

### Examples

Here are some examples what is or is not acceptable under the terms of the SLULA license version 1.1.

#### Allowed

- I want to download and deploy Snowplow internally in either single point of failure (SPOF) or highly available (HA) manner, and run synthetic data through it, for example for product evaluation or load testing purposes
- I want to download the Snowplow software and do research on it (university, hobby, etc), replaying old (real) data, or synthetic data through the pipeline
- I want to download and deploy in a Highly Available (HA) or Single Point of Failure (SPOF) manner Snowplow and publicly expose the pipeline to the internet, capturing events from my personal non-commercial website

#### Not allowed. Requires a commercial license

- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my personal website where I have a hobby business selling photos or art or crafts for my ‘side hustle’
- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my company website or mobile app.
- I want to download and deploy in a HA or SPOF manner Snowplow and publicly expose the pipeline to the internet, capturing events from my company website or mobile app to test if Snowplow can handle the event volume for my company on live data

## Can my company or I provide support to others who are running software under the Snowplow Limited Use License Agreement?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the Snowplow Limited Use License Agreement. Clients can engage you to do this work for them as long as they themselves are complying with our licenses. We do not consider this a competitive use.

## Can my company or I embed software under the Snowplow Limited Use License Agreement in the software I distribute?

Only the MIT/BSD and Apache 2 licensed software from Snowplow can be embedded and distributed. The Snowplow Limited Use License Agreement does not grant you rights to distribute software licensed under it. You do not have the right to distribute components licensed under the Snowplow Limited Use License Agreement as part of an on-premises deployed package, and you do not have the right to use them as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), or Infrastructure-as-a-Service (IaaS).

## How can I contact Snowplow in case of doubts?

For any further questions on licensing of Snowplow software, please send us an email at <ip-portfolio@snowplow.io>.

---

# Snowplow Personal and Academic License FAQ
> Frequently asked questions about the Snowplow Personal and Academic License eligibility and terms.
> Source: https://docs.snowplow.io/docs/licensing/personal-and-academic-license-faq/

> **Info:** This FAQ is not a substitute for reading [the license text](/personal-and-academic-license-1.0/).

This page explains some of the most common questions about the Snowplow Personal and Academic License.

## What is the SPAL?

The SPAL is a license from Snowplow Analytics, designed for products and components where we want to make the source available for non-commercial use. Some of our code is available under open source licenses.

## Which license is the SPAL based on?

The SPAL is based on the [PolyForm Noncommercial License](https://polyformproject.org/licenses/noncommercial/1.0.0/).

## I have commercially licensed software from Snowplow. Does this impact me?

No. If you have entered into a separate commercial licensing with Snowplow, for example, buying a Snowplow CDI commercial product, then the commercial license terms you have agreed to will continue to govern your use of the software.

## Why make the source available?

First, at Snowplow, we have a design principle that all data processing must be ‘glass box’: it must be possible as a data practitioner running Snowplow to understand exactly how the data processing affects output from end to end.

Second, we want to encourage the use of this code as a teaching aid for students.

Third, we want to encourage data practitioners to try out this code for personal uses without needing a commercial relationship with us.

## I am a student or just tinkering, can I run this code?

Yes, the SPAL encourages usage of the code by educators and students for educational purposes. If you are a data practitioner seeking to use this personally, for example on a blog or hobby project, that is also fine.

## I am an educational institution wanting to use this in the running of my business, can I?

No. That is a commercial use. Please contact us for commercial licensing, see [below](#i-want-to-use-this-code-commercially-what-is-the-next-step).

## Can my company or I provide support to others who are running software under the SPAL?

This license is not intended to stop you from providing professional services, such as development, installation, or support, to others who are running software under the SPAL. Clients can engage you to do this work for them as long as they themselves are complying with our licenses.

## Why don’t you release the code under an open source license?

We need to run a sustainable business. Snowplow’s open source software development is funded by our commercial efforts, which include charging to use code released under the SPAL for commercial uses.

## Is Snowplow moving away from open source?

No, we do not plan to wholly move away from open source, but like many companies today, we need to adjust our mix of open source and source-available offerings, to ensure we can continue to have the revenue and resources to support our open source community users.

We remain committed to the transparent processing of data, with the source code available for users to inspect and run for themselves as part of our “glass box” philosophy. Our source code will continue to be made available under one of the following: the Apache 2.0 license, the Snowplow Community License, the Snowplow Limited Use License Agreement, or the Snowplow Personal and Academic License. (See the [licensing overview](/docs/licensing/).)

That means our software will remain transparent and available.

## Can I contribute to this software?

Yes, we welcome community contributions. However, you will need to agree to our [Contribution License Agreement terms](/docs/licensing/contributor-license-agreement/).

## As a commercial user, can I make edits to code?

The SPAL does not allow commercial use. However, if you want to use the code for commercial purposes, we can discuss a license that will allow you to use and modify the code for your commercial purposes.

## I want to use this code commercially, what is the next step?

Please email us at <sales@snowplow.io>.

## Where can I ask my other IP-related questions not covered here?

Please email us at <ip-portfolio@snowplow.io>.

---

# Working with AI
> How Snowplow supports LLM and AI-powered workflows, from real-time behavioral context to tracking plan management and AI-readable documentation.
> Source: https://docs.snowplow.io/docs/llms-support/

Snowplow supports agentic and LLM-powered workflows in several ways.

## Snowplow CLI MCP server

The [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) includes a Model Context Protocol (MCP) server that [connects AI assistants to your Snowplow tracking plans](/docs/llms-support/mcp-server/). This can help you design tracking plans faster and more consistently.

## Documentation index in `llms.txt`

This documentation follows the [`llms.txt` standard](https://llmstxt.org/), providing structured information to help an LLM use the site.

An index is available at [`llms.txt`](/llms.txt).

An extended version is also available at [`llms-full.txt`](/llms-full.txt), that includes the complete text of the current pages. For token efficiency, sections relating to older versions of components aren't included in this file. These sections are still listed in the `llms.txt` index, labeled as `[previous version]`.

The `llms-full.txt` file is very large. It might be more effective to access individual pages as needed, using the Markdown access method described below.

## Documentation pages as Markdown

Every documentation page is available as Markdown. To download a page's content, use the **Download** or **Copy Markdown** buttons above the page title.

Following the `llms.txt` standard, you can access the Markdown page directly by changing the trailing `/` in the URL to `.md`. For example:

- HTML: `https://docs.snowplow.io/docs/signals/concepts/`
- Markdown: `https://docs.snowplow.io/docs/signals/concepts.md`

Let your LLM know that this format is available, so it can retrieve content efficiently.

## Signals

Use [Signals](/docs/signals/concepts/) to provide real-time behavioral context to your AI applications. It computes user attributes from your event stream and warehouse data, and makes them available to your applications via the [Profiles Store](/docs/signals/concepts/#profiles-store) API.

---

# MCP server
> Use the Snowplow CLI MCP server to interact with your tracking plans through AI assistants.
> Source: https://docs.snowplow.io/docs/llms-support/mcp-server/

The [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) includes a local Model Context Protocol (MCP) server that enables natural language interaction with AI assistants for creating, validating, and managing your Snowplow tracking plans. This allows you to:

- Create and validate data structures through conversation
- Analyze tracking requirements and suggest implementations
- Validate tracking plans and source applications

> **Info:** For a step-by-step guide to using the Snowplow MCP server, see the [MCP tutorial](/tutorials/snowplow-cli-mcp/introduction/).

The MCP server allows AI assistants to interact with the Snowplow CLI using the `snowplow-cli mcp` command.

## Install the MCP server

The MCP server is included with [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/). Start by installing the CLI and authenticating to your Snowplow account.

To connect your AI client to the MCP server, provide a configuration object. The exact structure depends which client you are using, and whether you're connecting to the Snowplow CLI installed locally or via npx. For example:

**Claude Desktop:**

Add the following to your Claude Desktop configuration file, found at:

- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`

For direct Snowplow CLI connection:

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "snowplow-cli",
      "args": ["mcp"]
    }
  }
}
```

Using npx:

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "npx",
      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]
    }
  }
}
```

> **Note:** Claude Desktop requires additional filesystem access to create and modify files.

**VS Code:**

Add to `.vscode/mcp.json` in your workspace.

For direct Snowplow CLI connection:

```json
{
  "servers": {
    "snowplow-cli": {
      "type": "stdio",
      "command": "snowplow-cli",
      "args": ["mcp"]
    }
  }
}
```

Using npx:

```json
{
  "servers": {
    "snowplow-cli": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]
    }
  }
}
```

**Cursor:**

Add to `.cursor/mcp.json` in your workspace:

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "snowplow-cli",
      "args": ["mcp", "--base-directory", "."]
    }
  }
}
```

***

After adding the configuration, restart the client. The Snowplow CLI MCP tools should be available for use.

## Provided tools

The Snowplow CLI MCP server provides these tools to your AI assistant:

- `get_context`: to call at the start of each conversation, this tool retrieves the underlying Snowplow rules that define how Snowplow components should be structured
- `get_uuid`: generates valid v4 UUIDs that are required by many Snowplow components
- `validate_data_structures`: the assistant will use this validation tool after creating or modifying any data structure.
- `validate_data_products`: the assistant will use this validation tool after creating or modifying any tracking plan or source application file

The `get_context` tool is the only one you might need to call manually.

## Using the tools

To get AI support with your Snowplow tracking plans, start a new conversation and ask the assistant to call `get_context`. It might call this tool automatically if it realises it needs the Snowplow knowledge.

You can then ask it for help. It has access to your existing data structures, tracking plans, or source application definitions. The assistant will automatically validate any file changes using the included validation tools.

Use the standard [Snowplow CLI commands](/docs/event-studio/programmatic-management/snowplow-cli/reference/) to publish your changes to [Console](https://console.snowplowanalytics.com) when ready, using the standard Snowplow CLI commands such as `snowplow-cli ds publish`.

> **Note:** The MCP server creates and validates files on your local filesystem only.

---

# Migrate from Adobe Analytics to Snowplow
> Migrate from Adobe Analytics to Snowplow with guidance on event tracking, eVars, props, and warehouse modeling.
> Source: https://docs.snowplow.io/docs/migration-guides/adobe-analytics/

This guide is to help technical implementers migrate from Adobe Analytics to Snowplow. For more advice on tracking plans, check out our [best practices guide](/docs/fundamentals/tracking-design-best-practice/).

## Platform differences

There are significant differences between Adobe Analytics and Snowplow as data platforms. For migration, it's important to understand how Snowplow structures events differently from Adobe Analytics. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Adobe Analytics                                                                                       | Snowplow                                                                                                                                                              |
| ----------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Uses numbered variables                                                                               | Direct tracker calls for built-in or custom [event](/docs/fundamentals/events/) types, including contextual data as reusable [entities](/docs/fundamentals/entities/) |
| Warehouse tables        | Data Feeds export to flat files with numbered columns (prop1-prop75, eVar1-eVar250, event1-event1000) | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity   |
| Data validation         | Processing rules and VISTA rules for transformation; no schema validation                             | All events and entities defined by JSON schemas                                                                                                                       |

### Event structure

Adobe Analytics uses two primary [tracking methods](https://experienceleague.adobe.com/en/docs/analytics/implementation/vars/functions/overview#): `s.t()` for page views and `s.tl()` for custom link tracking. Data is captured through props, eVars, and events.

Each of these types uses numbered variables with different persistence scopes:

- Props (`s.prop1-75`): hit-scoped traffic variables that expire after the hit
- eVars (`s.eVar1-250`): conversion variables with configurable expiration (hit, visit, visitor, or specific events)
- Events (`event1-1000`): counters and currency values for success metrics

This numbered variable system requires external documentation to map variable numbers to business meanings. For example, `eVar5` might represent "campaign code" in one implementation and "product category" in another.

Here's an example Adobe Analytics web ecommerce implementation. The code sets various variables before calling the tracking method:

```javascript
// Adobe Analytics purchase tracking
s.pageName = 'Order Confirmation';
s.events = 'purchase,event1';
s.products = ';SKU_123;1;25.42';
s.eVar1 = '12345';           // transaction_id
s.eVar2 = 'USD';             // currency
s.prop1 = 'Electronics';     // category
s.purchaseID = '12345';
s.t();
```

The `s.products` variable uses a specific syntax with semicolon delimiters: `category;product;quantity;price`. Success events like `purchase` are tracked in `s.events`. Context is spread across numbered props and eVars, requiring documentation to understand what each variable represents.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event:

```javascript
snowplow('trackTransaction', {
  transaction_id: 'T_12345',
  revenue: 25.42,
  currency: 'USD',
  products: [{
    id: 'SKU_123',
    name: 'Product Name',
    category: 'Electronics',
    price: 25.42,
    quantity: 1
  }]
});
```

As well as the tracking code looking different, there are key differences in the warehouse loading and data modeling:

- Snowplow processes this as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Tracking comparison

This table shows how Adobe Analytics tracking maps to Snowplow tracking:

| Adobe Analytics Method | Adobe Example                                                    | Snowplow Implementation                                                                                                                                                                 |
| ---------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `s.t()` (page view)    | `s.pageName='Home'; s.t();`                                      | Use `trackPageView`. The tracker automatically captures page title, URL, and referrer.                                                                                                  |
| `s.tl()` (link click)  | `s.tl(this,'o','Button Click');`                                 | Use `trackLinkClick` for links, or define a custom schema and track with `trackSelfDescribingEvent`.                                                                                    |
| `s.tl()` (custom)      | `s.linkTrackVars='eVar1'; s.eVar1='value'; s.tl(this,'o','CTA')` | Define a custom schema containing the relevant properties and track with `trackSelfDescribingEvent`.                                                                                    |
| Props (hit-scoped)     | `s.prop1 = 'value';`                                             | Attach as entities to the specific event.                                                                                                                                               |
| eVars (visit-scoped)   | `s.eVar1 = 'value';`                                             | Attach as entities. For visit-scoped persistence, use [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) or model persistence in the warehouse. |
| Success events         | `s.events = 'event1,purchase';`                                  | Track as specific event types or custom self-describing events. Use entities for associated values.                                                                                     |
| Products string        | `s.products = ';SKU;1;99.99';`                                   | Use built-in ecommerce tracking, or define a `product` entity schema with explicit properties.                                                                                          |
| `s.visitorID`          | `s.visitorID = 'user123';`                                       | Use `setUserId('user123')` to set user ID for all subsequent events.                                                                                                                    |

### Persistence and scope

Adobe Analytics eVars have [configurable persistence](https://experienceleague.adobe.com/en/docs/analytics/components/dimensions/evar#): they can expire after a hit, visit, or specific time period, or persist until a conversion event. This persistence is handled server-side by Adobe.

Snowplow takes an event-centric approach where each event is self-contained. There's no built-in persistence mechanism. Instead, you have two options:

1. **Global context**: attach entities to all events during a session using the tracker's [global context](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/) feature
2. **Warehouse modeling**: reconstruct persistence logic in your data models using SQL window functions

For most use cases, warehouse modeling provides more flexibility. You can define custom attribution windows, change persistence logic without re-instrumenting, and apply different persistence rules to historical data.

### Data validation

Adobe Analytics provides Processing Rules and VISTA rules for data transformation, but doesn't validate incoming data against schemas. Invalid or unexpected values are accepted and processed. Classification rules can categorize values post-collection, but there's no prevention of malformed data.

Snowplow requires schematic specification for all data. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

You've likely documented your Adobe Analytics implementation in a Solution Design Reference spreadsheet that maps business requirements to props, eVars, and events. Snowplow provides event data management tools for defining and managing tracking plans. Each [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to automatically generate the tracking code snippets.

## Tag management

If you're using Adobe Launch (Adobe Experience Platform Tags) to deploy Adobe Analytics, you'll need to decide how to deploy Snowplow tracking.

There are three main options:

| Approach                                                                                | Description                                            | Best for                                                                                 |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Direct [tracker SDK](/docs/sources/) implementation                                     | Add Snowplow tracker code directly to your application | Teams with engineering resources; cleanest long-term solution                            |
| [Google Tag Manager](/docs/sources/google-tag-manager/)                                 | Use Snowplow's GTM tag templates to deploy tracking    | Teams already using GTM for other tools, or wanting to move away from Adobe Launch       |
| [GTM Server-side](/docs/destinations/forwarding-events/google-tag-manager-server-side/) | Forward events through a server-side container         | Teams already using GTM Server-side, or with strict client-side performance requirements |

Direct SDK implementation is recommended for most migrations. It provides the cleanest separation from your previous Adobe implementation and full access to Snowplow features.

If you choose GTM, you can use Snowplow's tag templates to fire Snowplow events based on data layer pushes, similar to how Adobe Launch rules fire based on data elements.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Your existing Adobe Analytics Solution Design Reference will serve as the foundation for your new Snowplow implementation. Before migration, audit your tracking implementation and Solution Design Reference. Are all the props, eVars, and events still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Review your existing Adobe Analytics implementation using one or more of these methods:

- Solution Design Reference review: examine your tracking plan spreadsheet for the complete mapping of business requirements to variables
- Code audit: review all `s.t()` and `s.tl()` calls in your codebase, along with variable assignments
- Data Feeds analysis: analyze your Data Feeds exports to identify which props, eVars, and events contain data
- Report suite inspection: use the Adobe Analytics Admin Console to review variable configurations and processing rules

Pay special attention to:

- Variable persistence settings for eVars
- Processing rules that transform incoming data
- VISTA rules for server-side data manipulation
- Classification rules that categorize values
- Calculated metrics that combine multiple variables

You'll need to translate your Adobe Analytics configuration into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Adobe events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- How should you group props and eVars into logical [entities](/docs/fundamentals/entities/)?
- Which eVar persistence behaviors need to be replicated in warehouse modeling?
- How will you handle the `s.products` string? Snowplow's ecommerce tracking or custom product entities are more flexible alternatives.

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/llms-support/mcp-server/) to help you translate your Adobe Analytics event configuration into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Adobe Analytics Data Feeds in existing tables. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Adobe data into the Snowplow format. This requires a custom engineering project to export Data Feeds, reshape the numbered columns into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Adobe Analytics tracking:

- If you have a web platform, we recommend starting here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Adobe Analytics and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Replicate any eVar persistence logic in your warehouse models
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily page views, or unique visitors

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Adobe Analytics tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Adobe Analytics.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Adobe Analytics tracking from your codebases. Finally, archive your Adobe Analytics configuration and Data Feeds exports.

---

# Google Analytics to Snowplow
> Migrate from Google Analytics to Snowplow, including data layer implementations with Google Tag Manager.
> Source: https://docs.snowplow.io/docs/migration-guides/google-analytics/

This guide is to help technical implementers migrate from Google Analytics to Snowplow.

## Platform differences

There are [significant differences](https://snowplow.io/comparisons/snowplow-vs-google-analytics) between Google Analytics and Snowplow as data platforms. For migration, it's important to understand how Snowplow structures events differently from GA4. This affects both your data layer implementation and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Google Analytics 4                                                                                | Snowplow                                                                                                                                                           |
| ----------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Capturing user behavior | Uses `gtag('event', ...)` with event names, or `dataLayer.push()` with GTM triggers and variables | Direct tracker SDK calls for built-in or custom [event](/docs/fundamentals/events/) types                                                                          |
| Contextual event data   | Included in the event `parameters` object                                                         | Included in the event as reusable [entities](/docs/fundamentals/entities/)                                                                                         |
| Warehouse tables        | BigQuery only; daily partitioned `events_YYYYMMDD` table with nested fields                       | In [most warehouses](/docs/destinations/warehouses-lakes/) one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | None                                                                                              | All events and entities defined by JSON schemas                                                                                                                    |

### Event structure

The main GA4 tracking method is `gtag('event', ...)` with event names and parameter objects. The event name describes the action taken, and the parameters provide contextual information about that action.

You might have implemented these calls directly in your codebase. More likely, you're using Google Tag Manager (GTM) and tracking behavior via the data layer. When using GTM, your application pushes structured data to the `dataLayer` array. GTM then processes this data using triggers and variables to fire the appropriate `gtag()` calls.

Here's an example GA4 web ecommerce event using the data layer. The application pushes a `purchase` event with an `ecommerce` object containing transaction details:

```javascript
dataLayer.push({
  event: 'purchase',
  ecommerce: {
    transaction_id: '12345',
    value: 25.42,
    currency: 'USD',
    items: [{
      item_id: 'SKU_123',
      item_name: 'Product Name',
      price: 25.42,
      quantity: 1
    }]
  }
});
```

GTM processes this data layer push and fires the corresponding GA4 event. Alternatively, you might have implemented the event directly using `gtag()` calls, like this:

```javascript
gtag('event', 'purchase', {
  transaction_id: '12345',
  value: 25.42,
  currency: 'USD',
  items: [{
    item_id: 'SKU_123',
    item_name: 'Product Name',
    price: 25.42,
    quantity: 1
  }]
});
```

To directly track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. The tracking code looks similar:

```javascript
snowplow('trackTransaction', {
  transaction_id: 'T_12345',
  revenue: 25.42,
  currency: 'USD',
  products: [{
    id: 'SKU_123',
    name: 'Product Name',
    price: 25.42,
    quantity: 1
  }]
});
```

### Tracking comparison

This table shows how some data layer GA4 tracking maps to Snowplow web tracking:

| GA4 Event     | GA4 Data Layer Example                                                                                                      | Snowplow Implementation                                                                                                                                        |
| ------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `page_view`   | Sent by default by GTM, or `dataLayer.push({event: 'page_view'})`.                                                          | Use `trackPageView`                                                                                                                                            |
| `purchase`    | `dataLayer.push({event: 'purchase', ecommerce: {transaction_id: '123', value: 99.99}})`                                     | Use built-in ecommerce tracking, or define a custom `purchase` schema containing `transaction_id` and `value` properties, and track `trackSelfDescribingEvent` |
| Custom events | Define custom GTM configuration then call e.g. `dataLayer.push({event: 'read_article', article_title: 'Interesting text'})` | Define a custom `read_article` schema containing `article_title` property and track with `trackSelfDescribingEvent`                                            |
| Send user IDs | `dataLayer.push({user_id: 'USER_123'})`, configured as user property in GTM                                                 | Use `setUserId('USER_123')` to set the user ID                                                                                                                 |
| Page metadata | `dataLayer.push({page_type: 'product', category: 'electronics'})`, configured as custom dimensions in GTM                   | Define custom entities for page context and attach to relevant events                                                                                          |

### Warehouse loading and data modeling

The key differences between GA4 and Snowplow events are in the warehouse loading and data modeling.

With GA, the example `purchase` event would be loaded into an `events_YYYYMMDD` table in BigQuery, with nested RECORD fields for parameters and user properties. Each parameter exists as key-value pairs with separate columns for different data types: `string_value`, `int_value`, `float_value`, or `double_value`. Analysts must use complex `UNNEST` operations to extract parameters and join data.

With Snowplow, the example event would be processed as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/). The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser.

In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity. In Redshift, each event and entity is loaded into its own table.

You can add the same `product` entity to any other relevant event, such as `add_to_cart` or `view_item`. This simplifies analysis of product lifecycles, from initial product view to purchase.

Snowplow tracker SDKs provide built-in methods for tracking page views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) for initial modeling and common analytics use cases.

### Data validation

GA4 is loosely typed and doesn't validate incoming events. You can define expected properties and custom configuration, but events that don't match their specification are still processed. The defined configurations are mainly used for reporting in the GA4 interface.

The lack of validation leads to fragmented and low quality data. For example, the `price` for a purchase event might be accidentally tracked in different places as `25.42`, `"25.42"`, or even `2542`. In the warehouse, this parameter's values will be split over `float_value`, `string_value`, and `int_value` columns, making querying more complex.

Event specification and validation isn't optional for Snowplow. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

#### Tracking plans

You've probably defined which GA4 events and parameters to track in a tracking plan spreadsheet or similar document. Snowplow provides event data management tools for defining and managing tracking plans. Each [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, you may be able to use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to automatically generate the tracking code snippets.

## Migration approaches

In general, we recommend a parallel-run migration approach. This involves running both GA4 and Snowplow tracking simultaneously, allowing you to validate your Snowplow data in your warehouse before switching over completely.

If you're using GA4 with direct `gtag()` calls, you can replace these with Snowplow tracker SDK calls in your application code.

However, Snowplow also provides templates and tags for GTM and GTM Server-side. You could choose to migrate your data layer tracking and your modeling implementations in separate stages. This table summarizes the three possible approaches to reach a parallel-run dual tracking implementation:

| Approach                                                                                        | Phase 1                                                              | Phase 2                                                                                         | Phase 3                           | Best for                                                                              |
| ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------- |
| Direct [tracker SDK](/docs/sources/) calls                                                      | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Add Snowplow tracking calls; implement Snowplow data models                                     |                                   | Teams with tracking implementation engineer resources; greenfield validation projects |
| [Snowplow GTM templates](/docs/sources/google-tag-manager/)                                     | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Configure Snowplow tags to receive data layer events; implement Snowplow data models            | Implement Snowplow tracking calls | Validation of Snowplow data quality for analytics or marketing teams                  |
| [GTM Server-side](/docs/destinations/forwarding-events/google-tag-manager-server-side/) tagging | Define Snowplow [tracking plans](/docs/fundamentals/tracking-plans/) | Use GTM Server-side to forward your existing events to Snowplow; implement Snowplow data models | Implement Snowplow tracking calls | Validation of Snowplow data quality for teams already using GTM Server-side           |

Using Snowplow's GTM templates or tags is a temporary solution to help you migrate.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into four phases:

1. Assess and plan
2. Validate data
3. Finish implementation
4. Cutover and finalize

### 1. Assess and plan

Your existing GA4 event configuration will serve as the foundation for your new Snowplow implementation. Before migration, we advise auditing your tracking calls and event configuration. Are all the events and parameters still relevant? Does your configuration match your tracking plan?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Review your existing GA4 implementation using one or more of these methods:

- Tag manager inspection: if using GTM, export your container configuration to identify all GA4 tags, their triggers, and variable mappings
- Programmatic API export: use the [Google Analytics Admin API](https://developers.google.com/analytics/devguides/config/admin/v1) to retrieve the full JSON definition of all custom event configurations
- Code audit: review all `dataLayer.push()` or `gtag('event', ...)` calls in your codebase
- Warehouse inference: analyze your BigQuery `events_YYYYMMDD` tables to identify all event names and parameters in use

For data layer implementations, pay special attention to:

- The structure of your data layer objects and how properties are nested
- Which GTM variables extract and transform data layer properties
- How GTM triggers determine when events are sent
- Any custom JavaScript variables that process data layer data

You'll need to translate your GA4 event configuration into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which GA4 events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- Are there sets of event parameters used in multiple places that could be defined as [entities](/docs/fundamentals/entities/) instead?
- What's the best combination of event properties and entities to capture the same data as GA4?
- If you're using GTM, will you go straight to implementing direct tracker SDK calls, or use Snowplow's GTM templates or GTM Server-side tagging to maintain the data layer pattern initially?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/event-studio/programmatic-management/snowplow-cli/#mcp-server) to help you translate your GA4 event configuration into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical GA4 data in BigQuery. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical GA4 data into the Snowplow format. This requires a custom engineering project to export GA4 data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Validate data

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Start sending events to Snowplow and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

> **Info:** Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate your Snowplow events locally, before sending them to your warehouse.
>
> Snowplow Micro is a Dockerized local pipeline. It uses the same schema registry for event validation as your production pipeline.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

Your GA4 implementation will stay live until the final migration phase. By the end of this phase, you'll have a validated Snowplow implementation that captures or recreates your GA4 tracking, and recreates your modeling. The goal is to be confident that the new Snowplow pipeline is capturing critical business logic correctly.

#### Direct tracker SDK calls

For the most straightforward migration, add Snowplow tracking in parallel with your existing GA4 tracking:

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)
- Use the [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) to confirm that the tracker is generating the expected events
- Use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to generate custom tracking code for your tracking plans

#### Snowplow GTM templates

If you're using GTM and don't have the resources to implement Snowplow tracking yet, you can use [Snowplow's GTM tag templates](/docs/sources/google-tag-manager/) to generate Snowplow events without changing your application code:

- Keep your existing `dataLayer.push()` calls in your application code
- Install the [Snowplow GTM tag template](/docs/sources/google-tag-manager/) in your GTM web container
- Configure GTM triggers to fire Snowplow tags based on your data layer events
- Use GTM variables to map data layer properties to Snowplow event and entity fields

For example, the existing data layer push stays the same:

```javascript
dataLayer.push({
  event: 'purchase',
  ecommerce: {
    transaction_id: '12345',
    value: 25.42,
    currency: 'USD'
  }
});
```

In GTM, you would configure a Snowplow tag that fires when `event` equals `purchase`, mapping the ecommerce properties to a custom Snowplow self-describing event or using the built-in ecommerce tracking.

#### GTM server-side tagging

If you're already using GTM Server-side to forward events, you could configure Snowplow as a destination using the [Snowplow GTM Server-side tag](/docs/destinations/forwarding-events/google-tag-manager-server-side/):

1. Keep your existing `dataLayer.push()` calls in your application code
2. Configure GTM web container to forward events to GTM server-side
3. Set up Snowplow tracking in the GTM server-side container
4. Use GTM server-side variables and transformations to map data layer data to Snowplow events

### 3. Finish implementation

If you used Snowplow GTM templates or GTM server-side tagging in the previous phase, it's time to implement direct Snowplow tracker SDK calls in your application code.

The benefits of a full Snowplow implementation include:

- Enables a complete migration away from GA4
- Allows you to take advantage of all Snowplow features, including out-of-the-box tracking definitions and event data management tools
- Increased page performance
- Improved debugging and maintenance

### 4. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission GA4.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the GA4 tracking from your codebases. Finally, archive your GA4 configuration and data exports.

---

# Migration guides
> Migrate to Snowplow from other analytics solutions using parallel-run or full re-architecture approaches.
> Source: https://docs.snowplow.io/docs/migration-guides/

This section contains advice for migrating to Snowplow from other solutions. For more advice on tracking plans, check out our [best practices guide](/docs/fundamentals/tracking-design-best-practice/).

When migrating to Snowplow from another analytics solution, there are two possible strategies: parallel-run or full re-architecture.

We recommend a **parallel-run** approach as the lowest-risk strategy. It involves running both systems simultaneously (dual tracking) before switching over to Snowplow entirely. This allows you to test and validate your new Snowplow data in your warehouse, without affecting any existing workflows or production systems.

A **full re-architecture** or "rip-and-replace" approach is faster but riskier, involving a direct switch from your existing system to Snowplow. This is best suited for:

- Major application refactors where the switch can be part of a larger effort
- Teams with high risk tolerance and robust automated testing frameworks
- New projects or applications with minimal legacy systems

A full re-architecture strategy requires thorough testing in a staging environment to prevent data loss.

---

# Migrate from Mixpanel to Snowplow
> Migrate from Mixpanel to Snowplow with guidance on event tracking, user profiles, super properties, and warehouse modeling.
> Source: https://docs.snowplow.io/docs/migration-guides/mixpanel/

This guide is to help technical implementers migrate from Mixpanel to Snowplow.

## Platform differences

There are significant differences between Mixpanel and Snowplow as data platforms. For migration, it's important to be aware of how Snowplow structures events differently from Mixpanel. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Mixpanel                                                                   | Snowplow                                                                                                                                                            |
| ----------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Tracked as named events with properties                                    | Tracked as structured events with built-in types or custom [events](/docs/fundamentals/events/) with validated schemas                                              |
| Contextual event data   | Attached as properties within the event                                    | Attached as separate reusable [entities](/docs/fundamentals/entities/)                                                                                              |
| User and group data     | Tracked separately                                                         | Attached to events as part of the event payload or as reusable entities                                                                                             |
| Warehouse tables        | Events in single or per-event-type tables; user profiles in separate table | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | Optional Tracking Plans                                                    | All events and entities defined by JSON schemas                                                                                                                     |

### Event structure

Mixpanel tracks user behavior with `track()`. Here's an example custom web ecommerce Mixpanel event. The event name is `Product Purchased`, describing the action taken. The properties object provides contextual information about the specific purchase:

```javascript
mixpanel.track('Product Purchased', {
  product_id: 'SKU_123',
  product_name: 'Widget',
  price: 49.95,
  quantity: 2,
  category: 'Electronics'
});
```

This event loads into Mixpanel's events warehouse table with properties as columns. The table structure depends on your Mixpanel warehouse export configuration.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. Alternatively, you could define a custom `product_purchased` schema and track it as a [self-describing event](/docs/fundamentals/events/#self-describing-events):

```javascript
snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.acme/product_purchased/jsonschema/1-0-0',
    data: {
      product_id: 'SKU_123',
      product_name: 'Widget',
      price: 49.95,
      quantity: 2,
      category: 'Electronics'
    }
  }
});
```

The differences are in the warehouse loading and data modeling:

- Snowplow processes this as a self-describing [event](/docs/fundamentals/events/) with validated [schema](/docs/fundamentals/schemas/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery, Databricks, or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Events and users

Mixpanel separates timestamped events from persistent user profiles. Events capture actions with timestamps, while user profiles store persistent attributes like name, email, or subscription tier. Here's the Mixpanel pattern:

```javascript
// Track event
mixpanel.track('Page Viewed');

// Set user profile properties
mixpanel.people.set({
  'name': 'Jane Doe',
  'email': 'jane@example.com',
  'plan': 'Premium'
});
```

In Mixpanel, events and user profiles are typically stored in separate tables. Analysts must join these tables to analyze user behavior with profile attributes.

There's no separate user profile system in Snowplow. You can set a user identifier as part of every event, and attach user attribute entities to events as needed:

```javascript
// Set user ID
snowplow('setUserId', 'user_123');

// Track event with user context entity
snowplow('trackPageView', {
  context: [{
    schema: 'iglu:com.acme/user/jsonschema/1-0-0',
    data: {
      name: 'Jane Doe',
      email: 'jane@example.com',
      plan: 'Premium'
    }
  }]
});
```

This event-centric approach means you can reconstruct user state from the events in the warehouse. This provides a complete audit trail of all attribute changes over time.

### Super properties

In Mixpanel, super properties are event properties that are automatically attached to every event you track. They're used to capture contextual information that applies across all events, such as user plan type, app version, or environment.

```javascript
// Register super properties in Mixpanel
mixpanel.register({
  app_version: '2.1.0',
  environment: 'production',
  user_plan: 'premium'
});

// All subsequent events include these properties
mixpanel.track('Page Viewed');
mixpanel.track('Image Expanded');
```

There's no direct Snowplow equivalent to super properties. Instead, you attach "global context" [entities](/docs/fundamentals/entities/) to every event. This approach provides the same functionality with stronger typing and validation through schemas.

Here's how to implement the same pattern with the [JavaScript tracker](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/):

```javascript
// Define your global context entity
const appContext = {
  schema: 'iglu:com.acme/app_context/jsonschema/1-0-0',
  data: {
    app_version: '2.1.0',
    environment: 'production',
    user_plan: 'premium'
  }
};

// Add as global context - attached to all events
snowplow('addGlobalContexts', [appContext]);

// All subsequent events include this entity
snowplow('trackPageView');
snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.acme/image_expanded/jsonschema/1-0-0',
    data: { button_id: 'submit' }
  }
});
```

You can also update the global context properties dynamically during runtime. This is useful when values change during the session.

### Tracking comparison

This table shows how example Mixpanel tracking calls could be mapped to Snowplow tracking:

| Mixpanel API Call      | Mixpanel Example                                                         | Snowplow Implementation                                                                                                                                                       |
| ---------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `track()`              | `mixpanel.track('Order Completed', {revenue: 99.99})`                    | Use one of the built-in event types, or define a custom `order_completed` schema containing `revenue` property and track with `trackSelfDescribingEvent`.                     |
| `identify()`           | `mixpanel.identify('user_123')`                                          | Use `setUserId('user_123')` to set user ID for all subsequent events.                                                                                                         |
| `alias()`              | `mixpanel.alias('user_123', 'anon_456')`                                 | No direct equivalent. Track identity changes as custom events with `user_alias_created` schema if needed for analysis. Use `setUserId` to update the current user identifier. |
| `register()`           | `mixpanel.register({plan: 'premium'})`                                   | Attach as entities to all events.                                                                                                                                             |
| `.people.set()`        | `mixpanel.people.set({email: 'user@example.com'})`                       | Attach custom user entities to events.                                                                                                                                        |
| `.people.increment()`  | `mixpanel.people.increment('page_views')`                                | No direct equivalent. Track events and calculate aggregates in warehouse.                                                                                                     |
| `.track_with_groups()` | `mixpanel.track_with_groups('Button Clicked', {groupId: 'company-123'})` | Attach custom group entities to events containing group identifiers and relevant group properties.                                                                            |
| `reset()`              | `mixpanel.reset()`                                                       | Clear user ID with `setUserId(null)` or equivalent, depending on tracker. The tracker will generate a new anonymous identifier.                                               |

### Data validation

You've presumably defined Tracking Plans for your Mixpanel implementation. The suggested Mixpanel approach is to create and maintain an external spreadsheet of events and properties to track. Within Mixpanel, the Lexicon feature allows you to manage definitions, and hide or drop events. There's no built-in validation of tracked events.

Snowplow requires schematic specification for all data. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out.

To help maintain high data quality, Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets.

Snowplow includes in-product management and tools for [tracking plans](/docs/fundamentals/tracking-plans/). Each tracking plan contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to automatically generate the tracking code snippets.

> **Info:** The Snowplow CLI includes an [MCP server](/docs/llms-support/mcp-server/) to help you translate your Mixpanel Tracking Plans into Snowplow tracking plans.

## Migration phases

Use a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Use your existing Mixpanel Tracking Plans as the foundation for your new Snowplow implementation. Before migration, audit your tracking calls and Tracking Plans. Are all the events and properties still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Export your Tracking Plans using one of these methods:

- Programmatic API export (recommended): use the [Mixpanel Public API](https://developer.mixpanel.com/reference/overview) to retrieve the full definition of each Tracking Plan
- Manual CSV download: download a CSV file from the Mixpanel UI for a less detailed but quick inventory of your events and properties
- Warehouse inference: if you no longer have access to your Mixpanel account, you can infer the events and properties from your warehouse data

You'll need to translate your Mixpanel Tracking Plans into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Tracking Plan events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- How should you migrate Mixpanel super properties? These are typically best represented as global context [entities](/docs/fundamentals/entities/) that are attached to all events.
- Which user profile properties should be captured as user entities? You might not need to attach user entities to every event, only those where the user context is relevant for analysis.
- Which events use group analytics? You'll need to define group entities for these.
- Are there sets of event properties used in multiple places that could be defined as reusable entities instead?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plan tracking plans into Snowplow is to use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/).

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Mixpanel data in existing tables. Write queries that combine data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Mixpanel data into the Snowplow format. This requires a custom engineering project to export Mixpanel data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset. Note that this is more complex for Mixpanel than other platforms due to the separation of events and user profiles.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Mixpanel tracking:

- If you have a web platform, start here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Mixpanel and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Mixpanel tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Mixpanel.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Mixpanel tracking from your codebases. Finally, close your Mixpanel account.

---

# Segment to Snowplow
> A guide for technical implementers migrating from Segment to Snowplow, covering event structure differences, tracking plan translation, and a phased migration approach.
> Source: https://docs.snowplow.io/docs/migration-guides/segment/

This guide is to help technical implementers migrate from Segment to Snowplow.

## Platform differences

There are a [number of differences](https://snowplow.io/comparisons/snowplow-vs-segment) between Segment and Snowplow as a data platform. For migration, it's important to be aware of how Snowplow structures events differently from Segment. This affects how you'll implement tracking and how you'll model the warehouse data.

This table shows some key differences:

| Feature                 | Segment                                   | Snowplow                                                                                                                                                            |
| ----------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Capturing user behavior | Uses `track` events                       | Choose from the built-in event types, or use `trackSelfDescribingEvent` to track a custom [event](/docs/fundamentals/events/)                                       |
| Contextual event data   | Included in the event `properties` object | Included in the event as reusable [entities](/docs/fundamentals/entities/)                                                                                          |
| User identity           | Tracked separately as an `identify` event | Included in the event as a reusable entity                                                                                                                          |
| Warehouse tables        | One table per custom event type           | In [most warehouses](/docs/destinations/warehouses-lakes/), one big table with columns for every event or entity; in Redshift, one table per custom event or entity |
| Data validation         | Optional Tracking Plan                    | All events and entities defined by JSON schemas                                                                                                                     |

### Event structure

Segment's core method for tracking user behavior is `track`. Here's an example custom web ecommerce Segment event. The event name is `Transaction Completed`, describing the action taken. The properties object provides contextual information about the specific transaction, including an array of product objects:

```javascript
analytics.track('Transaction Completed', {
  transaction_id: 'T_12345',
  revenue: 99.99,
  currency: 'USD',
  products: [{
    id: 'ABC123',
    name: 'Widget',
    price: 99.99,
    quantity: 1
  }]
})
```

This event would be loaded into a `Transaction Completed` warehouse table. Analysts must `UNION` tables together to reconstruct user journeys.

To track the same behavior with Snowplow, you could use the [web ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) `trackTransaction` event. The tracking code looks very similar:

```javascript
snowplow('trackTransaction', {
  transaction_id: 'T_12345',
  revenue: 99.99,
  currency: 'USD',
  products: [{
    id: 'ABC123',
    name: 'Widget',
    price: 99.99,
    quantity: 1
  }]
})
```

The differences are in the warehouse loading and data modeling:

- Snowplow processes this as a `transaction` [event](/docs/fundamentals/events/) with one `product` [entity](/docs/fundamentals/entities/)

- The Snowplow [tracker SDKs](/docs/sources/) and pipeline add additional contextual entities to each event, for example information about the specific page or screen view, the user's session, or the browser

- In BigQuery or Snowflake, the event will be loaded into the single `atomic.events` table, with a column for the `transaction` event and a column for each entity

  - Columns have type `OBJECT` in Snowflake, and `REPEATED RECORD` in BigQuery
  - In Redshift, each event and entity is loaded into its own table

You can add the same `product` entity to any other relevant event, such as `add_to_cart` or `view_product`. This simplifies analysis of product lifecycles, from initial product view to purchase.

Snowplow tracker SDKs provide built-in methods for tracking page views and screen views, along with many other kinds of events. For example, button clicks, form submissions, page pings (activity), or media interactions. The additional entities added depend on which Snowplow SDK you're using, and which [enrichments](/docs/pipeline/enrichments/) you've configured.

Snowplow provides out-of-the-box [dbt data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling and common analytics use cases.

### Tracking comparison

This table shows how example Segment tracking calls could be mapped to Snowplow tracking:

| Segment API Call | Segment Example                                                 | Snowplow Implementation                                                                                                                                                      |
| ---------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `track()`        | `track('Order Completed', {revenue: 99.99, currency: 'USD'})`   | Use one of the built-in event types, or define a custom `order_completed` schema containing `revenue` and `currency` properties, and track `trackSelfDescribingEvent`.       |
| `page()`         | `page('Pricing')`                                               | Use `trackPageView`. The tracker SDK will automatically capture details such as `title` and `url`.                                                                           |
| `screen()`       | `screen('Home Screen')`                                         | Use `trackScreenView`.                                                                                                                                                       |
| `identify()`     | `identify('user123', {plan: 'pro', created_at: '2024-01-15'})`  | Call `setUserId('user123')` to track the ID in all events. Attach a custom `user` entity with schema containing `plan` and `created_at` properties.                          |
| `group()`        | `group('company-123', {name: 'Acme Corp', plan: 'Enterprise'})` | No direct equivalent. Attach a custom `group` entity to your events, or track group membership changes as custom events with `group_joined` or `group_updated` schemas.      |
| `alias()`        | `alias('new-user-id', 'anonymous-id')`                          | No direct equivalent. Track identity changes as custom events with `user_alias_created` schema. Use `setUserId` to update the current user identifier for subsequent events. |

### Data validation

You've presumably defined Tracking Plans for your Segment implementation. The suggested Segment approach for creating a Tracking Plan is to first track some events, then use those events as the basis for a template. You choose how to handle future validation failures: filter out events that don't match their specification, allow with warnings, or pass after autocorrecting minor issues. Using a Tracking Plan is optional.

Schematic event specification isn't optional for Snowplow. Every event and entity is defined by a [JSON schema](/docs/fundamentals/schemas/), called a data structure. The data payload itself contains a reference to the specific data structure and version that defines it. The events are always validated as they're processed through the Snowplow pipeline, and events that fail validation are filtered out. There's no pass-with-warning or autocorrection feature.

Snowplow provides [monitoring](/docs/monitoring/) and alerting for failed events. You can choose to load failed events into a separate table in your warehouse, or to analyze them in temporary buckets. This strict approach ensures high data quality.

Snowplow also provides event data management tools for defining and managing tracking plans. Each Snowplow [tracking plan](/docs/fundamentals/tracking-plans/) contains a set of related event specifications. Each event specification has one event data structure, and any number of entity data structures.

You can use the Snowplow Console, API, or CLI to [define your tracking data structures](/docs/event-studio/tracking-plans/). For each event you can specify when it should be tracked, and which entities should be added. Once you've defined your event specifications, use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to automatically generate the tracking code snippets.

## Migration phases

We recommend using a parallel-run migration approach. This process can be divided into three phases:

1. Assess and plan
2. Implement and validate
3. Cutover and finalize

### 1. Assess and plan

Your existing Segment Tracking Plans will serve as the foundations for your new Snowplow implementation. Before migration, we advise auditing your tracking calls and Tracking Plans. Are all the events and properties still relevant?

Start documenting all downstream data consumers, such as BI dashboards, dbt models, ML pipelines, or marketing automation workflows, that may need updating after migration.

Export your Tracking Plans using one of these methods:

- Programmatic API export (recommended): use the [Segment Public API](https://segment.com/docs/api/public-api/) to retrieve the full JSON definition of each Tracking Plan
- Manual CSV download: download a CSV file from the Segment UI for a less nested but potentially incomplete inventory of your events and properties
- Warehouse inference: if you no longer have access to your Segment account, you can infer the events and properties from your warehouse data

You'll need to translate your Segment Tracking Plans into Snowplow [tracking plans](/docs/fundamentals/tracking-plans/). Some things to consider:

- Which platforms will you be tracking on? Different Snowplow tracker SDKs include different built-in event types. The [web](/docs/sources/web-trackers/) and [native mobile](/docs/sources/mobile-trackers/) trackers are the most fully featured.
- Which Tracking Plan events can be migrated to built-in Snowplow events, and which should be custom [self-describing events](/docs/fundamentals/events/#self-describing-events)?
- Are there sets of event properties used in multiple places that could be defined as [entities](/docs/fundamentals/entities/) instead?
- What's the best combination of event properties and entities to capture the same data as the non-`track` Segment events?

The goal is to create a set of JSON data structures for all your events and entities, organized into tracking plans and [event specifications](/docs/event-studio/tracking-plans/event-specifications/). The best way to import your new tracking plans into Snowplow is to use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/).

> **Info:** The Snowplow CLI includes an [MCP server](/docs/event-studio/programmatic-management/snowplow-cli/#mcp-server) to help you translate your Segment Tracking Plans into Snowplow tracking plans.

In this phase, you'll also need to decide what to do with historical data. There are two main choices:

- Coexistence: leave historical Segment data in existing tables. Write queries that `UNION` data from both systems, using a transformation layer (for example, in dbt) to create compatible structures.
- Unification: transform and backfill historical Segment data into the Snowplow format. This requires a custom engineering project to export Segment data, reshape it into the Snowplow enriched event format, and load it into the warehouse. The result is a unified historical dataset.

### 2. Implement and validate

This phase involves three main tasks:

- Set up your Snowplow infrastructure
- Implement and test your new tracking plan
- Set up data models

Follow the [Snowplow CDI getting started instructions](/docs/get-started/private-managed-cloud/) to set up your Snowplow infrastructure.

If you haven't done this yet, use the [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) to import your new tracking plans into Snowplow. You can also inspect and edit tracking plans using the Snowplow Console. They'll be available to the Snowplow pipeline for data validation on publishing. Use the Snowplow CLI or Console to publish.

Add Snowplow tracking in parallel with your existing Segment tracking:

- If you have a web platform, we recommend starting here

  - The [Snowplow Inspector browser extension](/docs/testing/snowplow-inspector/) is a useful manual testing tool
  - Start with non-critical pages or features

- Implement a tracker SDK, and track a small number of built-in events, such as [page views](/docs/sources/web-trackers/quick-start-guide/)

  - Use the Snowplow Inspector to confirm that the tracker is generating the expected events
  - Use [Snowplow Micro](/docs/testing/snowplow-micro/) to test and validate locally
  - Finally, confirm that the tracker can also send events to your warehouse

- Use [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to generate custom tracking code for your tracking plans

- Test and validate your custom tracking using Micro as before

- Gradually continue this process until you have a complete Segment and Snowplow dual tracking implementation

- Gradually roll out tracking to production

> **Info:** [Snowplow Micro](/docs/testing/snowplow-micro/) is a Dockerized local pipeline. It uses the same schema registry as your production pipeline, allowing developers to quickly confirm that the tracked events are valid.

Use the Snowplow data quality [monitoring tools](/docs/monitoring/) for confidence in your tracking implementation.

Set up your analytics:

- Configure the Snowplow [dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/) for initial modeling
- Update your existing data models to handle the new data structures
- Create queries to reconcile your old and new data
- Compare high-level metrics between systems e.g. daily event counts, or unique users

By the end of this phase, you'll have a validated Snowplow implementation that recreates your Segment tracking and modeling. The goal is to be confident that the new Snowplow pipeline is capturing all critical business logic correctly.

### 3. Cutover and finalize

In the final phase, you'll update downstream data consumers, and decommission Segment.

Use Snowplow [event forwarding](/docs/destinations/forwarding-events/) to integrate with third-party destinations. Update all your data consumers to query the new Snowplow data tables.

Remove the Segment tracking from your codebases. Keep the Segment sources active for a short time as a final fallback before fully decommissioning. Finally, close your Segment account.

---

# Automatically generated data models
> Generate optimized, analysis-ready data models directly from your tracking plans for views or incremental dbt models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/automatically-generated-data-models/

Automatic data model generation allows you to generate optimized models directly from your [tracking plans](/docs/event-studio/tracking-plans/) in [Snowplow Console](https://console.snowplowanalytics.com). You can use the models directly in your warehouse, or integrate them into existing dbt projects. The models are ready for analysis, BI tool, or reverse ETL use cases.

The autogenerated data models filter for only the events of interest, while flattening your event and entity data structures into a column for each property.

> **Note:** Automatically generated data models is not supported for Redshift.

## Key features

Choose exactly how you want the model to look, based on your use case.

### Configurable events and entities

You can specify which parts of the tracking plan to include:

- Event specifications
- Entities
- [Atomic Snowplow event properties](/docs/fundamentals/canonical-event/#common-fields)

### Automatic data flattening

To create a single wide table, the tool automatically flattens data structures into individual columns.

For event data structures, all properties are flattened into their own columns.

For entities, flattening depends on cardinality:

- Single entities, such as `user`, are flattened into separate columns, e.g., `user_id`, `user_email`
- Multiple entities, such as a `products` array, are stored as a single array column that you can unnest later

### Event specification inference

If you're using [Snowtype](/docs/event-studio/implement-tracking/snowtype/) to generate your tracking code, the model generator can filter for event specification IDs, using the attached event specification entity. This guarantees that all events were sent with the intent to implement this tracking plan, and ensures high data quality.

If you're not using Snowtype, the generated models perform more complex event filtering:

- Models include all rows matching the tracking plan definition
- Use "best effort" matching based on events, entities, cardinalities, and rules

These no-Snowtype models may include some unintended events that aren't part of the tracking plan. However, this approach allows access to historical data, and requires no adjustments to existing tracking implementations.

### Deployment options

You can choose to create a view or an incremental dbt model.

**Views** are best for lightweight use cases when you need immediate access to data, with minimal setup.

View statements can be run directly in your data warehouse without the need for additional tools. We recommend using views when your atomic events table is small, or when you won't be regularly querying the data. When using a view, ensure you reference the partition key of the atomic table to ensure your queries are performant and cost efficient.

**Incremental dbt models** are ideal for production pipelines when you want to materialize data efficiently, and integrate it into existing dbt projects. We recommend using incremental models when your atomic events table is large, or when you'll be regularly querying the data.

Choose to create a simple model without dependencies, or one based on the Snowplow [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) or [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) packages:

- **Simple** models are compatible with any dbt project with no package dependencies. These models will materialize all data for the model on the first run, while processing only new events for every subsequent run.
- **Models based on [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) or [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/)** are designed to work with our existing dbt packages. They build on `events_this_run` tables for efficient incremental processing.

This table compares the different options:

| Consideration             | View                                        | Simple incremental                           | Unified Digital or Normalize incremental   |
| ------------------------- | ------------------------------------------- | -------------------------------------------- | ------------------------------------------ |
| **Best for**              | Real-time data access, exploratory analysis | Standalone projects, new dbt implementations | Existing Snowplow dbt package users        |
| **Dependencies**          | None                                        | None                                         | Requires Snowplow dbt packages             |
| **Setup complexity**      | Minimal                                     | Minimal                                      | Moderate (variable configuration required) |
| **Processing efficiency** | Query-time processing                       | Standard incremental                         | Optimized with `events_this_run` tables    |
| **Integration**           | Independent operation                       | Independent operation                        | Coordinated with existing Snowplow models  |
| **Storage requirements**  | No additional storage                       | Materializes data                            | Materializes data                          |
| **Query performance**     | Depends on underlying data volume           | Fast (pre-computed)                          | Fast (pre-computed)                        |
| **Data freshness**        | Real-time                                   | Batch incremental updates                    | Batch incremental updates                  |

## Generating models

To generate a data model, find the **Data models** tab in [Console](https://console.snowplowanalytics.com/) > **Data collection** > **Tracking plans** > tracking plan details page.

![](/assets/images/landing-page-2482fec9450212638d2380bbe4775980.png)

Follow the instructions to create your model:

- Provide basic information such as name and warehouse destination
- Choose which event specifications, entities, and columns to include
- Confirm if you're using Snowtype
- Choose which kind of model to create

Once you've completed these steps, click **Save to tracking plan** to save the model. You can also download the model now.

![](/assets/images/finalize-f0ea01bc7bfd683b530342d592f882ad.png)

You can view the SQL at any time by choosing **View data model code** from the `⋮` menu.

![](/assets/images/created-f14206d5ab0b2aa9892a42b16160e1c2.png)

## Deploying models

How to deploy the model depends on the kind of model you've generated. The first step is the same for all: download the model from Console.

### Views

Deployment steps:

1. Configure the model schema to specify where to create the view
2. Run the model to create the view in your warehouse

### Simple incremental

Deployment steps:

1. Add the SQL file to your dbt project's models directory
2. Run the model

The initial run will materialize all historical data from your `atomic` events table. Subsequent runs will process only new events since the last run timestamp.

### Unified Digital or Normalize incremental

Deployment steps:

1. Add the SQL file to your dbt project's models directory

2. Configure dbt project variables:

   - Set `snowplow__start_date` in your dbt project variables
   - Configure `snowplow__backfill_limit_days` to control the volume of data processed per run

3. Run the model iteratively until the new model catches up to your existing models

4. Monitor progress through dbt logs and manifest updates

New models begin processing from `snowplow__start_date` since they're not yet tracked in the Snowplow Manifest table.

The model won't run if the required dependency isn't already installed in your dbt project.

---

# Modeling data
> An overview of the purpose of data modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/

At Snowplow, we define data modeling as the process of using business logic to, transform, aggregate, or otherwise process event level data. It occurs in the data warehouse, and allows you to extract value from your events data.

The Snowplow atomic data acts as an immutable log of all the actions that occurred across your digital products. The data model takes that data and transforms it into a set of derived tables optimized for analysis.

It is at the data modeling stage that you add your business logic to your Snowplow data; for example, how you define your marketing channels, what counts as a conversion, or how you segment your customers. As this logic is separate from the data you collect, you are able to update the logic and rerun it over your entire set of raw, immutable data to produce new derived data if you change your mind at a later stage. Not only can you add business logic to the data during modeling, you can also aggregate the same data into different tables based on the use cases the different teams in your organization have. Therefore, data modeling is a crucial step towards deriving insights from your data.

Snowplow provides [multiple dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/) to get you started modeling your data.

---

# Configure the Attribution data model
> Configure the Attribution dbt package for marketing attribution analysis with customizable conversion paths and channel groupings.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/

This page helps you configure the Snowplow Attribution dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.6.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `conversion_path_source` | The source (schema and table) of the  paths (touchpoints). By default it is the derived `snowplow_unified_views` table. Optionally it may be hardcoded with a string reference instead of a source with a schema.table or database.schema.table format. | `{{ source('derived', 'snowplow_unified_views') }}` |
| `conversions_source` | The source (schema and table) of your conversion events,  likely the conversions table generated by the Unified package. Optionally it may be hardcoded with a string reference instead of a source with a schema.table or database.schema.table format. | `{{ source('derived', 'snowplow_unified_conversions') }}` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `spend_source` | The source (schema and table) of your marketing spend source. Optional, needed for the ROAS calculation of the snowplow_attribution_overview. Should be changed to a table reference with `spend` by `channel` and/or `campaign` by `spend_tstamp` (which denotes a timestamp field) information. | `not defined` |
| `user_mapping_source` | The source (schema and table) of the user mapping table produced by the unified package. In case the user mapping table is not available, the paths_to_conversion macro needs to be overwritten in the dbt project where the package is referenced. | `{{ source('derived', 'snowplow_unified_user_mapping') }}` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `attribution_start_date` | The date to start processing events from in the package on first run or a full refresh, based on the cv_tstamp (conversion timestamp). | `2023-01-01` |
| `path_lookback_days` | Restricts the model to marketing channels within this many days of the conversion (values of 30, 14 or 7 are recommended). | `30` |
| `path_lookback_steps` | The limit for the number of marketing channels to look at before the conversion. | `0 (unlimited)` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `conversion_clause` | A string of sql to filter on certain conversion events. | `cv_value > 0 and ev.user_identifier is not null` |
| `conversion_window_days` | The last complete nth number of days (calculated from the last processed pageview within page_views_source) to dynamically update the conversion_window_start_date and end_date with. Will only apply if both variables are left as an empty string. | `30` |
| `conversion_window_end_date` | The end date in UTC for the window of conversions to include. It is only used for the drop and recompute report tables/views. | `""` |
| `conversion_window_start_date` | The start date in UTC for the window of conversions to include. It is only used for the drop and recompute report tables/views. | `current_date()-31` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `attribution_list` | List of attribution types to use for reporting. Can be at least one of: first_touch, last_touch, linear, position_based). | `['first_touch', 'last_touch', 'linear', 'position_based']` |
| `campaigns_to_exclude` | List of campaigns to exclude from analysis (empty to keep all campaigns). | `[ ] (no filter applied)` |
| `campaigns_to_include` | List of campaigns to include in the analysis (empty to keep all campaigns). | `[ ] (no filter applied)` |
| `channels_to_exclude` | List of channels to exclude from analysis (empty to keep all channels). For example, users may want to exclude the `Direct` channel from the analysis. | `[ ] (no filter applied)` |
| `channels_to_include` | List of channels to include in the analysis (empty to keep all channels). For example, users may want to include the `Direct` channel only in the analysis. | `[ ] (no filter applied)` |
| `consider_intrasession_channels` | If `false`, only considers the channel at the start of the session (i.e. first page view). If `true`, considers multiple channels in the conversion session as well as historically. | `true` |
| `conversion_hosts` | `url_hosts` to filter to in the data processing | `[] (no filter applied)` |
| `conversion_stitching` | This should be set to true if both the snowplow__conversion_stitching and snowplow__view_stitching variables are also enabled in the Unified package. If allowed it will consider the stitched_user_id field, not the user_identifier in the source data for more accurate results. | `false` |
| `enable_paths_to_non_conversion` | If `true`, enable the paths_to_non_conversion model, which is a drop and recompute table that may be needed for more in-depth attribution analysis (used in the `path_summary` table as well) | `false` |
| `enable_attribution_overview` | If `true`, enable the attribution_overview model, which is a view that creates a great source for BI tools to use for reporting | `true` |
| `path_transforms` | Dictionary of path transforms (and their argument) to perform on the conversion path (see the transform path options in our [model docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)). For `remove_if_not_all` and `remove_if_last_and_not_all` transformations, the argument has to be a non-empty array, for the others it should be null. | `{'exposure_path': null}` |


---

# Configure the Ecommerce data model
> Configure the Ecommerce dbt package for modeling transactions, carts, checkouts, and product performance.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/

This page helps you configure the Snowplow Ecommerce dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.9.3)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `number_checkout_steps` | The index of the checkout step which represents a completed transaction. This is required to enable working checkout funnel analysis, and has a default value of 4. | `4` |
| `categories_separator` | The separator used to split out your subcategories from your main subcategory. If for example your category field is filled as follows: `books/fiction/magical-fiction` then you should specify `"/"` as the separator in order for the subcolumns to be properly parsed. | `/` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `number_category_levels` | The **maximum** number of levels (depth) of subcategories that exist on your website for products. These subcategories are recorded in the category field of the product context, and should be separated using the separator which is defined below. For example, `books/fiction/magical-fiction` has a level of 3. The value is the number of columns that will be generated in the product tables created by this Snowplow dbt package. Please note that some products can have less than the maximum number of categories specified. | `4` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `enable_mobile_events` | A boolean whether to use the mobile contexts for mobile e-commerce events in the processing (based on the client session and screen view context). | `false` |
| `use_product_quantity` | Flag to use the product quantity value for the number of products in a transaction, instead of the count of events. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `ecommerce_event_names` | The list of event names that the Snowplow e-commerce package will filter on when extracting events from your atomic events table. If you have included any custom e-commerce events, feel free to add their event name in this list to include them in your data models. | `['snowplow_ecommerce_action']` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_ecommerce_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `carts_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `add_to_cart`, `remove_from_cart`, or `transaction` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `checkout_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `transaction` or `checkout_step` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `disable_ecommerce_carts` | Flag to exclude the `Snowplow E-commerce` [cart entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#cart-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_checkouts` | Flag to exclude the `Snowplow E-commerce` [checkout entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#checkout-step-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_page_context` | Flag to exclude the `Snowplow E-commerce` [page entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#ecommerce-page-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_products` | Flag to exclude the `Snowplow E-commerce` [product entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#product-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_transactions` | Flag to exclude the `Snowplow E-commerce` [transaction entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#transaction-entity) context in case this is disabled or not used in your tracking. | `false` |
| `disable_ecommerce_user_context` | Flag to exclude the `Snowplow E-commerce` [user entity](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/plugins/snowplow-ecommerce/#ecommerce-user-entity) context in case this is disabled or not used in your tracking. | `false` |
| `products_passthroughs` | Field(s) to carry through from the events table to the derived table. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the first event record in the session based on `derived_tstamp` then `dvce_created_tstamp`. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `transaction_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `transaction` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure Snowplow dbt packages
> Configure Snowplow dbt packages, including warehouse-specific settings for Postgres, Spark, Databricks, and BigQuery.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/

This page details general configurations that can apply across many of our packages. Each package has specific configuration variables that define how the models run, please see each child page for the specifics of each package.

## Variables

> **Tip:** Do not copy the project yaml from the package into your main project, this can lead to issues. Only add variables and model configurations that you wish to change from the default behavior. Each configuration page contains a UI to help to build your variables.

> **Warning:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>   snowplow_web:
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>   snowplow_mobile:
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

## Warehouse specific configurations

### Postgres

To optimism performance of large Postgres datasets you can create [indexes](https://docs.getdbt.com/reference/resource-configs/postgres-configs#indexes) in your dbt model config for columns that are commonly used in joins or where clauses. For example:

```yaml
# snowplow_web_sessions_custom.sql
{{
  config(
    ...
    indexes=[\{'columns': ['domain_sessionid'], 'unique': True}]
  )
}}
```

### Spark

For Spark environments, Iceberg is currently the supported file format for external tables. We have successfully tested this setup using both Glue and Thrift as connection methods. To use these models, create an external table from the Iceberg lake format in Spark and point your dbt model to this table.

Here's an example profiles.yml configuration for Spark using Thrift:

```yaml
spark:
  type: spark
  host: localhost
  method: thrift
  port: 10000
  schema: default
```

In your dbt\_project.yml, the file\_format is set to `iceberg` by default for Spark. While you can override this in your project's dbt YAML file to use a different file format, please note that Iceberg is currently the only officially supported format.

### Databricks

You can connect to Databricks using either the `dbt-spark` or the `dbt-databricks` connectors. The `dbt-spark` adapter does not allow dbt to take advantage of certain features that are unique to Databricks, which you can take advantage of when using the `dbt-databricks` adapter. Where possible, we would recommend using the `dbt-databricks` adapter.

#### Unity Catalog support

With the rollout of Unity Catalog (UC), the `dbt-databricks` adapter has added support in dbt for the three-level-namespace as of `dbt-databricks>=1.1.1`. As a result of this, we have introduced the `snowplow__databricks_catalog` variable which should be used **if** your Databricks environment has UC enabled, and you are using a version of the `dbt-databricks` adapter that supports UC. The default value for this variable is `hive_metastore` which is also the default name of your UC, but this can be changed with the `snowplow__databricks_catalog` variable.

Since there are many different situations, we've created the following table to help guide your setup process (this should help resolve the `Cannot set database in Databricks!` error):

|                                             | Adapter supports UC and UC Enabled                                                                    | Adapter supports UC and UC not enabled          | Adapter does not support UC                                                                          |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Events land in default `atomic` schema      | `snowplow__databricks_catalog = '\{name_of_catalog}'`                                                 | Nothing needed                                  | `snowplow__databricks_catalog = 'atomic'`                                                            |
| Events land in custom schema (not `atomic`) | `snowplow__atomic_schema = '\{name_of_schema}'` `snowplow__databricks_catalog = '\{name_of_catalog}'` | `snowplow__atomic_schema = '\{name_of_schema}'` | `snowplow__atomic_schema = '\{name_of_schema}'` `snowplow__databricks_catalog = '\{name_of_schema}'` |

#### Optimization of models

The `dbt-databricks` adapter allows our data models to take advantage of the auto-optimization features in Databricks. If you are using the `dbt-spark` adapter, you will need to manually alter the table properties of your derived and manifest tables using the following command after running the data model at least once. You will need to run the command in your Databricks environment once for each table, and we would recommend applying this to the tables in the `_derived` and `_snowplow_manifest` schemas:

```SQL
ALTER TABLE \{TABLE_NAME} SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true, delta.autoOptimize.autoCompact = true);
```

### BigQuery

As mentioned in the [Quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/), many of our packages allow you to specify which column your events table is partitioned on using the `snowplow__session_timestamp` variable. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:

```yml
vars:
  snowplow_mobile:
    snowplow__derived_tstamp_partitioned: false
```

---

# Configure the Fractribution data model
> Configure the legacy Fractribution dbt package for marketing attribution analysis. Superseded by the Attribution package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/fractribution/

> **Warning:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

This page helps you configure the Snowplow Fractribution dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.3.6)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `web_user_mapping_table` | The schema and table name of the snowplow web user mapping table, if different to default. | `derived.snowplow_web_user_mapping` |
| `conversions_source` | The source (schema and table) of your conversion events,  likely the conversions table generated by the Unified package. | `{{ source('atomic', 'events') }}` |
| `page_views_source` | The source (schema and table) of the derived `snowplow_web_page_views` table. | `{{ source('derived', 'snowplow_web_page_views') }}` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `path_lookback_days` | Restricts the model to marketing channels within this many days of the conversion (values of 30, 14 or 7 are recommended). | `30` |
| `path_lookback_steps` | The limit for the number of marketing channels to look at before the conversion. | `0 (unlimited)` |
| `conversion_window_days` | The last complete nth number of days (calculated from the last processed pageview within page_views_source) to dynamically update the conversion_window_start_date and end_date with. Will only apply if both variables are left as an empty string. | `30` |
| `conversion_window_end_date` | The end date in UTC for the window of conversions to include. | `""` |
| `conversion_window_start_date` | The start date in UTC for the window of conversions to include. | `current_date()-31` |
| `conversions_source_filter` | A timestamp field the conversion source field is partitioned on (ideally) for optimized filtering, when left blank `derived_tstamp` is used. | `blank` |
| `conversions_source_filter_buffer_days` | The number of days to extend the filter. | `1` |
| `use_snowplow_web_user_mapping_table` | `true` if you are using the Snowplow web model for web user mappings (`domain_userid` => `user_id`) | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `channels_to_exclude` | List of channels to exclude from analysis (empty to keep all channels). For example, users may want to exclude the `Direct` channel from the analysis. | `[ ] (no filter applied)` |
| `channels_to_include` | List of channels to include in the analysis (empty to keep all channels). For example, users may want to include the `Direct` channel only in the analysis. | `[ ] (no filter applied)` |
| `consider_intrasession_channels` | If `false`, only considers the channel at the start of the session (i.e. first page view). If `true`, considers multiple channels in the conversion session as well as historically. | `false` |
| `conversion_hosts` | `url_hosts` to filter to in the data processing | `[] (no filter applied)` |
| `path_transforms` | Dictionary of path transforms (and their argument, `null` if none) to perform on the full conversion path (see [udfs.sql file](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/path_transformations/create_udfs.sql)). | `{'exposure_path': null}` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `run_python_script_in_snowpark` | A flag for if you wish to run the python scripts using Snowpark. | `false` | Snowflake |
| `attribution_model_for_snowpark` | The attribution model to use when running in Snowpark; one of `shapley`, `first_touch`, `last_touch`, `position_based`, `linear`. See the [package docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-fractribution-data-model/#attribution-models) for more information. | `shapley` | Snowflake |


---

# Configure the legacy dbt packages
> Configuration for legacy Snowplow dbt packages including Web, Mobile, and Fractribution that have been superseded by newer packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/

> **Warning:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Configure the Mobile data model
> Configure the legacy Mobile dbt package for mobile event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/mobile/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

This page helps you configure the Snowplow Mobile dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_mobile_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-mobile/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_mobile_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "contexts_com_snowplowanalytics_snowplow_cli...` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "contexts_com_snowplowanalytics_snowplow_cli...` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_mobile_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `enable_geolocation_context` | Flag to include the Geolocation context (device latitude, longitude, bearing, etc.) columns in the models. | `false` |
| `enable_app_errors_module` | Flag to enable the app errors module (details relating to app errors that occur during sessions). | `false` |
| `enable_application_context` | Flag to include the Application context (app version and build) columns in the models. | `false` |
| `enable_mobile_context` | Flag to include the Mobile context (device type, OS, etc.) columns in the models. | `false` |
| `enable_screen_context` | Flag to include the Screen context (screen details associated with mobile event) columns in the models. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `platform` | A list of `platform`s to filter the events table on for processing within the package. | `['mob']` |
| `screen_view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `screen_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Web data model
> Configure the legacy Web dbt package for web event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

This page helps you configure the Snowplow Web dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.1)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

> **Tip:** When modifying the `session/user_identifiers` or using `session/user_sql` in the web package these will overwrite the `domain_sessionid` and `domain_userid` fields in tables, rather than being `session/user_identifier` as in the core utils implementation. This is for historic reasons to mitigate breaking changes. Original values for these fields can be found in `original_domain_session/userid` in each table.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `heartbeat` | Page ping heartbeat time as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `10` |
| `min_visit_length` | Minimum visit length as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `5` |
| `rfc_5646_seed` | Name of the model for the RFC 5646 (language) mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_rfc_5646_language_mapping` |
| `ga4_categories_seed` | Name of the model for the GA4 category mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_ga4_source_categories` |
| `geo_mapping_seed` | Name of the model for the Geo mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_web_dim_geo_country_mapping` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_web_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-web/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_web_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `list_event_counts` | A boolean whether to include a json-type (varies by warehouse) column in the sessions table with a count of events for each `event_type` in that session. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `total_all_conversions` | A boolean flag whether to calculate and add the `cv__all_volume` and `cv__all_total` columns. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/conversions/). | `false` |
| `conversion_events` | A list of dictionaries that define a conversion event for your modeling, to add the relevant columns to the sessions table. The dictionary keys are `name` (required), `condition` (required), `value`, `default_value`, and `list_events`. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/conversions/). | `""` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_stitching_id` | This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw `sql` expressions. | `user_id` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `page_view_stitching` | Determines whether to apply the user mapping to the page views table. Note this can be an expensive operation to do every run. One way to mitigate this is by running this update with less frequency than your usual run by enabling this variable only for that specific run. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `false` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `cwv_days_to_measure` | The number of days to use for web vital measurements (if enabled). | `28` |
| `cwv_percentile` | The percentile that the web vitals measurements that are produced for all page views (if enabled). | `75` |
| `limit_page_views_to_session` | A boolean whether to ensure page view aggregations are limited to pings in the same session as the `page_view` event, to ensure deterministic behavior. If false you may get different results for the same `page_view` depending on which sessions are included in a run. See the [stray page ping](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/#stray-page-pings) section for more information. | `true` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_web_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `ua_bot_filter` | Flag to filter out bots via the `useragent` string pattern match. | `true` |
| `enable_iab` | Flag to include the [IAB enrichment](/docs/enriching-your-data/available-enrichments/iab-enrichment/) data in the models. | `false` |
| `enable_yauaa` | Flag to include the [YAUAA enrichment](/docs/enriching-your-data/available-enrichments/yauaa-enrichment/) data in the models. | `false` |
| `enable_ua` | Flag to include the [UA Parser enrichment](/docs/enriching-your-data/available-enrichments/ua-parser-enrichment/) data in the models. | `false` |
| `enable_consent` | Flag to enable the [consent](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/consent-module/) module. | `false` |
| `enable_cwv` | Flag to enable the [Core Web Vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-web-data-model/core-web-vitals-module/) module. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `page_view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `page_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Media Player data model
> Configure the Media Player dbt package for tracking video and audio playback metrics with customizable variables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/

This page helps you configure the Snowplow Media Player dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.9.4)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `percent_progress_boundaries` | The list of percent progress values. It needs to be aligned with the values being tracked by the tracker. It is worth noting that the more these percent progress boundaries are being tracked the more accurate the play time calculations become. Please note that tracking 100% is unnecessary as there is a separate `ended` event which the model equates to achieving 100% and it also gets included automatically to this list, in case it is not added (you can refer to the helper macro `get_percentage_boundaries` ([source](https://snowplow.github.io/dbt-snowplow-media-player/#!/macro/macro.snowplow_media_player.get_percentage_boundaries)) for details). | `[10, 25, 50, 75]` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `valid_play_sec` | The minimum number of seconds that a media play needs to last to consider that interaction a valid play. The default is 30 seconds (based on the YouTube standard) but it can be modified here, if needed. | `30` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema": "contexts_com_snowplowanalytics_snowplow_medi...` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid", "prefix...` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `complete_play_rate` | The rate to set what percentage of a media needs to be played in order to consider that complete. 0.99 (=99%) is set as a default value here but it may be increased to 1 (or decreased) depending on the use case. | `0.99` |
| `max_media_pv_window` | The number of hours that needs to pass before new page_view level media player metrics from the `snowplow_media_player_base` table are safe to be processed by the model downstream in the `snowplow_media_player_media_stats` table. Please note that even if new events are added later on ( e.g. new `percentprogress` events are fired indicating potential replay) and the `snowplow_media_player_base` table is changed, the model will not update them in the media_stats table, therefore it is safer to set as big of a number as still convenient for analysis and reporting. | `10` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_media_player_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `surrogate_key_treat_nulls_as_empty_strings` | Passed through to `dbt_utils` to match legacy surrogate key behavior. | `true` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `enable_web_events` | Whether to use the web contexts for web media events in the processing (based on the web page context). | `true` |
| `enable_mobile_events` | Whether to use the mobile contexts for mobile media events in the processing (based on the client session and screen view context). | `false` |
| `enable_ad_quartile_event` | Set to `true` if [ad quartile events](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_quartile_event/jsonschema/1-0-0) are tracked during media ad playback. | `false` |
| `enable_media_ad` | Set to `true` if the [media ad context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers along with ad events. | `false` |
| `enable_media_ad_break` | Set to `true` if the [media ad-break context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers when ad breaks are tracked along with ad events. | `false` |
| `enable_media_player_v1` | Set to `true` if the [version 1 of the media player context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0) is enabled. This schema was used in our older media plugins on the JavaScript tracker. It is not tracked in the latest versions. | `false` |
| `enable_media_player_v2` | Set to `true` if the [version 2 of the media player context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers. | `true` |
| `enable_media_session` | Set to `true` if the [media session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0) is enabled. This is tracked by our latest JavaScript and mobile trackers (optional but enabled by default). | `false` |
| `enable_whatwg_media` | Set to `true` if the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `enable_whatwg_video` | Set to `true` if the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `enable_youtube` | Set to `true` if the [YouTube context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) is enabled. This variable is used to handle syntax depending on whether the context fields are available in the database or not. | `false` |
| `ad_views_passthroughs` | Field(s) to carry through from the events table to the `snowplow_media_player_media_ad_views` derived table. The field is from the ad view event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `base_passthroughs` | Field(s) to carry through from the events table to the `snowplow_media_player_base` and `snowplow_media_player_plays_by_page_view` derived tables. The field is from the media base event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `media_event_names` | A list of names for media events to include for processing. | `['media_player_event']` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Configure the Normalize data model
> Configure the Normalize dbt package for flattening and unnesting self-describing events and entities into dedicated tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/normalize/

This page helps you configure the Snowplow Normalize dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v0.4.1)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `events` | This is used internally by the packages to reference your events table based on other variable values and should not be changed. | `events` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `session_timestamp` | This determines which timestamp is used to process sessions of data. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `partition_tstamp` | This determines which timestamp is used to partition the derived tables. You need to make sure that this timestamp will be present in the flattened events table. | `collector_tstamp` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |


---

# Configure the Unified Digital data model
> Configure the Unified Digital dbt package with variables for sessions, users, and views using the interactive config generator.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/

This page helps you configure the Snowplow Unified Digital dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

> **Tip:** When modifying the `session/user_identifiers` or using `session/user_sql` in the unified package these will overwrite the `domain_sessionid` and `domain_userid` fields in tables, rather than being `session/user_identifier` as in the core utils implementation. This is for historic reasons to mitigate breaking changes. Original values for these fields can be found in `original_domain_session/userid` in each table.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `atomic_schema` | The schema (dataset for BigQuery) that contains your atomic events table. | `atomic` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `enable_web` | Flag to process web events throughout the package. | `true` |
| `enable_mobile` | Flag to process mobile events throughout the package. | `true` |
| `heartbeat` | Page ping heartbeat time as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `10` |
| `min_visit_length` | Minimum visit length as defined in your [tracker configuration](/docs/collecting-data/collecting-from-own-applications/javascript-trackers/web-tracker/tracking-events/#activity-tracking-page-pings). | `5` |
| `rfc_5646_seed` | Name of the model for the RFC 5646 (language) mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_rfc_5646_language_mapping` |
| `ga4_categories_seed` | Name of the model for the GA4 category mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_ga4_source_categories` |
| `geo_mapping_seed` | Name of the model for the Geo mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | `snowplow_unified_dim_geo_country_mapping` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |
| `grant_schema_usage` | Enables granting usage on schemas interacted with on a dbt run | `true` |
| `grant_select_to` | A list of users to grant select to all tables created by this package to. | `[]` |
| `sessions_table` | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. `{{ ref(\'snowplow_unified_sessions_custom\') }}`. Please see the [README](https://github.com/snowplow/dbt-snowplow-unified/tree/main/custom_example) in the `custom_example` directory for more information on this sort of implementation. | `"{{ ref( \'snowplow_unified_sessions\' ) }}"` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `start_date` | The date to start processing events from in the package on first run or a full refresh, based on `collector_tstamp` | `2020-01-01` |
| `list_event_counts` | A boolean whether to include a json-type (varies by warehouse) column in the sessions table with a count of events for each `event_type` in that session. | `false` |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `lookback_window_hours` | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | `6` |
| `total_all_conversions` | A boolean flag whether to calculate and add the `cv__all_volume` and `cv__all_total` columns. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/). | `false` |
| `conversion_events` | A list of dictionaries that define a conversion event for your modeling, to add the relevant columns to the sessions table. The dictionary keys are `name` (required), `condition` (required), `value`, `default_value`, and `list_events`. For more information see the [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/). | `""` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_stitching_id` | This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw `sql` expressions. | `user_id` |
| `session_stitching` | Determines whether to apply the user mapping to the sessions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `view_stitching` | Determines whether to apply the user mapping to the views table. Note this can be an expensive operation to do every run. One way to mitigate this is by running this update with less frequency than your usual run by enabling this variable only for that specific run. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `false` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `conversion_stitching` | Determines whether to apply the user mapping to the conversions table. Please see the [User Mapping](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) section for more details. | `true` |
| `custom_sql` | This allows you to introduce custom sql to the `snowplow_unified_events_this_run` table, which you can then leverage in downstream models. For more information on the usage, see the following page on the [advanced usage of the utils package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/). | `""` |
| `cwv_days_to_measure` | The number of days to use for web vital measurements (if enabled). | `28` |
| `cwv_percentile` | The percentile that the web vitals measurements that are produced for all page views (if enabled). | `75` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_unified_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |
| `use_refr_if_mkt_null` | Use the refr fields when the mkt fields are null for the definition of default channel group. This is a common case when for example a landing page has a redirect. | `false` |
| `enable_initial_checks` | Use this field to test the existance of all the seed models in the dwh and the existence of contexts that are needed based on your configuration. This is useful when changing the package configuration to ensure quick fails if the configuration is incorrect. | `false` |
| `allow_null_dvce_tstamps` | Use this field to bypass optimization which avoids long scans due to late arriving data. Warning, use this field only when using specific older versions of the AMP tracker or the Pixel tracker. | `false` |
| `license_accepted` | Use this field to accept the Snowplow Personal and Academic License or specify that you have a commercial agreement with Snowplow. | `false` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |
| `ua_bot_filter` | Flag to filter out bots via the `useragent` string pattern match. | `true` |
| `enable_conversions` | Flag to enable the conversions optional module. | `false` |
| `enable_deep_link_context` | Flag to include the deep link context data in the models. | `false` |
| `enable_geolocation_context` | Flag to include the geolocation data in the models. | `false` |
| `enable_iab` | Flag to include the [IAB enrichment](/docs/enriching-your-data/available-enrichments/iab-enrichment/) data in the models. | `false` |
| `enable_yauaa` | Flag to include the [YAUAA enrichment](/docs/enriching-your-data/available-enrichments/yauaa-enrichment/) data in the models. | `false` |
| `enable_ua` | Flag to include the [UA Parser enrichment](/docs/enriching-your-data/available-enrichments/ua-parser-enrichment/) data in the models. | `false` |
| `enable_browser_context` | Flag to include fields extracted from the [Browser context](/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/ootb-data/device-and-browser/index.md#browser-context) data in the models (JS tracker <3.17), can be used together with `snowplow__enable_browser_context_2` (supports 2nd major schema version) to coalesce the two. | `false` |
| `enable_browser_context_2` | Flag to include fields extracted from the 2nd major version of schemas related to the [Browser context](/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/ootb-data/device-and-browser/index.md#browser-context) data in the models (JS tracker (>=3.17), can be used together with `snowplow__enable_browser_context` for previous versions to coalesce the two. | `false` |
| `conversion_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the events_this_run table therefore taking all events. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output.  See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information.Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |
| `enable_app_errors` | Flag to include the mobile app error data in the models. | `false` |
| `enable_application_context` | Flag to include the app context data in the models. | `false` |
| `enable_consent` | Flag to enable the [consent](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/) module. | `false` |
| `enable_cwv` | Flag to enable the [Core Web Vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/) module. | `false` |
| `enable_mobile_context` | Flag to include mobile context data in the models | `false` |
| `enable_screen_context` | Flag to include the mobile screen data in the models. | `false` |
| `enable_screen_summary_context` | Flag to process the screen engagement information in the screen summary context on mobile events. | `false` |
| `has_log_enabled` | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to `false`. | `true` |
| `session_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `session_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first `page/screen_view` or `page_ping` event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `user_first_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. | `[ ] (no passthroughs)` |
| `user_last_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the sessions table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information. Note flat fields will be aliased with a `last_` prefix, dictionary provided aliases will not by default. | `[ ] (no passthroughs)` |
| `view_aggregations` | Aggregations to calculate as part of the the derived table. A list of dictionaries defining the type (sum, avg, min, max, count, countd), field to aggregate, and what to alias the column. | `[ ] (no aggregations)` |
| `view_passthroughs` | Field(s) to carry through from the events table to the derived table. The field is from the `page/screen_view` event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys `sql` for the SQL code to select the column and `alias` for the alias of the column in the output. See the [Passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) docs for more information. | `[ ] (no passthroughs)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `databricks_catalog` | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to `hive_metastore`) or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). | `hive_metastore` | Databricks |
| `derived_tstamp_partitioned` | Boolean to enable filtering the events table on `derived_tstamp` in addition to `collector_tstamp`. | `true` | Bigquery |
| `enable_load_tstamp` | Flag to include the `load_tstamp` column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | `true` | Redshift |
| `entities_or_sdes` | A list of dictionaries defining the `entity` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |
| `snowflake_lakeloader` | To use the models on an iceberg events table created via the lakeloader on Snowflake, this will need to be enabled due to different table structures. | `false` | Snowflake |


---

# Configure the Utils data model
> Configure the Utils dbt package with core macros and helper functions for Snowplow data modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/

This page helps you configure the Snowplow Utils dbt package. You can customize variables, generate configuration code, and set output schemas.

## Package configuration variables (v1.0.0)

This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your `dbt_project.yml` file.

> **Note:** All variables in Snowplow packages start with `snowplow__` but we have removed these in the below tables for brevity.

### Warehouse and tracker

| Variable | Description | Default |
| --- | --- | --- |
| `database` | The database that contains your atomic events table. | `target.database` |
| `events_schema` | The name of the schema that contains your atomic events. | `events` |
| `events_table` | The name of the table that contains your atomic events. | `events` |
| `dev_target_name` | The [target name](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) of your development environment as defined in your `profiles.yml` file. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `dev` |

### Operation and logic

| Variable | Description | Default |
| --- | --- | --- |
| `allow_refresh` | Used as the default value to return from the `allow_refresh()` macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the [Manifest Tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) section for more details. | `false` |
| `backfill_limit_days` | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#package-state) section for more details. | `30` |
| `days_late_allowed` | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | `3` |
| `session_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_sessionid"}]` |
| `session_sql` | This allows you to override the `session_identifiers` SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the `session_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `user_identifiers` | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the `schema` value `atomic`, then this refers to a field found directly in the atomic `events` table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the `schema` field. It will use the specified value in the `field` key as the field name to access. For Redshift/Postgres, using the `schema` key the package will try to find a table in your `snowplow__events_schema` schema with the same name as the `schema` value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details. | `[{"schema" : "atomic", "field" : "domain_userid"}]` |
| `user_sql` | This allows you to override the `user_identifiers` SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the `user_identifiers` variable, please see the [Custom Identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) section for more details on how to do that. | `""` |
| `session_timestamp` | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | `collector_tstamp` |
| `custom_sql` | This allows you to introduce custom sql to the `snowplow_base_events_this_run` table, which you can then leverage in downstream models. For more information on the usage, see the following page on the [advanced usage of the utils package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/). | `""` |
| `max_session_days` | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | `3` |
| `package_name` | The name of the package you are running this macro under. This has implications for your `manifest` table. | `snowplow` |
| `session_lookback_days` | Number of days to limit scan on `snowplow_web_base_sessions_lifecycle_manifest` manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | `730` |
| `upsert_lookback_days` | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the [Snowplow Optimized Materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) section for more details. | `30` |

### Entities (contexts), filters, and logs

| Variable | Description | Default |
| --- | --- | --- |
| `app_id` | A list of `app_id`s to filter the events table on for processing within the package. | `[ ] (no filter applied)` |

### Warehouse-specific

| Variable | Description | Default | Warehouse |
| --- | --- | --- | --- |
| `entities_or_sdes` | A list of dictionaries defining the `context` or `self-describing` event tables to join onto your base events table. Please use the tool below or see the section on [Utilizing custom contexts or SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for details of the structure. | `[]` | Redshift |


---

# Adding fields to a derived table
> Add custom fields to Snowplow derived tables using passthrough fields, custom aggregations, or custom derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/

You may often wish to add fields onto the derived tables within our packages; perhaps there is some additional dimension you need for your analysis, or some extra calculation you wish to perform. In most cases this is relatively straight forward to do, and even in the most complex case you don't need to alter the logic for the derived table itself.

## Option 1: Passthrough fields

Where possible, if the field you want to add is attached to the original event for that table (e.g. the page/screen view for views) then you can make use of the [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) feature in our packages.

## Option 2: Custom aggregations

If you require aggregation at the level of the table (e.g. session identifier for sessions) you can make use of the [custom aggregations](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/) feature in our packages that support it.

## Option 3: Custom derived table

If none of the above options are suitable, the best approach is to build a custom version of our derived table, and read from the existing [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) table to then add in your additional fields.

> **Tip:** For Redshift, if you need any additional self-describing event or entity fields in the events this run table, check out the [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/#custom-entities--events) page for how to add these.

As an example, let's say we want to add some field to the sessions table in the unified package, that for simplicity is just going to be an `is_test` calculation based on the app\_id.

### Disable the derived model in the package

The first step is to disable the derived model in the package itself. You do this in your top level project yaml:

```yaml
...
models:
  snowplow_unified:
    sessions:
      snowplow_unified_sessions:
        +enabled: false
```

### Add a new derived model

Next we add a new model to replace the one we just disabled; the easiest thing to do is just copy the contents of the model we just disabled. While you could name this anything, it's easiest for downstream use cases to keep the original name

```jinja2
{{
  config(
    tags=['snowplow_unified_incremental', 'derived'],
    ...
  )
}}

select *
  {% if target.type in ['databricks', 'spark'] -%}
  , DATE(start_tstamp) as start_tstamp_date
  {%- endif %}
from {{ ref('snowplow_unified_sessions_this_run') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

### Modify the new model as needed

```jinja2
{{
  config(
    ...
  )
}}

select *
  , case when app_id like '%_test' then 'test' else 'prod' end as app_type
  {% if target.type in ['databricks', 'spark'] -%}
  , DATE(start_tstamp) as start_tstamp_date
  {%- endif %}
from {{ ref('snowplow_unified_sessions_this_run') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

> **Tip:** If you are using BigQuery, you should look at the [combine column versions](https://github.com/snowplow/dbt-snowplow-utils?tab=readme-ov-file#combine_column_versions-source) macro we provide to automatically combine minor versions of your schemas in the model.

### (Optional) Backfill the model

Follow the steps to [backfill models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) the model if you want this field to be populated historically.

---

# Additional SQL on events this run
> Add custom SQL to the events this run table using the snowplow__custom_sql variable for calculated fields and context extraction.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/additional-sql-on-events-this-run/

There may be times when the [events this run table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) requires additional fields on it for you to make use of to [add to your derived tables](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/). It is not recommended to alter this model directly, as it is a core part of the packages, but you can use the `snowplow__custom_sql` variable in packages that support it to add custom sql into the `select` block of the events this run model.

To find out if your package supports this, check the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page.

## Using `snowplow__custom_sql`

The `snowplow__custom_sql` variable is passed as-is through to the `select` block of the [`events_this_run` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/19bfd655fea1338f28cd6b2f8ca5863cc137aac7/macros/base/base_create_snowplow_events_this_run.sql#L39). Because of where this is passed in, fields such as the session and user identifiers are not available at this point, but any other field should be. In the case of Redshift any self-describing events or entities provided in the [`snowplow__entities_or_sdes` variable](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) will be available with the provided prefix and table alias.

Say you wanted to add a calculated field based on your app id, you would set this as such:

```yml
...
vars:
  snowplow_<package_name>:
    snowplow__custom_sql: "case when app_id like '%_test' then 'test' else 'prod' end as app_type"
...
```

### Utilizing custom contexts or SDEs

Suppose you have a custom context called `contexts_com_mycompany_click_1` which contains a `click_id` that you want to concat with Snowplow's `domain_sessionid`. If you want to make this available in the events this run table directly you could do the following:

**Databricks & Snowflake:**

```yml
vars:
    ...
    snowplow__custom_sql: "CONCAT(com_mycompany_click_1[0].click_id, '_', domain_sessionid) as click_session_id"
    ...
...
```

**BigQuery:**

```yml
vars:
    ...
    snowplow__custom_sql: "CONCAT(com_mycompany_click_1[safe_offset(0)].click_id, '_', domain_sessionid) as click_session_id"
    ...
...
```

**Redshift & Postgres:**

```yml
vars:
    ...
    snowplow__custom_sql: "com_mycompany_click_1.click_id || '_' || domain_sessionid as click_session_id"
    ...
...
```

***

### Adding multiple fields

If you'd like to add multiple lines of SQL, you can do that as well by making this string a multi-line string, ending each non-final line with a comma. You can do that as follows:

```yml
vars:
    ...
    snowplow__custom_sql: |
        com_mycompany_click_1.click_id || '_' || domain_sessionid as click_session_id,
        case when app_id like '%_test' then 'test' else 'prod' end as app_type
    ...
...
```

---

# Daily aggregate model
> Build daily aggregate tables in Snowplow dbt packages by scanning derived tables with date range filters for accurate aggregations.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/daily-aggregate-table/

Because of the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages, all of the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) tables contain events (or a higher level) for only the sessions being processed in that run. For this reason, it is not possible to build daily aggregate tables directly off of any this run tables.

To better understand why, consider a late arriving event for a session that started 2 days ago; because we would reprocess all events for this session our sessions this run table would contain a record for that session with a start date 2 days ago, but all other sessions for that day would already have been processed and would **not** be in this table. If you try to get a count of sessions, you'll report that there was only 1 session that day. You could try to be clever with incremental additions, but this session may have already been partially processed so may or may not have already been counted.

The only way to accurately do daily aggregates in our packages is to calculate these off the _derived tables_, i.e. the persistent tables in the package. This would be inefficient to scan the whole table every time, so we can be smart and use the _range of dates_ in the [events this run table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) to identify which dates we need to rescan, thus reducing the compute cost.

## A simple example

Let's say we want to do a simple daily aggregates model built in the Unified package - we want to keep a count of the number of sessions and the average engaged time per day. This table would be 1 row per day, and we would need to read from the `snowplow_unified_sessions` _derived_ table for the reasons discussed above.

The skeleton of the model would be to make an incremental model, ensure we add the [optimized upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/), checking for new events, and selecting from the sessions model.

Note there is no need to [tag](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/#tagging-models) the model as this is not reading directly from a this run table.

```jinja2
{{
  config(
    materialized='incremental',
    unique_key='session_date',
    upsert_date_key='session_date',
    tags=['derived'],
    snowplow_optimize = true
  )
}}

-- depends_on: {{ ref('snowplow_unified_events_this_run') }}

{# Get the range of dates processed in the current run to limit the later scan #}
{% if is_incremental() %}
{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref('snowplow_unified_events_this_run'),
                                                                           'derived_tstamp',
                                                                           'derived_tstamp') %}
{% endif %}

select
  date(start_tstamp) as session_date,
  count(*) as num_sessions,
  avg(engaged_time_in_s) as average_engaged_time
from {{ ref('snowplow_unified_sessions') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
{% if is_incremental() %}
  and start_tstamp >= date({{ lower_limit }})
  and start_tstamp <= {{ dateadd(datepart="day", interval=1, from_date_or_timestamp= upper_limit) }}
{% endif %}
group by 1
```

> **Tip:** Notice that we are filtering on the `start_tstamp` column from the sessions table, because this is the partition/sort key for that table. Make sure you used the right column for the derived table you are querying.

We need to add a dependency on events this run to ensure dbt runs everything in the right order, we do this by adding the `-- depends_on: {{ ref('snowplow_unified_events_this_run') }}` comment. Next we get the limits for the current run and those are the dates we then use to work out what range we have to scan the sessions table on. On the first run, it will do a full scan to backfill the table from the existing derived sessions data.

You could add additional buffers around the limits if required, but make sure that you are always processing full days of data from the sessions table (at least up to the current max timestamp). You can also extend this as needed to add other fields, joins, or anything else you require.

If you need weekly/monthly aggregates, the ideas are the same, just make sure you are always scanning and reprocessing a whole period in each run.

---

# Incorporating non-Snowplow data
> Incorporate external data sources into Snowplow derived tables by joining non-Snowplow data with custom derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/incorporating-non-snowplow-data/

There may be times when you wish to add some fields to our derived tables that are not coming from the events table itself but from your own external data sources.

Our packages are designed to process events associated to sessions, based on newly incoming events from the `atomic.events` table. Allowing for incorporating external data into the processing by default is not planned so far, not only because there can be a myriad of ways how different users would want to link their data to our events/sessions, but also because the package would then need some sort of governance over the data source being readily available for the period to be processed, which it has no impact over. However, there is a way to accomplish that through custom models.

**Example Scenario 1:** Imagine you run an e-commerce site, and you are only tracking the products\` SKU, but you have an internal table with richer information about the particular product. You would like to add some extra information to your views table through the custom page\_view context.

**Example Scenario 2:** Wanting to incorporate additional user based information into the sessions table with the use of the external User Profiles table.

In both cases what you could do is follow the recommendations on [adding a custom derived table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/#option-3-custom-derived-table), basically by disabling the package provided derived table, copying the content of the existing derived table as is, which is normally just a select \* from the upstream this\_run table.

> **Info:** You might want to add extra safety measures to avoid misaligned data between the package and the table you want to bring data from. Consider using a pre-hook that you can add to the config section of your derived table for instance. Update schedules would also need to be synced for a hassle-free run and consider the impact when there are changes, refreshes to the source table, in case you need to run a full or partial refresh afterwards.

When modifying the model you would typically want to add left joins using the key that makes sense for your use case (e.g. user\_identifier, session\_identifier, view\_id etc.) and list the additional field(s) which you would like to be incorporated. As long as you keep a `this_run` table as a basis and select everything from it, and also tag your custom model as `snowplow_{{package_name}}_incremental` it will become part of the run and the package will keep everything in sync through the manifest table.

Bear in mind that if you make changes to the Sessions table, you might break the Users table in the Unified Digital package depending on the changes as the Users table is built on top of the Sessions table (e.g. it is generally safe to add more fields (be careful about name clashes), though you might also need to add them as a [passthrough field](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) for them to show up in the Users table, but probably not to remove them, best to test it first), otherwise they are safe to update independently.

> **Info:** You can inspect the lineage graph to verify this if you are unsure by using dbt's built in data lineage feature through dbt docs. We also update that for each of our latest releases on github: <https://snowplow.github.io/dbt-snowplow-unified/#!/overview?g_v=1>

---

# Example custom dbt models
> Example custom models for Snowplow dbt packages including derived tables, aggregations, and sessionization patterns.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/

The following pages contain specific examples for building custom models to do specific things. They are meant to help you understand how to go about building certain types of custom models and are by no means an exhaustive list of what you can do, and you can mix and match elements to fit your needs.

If you would like to see an example of how this implementation works in practice, check out our [example dbt project](https://github.com/snowplow-incubator/dbt-example-project/tree/main/custom_event_table_unified) on Github. It demonstrates how the Unified Digital package can be used as a base for customizations, allowing you modify its behavior without forking the package (e.g. macro overwrite, adding custom data models, overwriting variables etc.).

---

# New derived model
> Create new incremental derived models in Snowplow dbt packages building from events this run with optimized upserts.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/new-derived-model/

We have always seen our packages as a starting point for your analysis and insight generation; there to get you started and give you an idea of what you can do, while providing the framework to build your own models on top to meet your bespoke needs. Assuming that customizing the existing derived tables does not suit your needs the next steps is to create a custom derived model.

As you are building this for yourself, there is no need to follow our usual this run -> derived approach, and you can just build an incremental model on top of events this run!

> **Note:** Any custom derived model must build only to session-level or _lower_ because of the [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages. For example you can't build a table that aggregates per day using the standard sessionisation as not all events from a given day are always processed in the same run, at least not on any of the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/), see the [Daily Aggregate Table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/daily-aggregate-table/) for how to do this.

## A simple example

Let's say we want to do a very simple model built in the unified package - we want to keep a count of the number of each type of event per session in a table. This table would be 1 row per session+event name, and can build incrementally as it's at a lower level than session.

The skeleton of the model would be to make an incremental model, ensure we add the [optimized upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/), [tag](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/#tagging-models) the model, checking for new events, and selecting from [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run).

```jinja2
{{
  config(
    materialized='incremental',
    unique_key='session_identifier',
    upsert_date_key='start_tstamp',
    tags=['snowplow_unified_incremental', 'derived'],
    snowplow_optimize = true
  )
}}

select *
    ...
from {{ ref('snowplow_unified_events_this_run') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
```

Then it is simply a case of writing the actual model content itself, which in this case is very straight forward. You may have a more complex model, perhaps bringing in data from another source or model in your dbt project, but the idea is the same. Note that we need a date key to be able to use our optimized upsert.

```jinja2
{{
  config(
    materialized='incremental',
    unique_key='session_identifier',
    upsert_date_key='start_tstamp',
    snowplow_optimize = true
  )
}}

select
    session_identifier,
    min(derived_tstamp) as start_tstamp,
    event_name,
    count(*) as event_vol
from {{ ref('snowplow_unified_events_this_run') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_unified') }} --returns false if run doesn't contain new events.
group by 1, 3
```

Follow the steps to [backfill](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) the model. Make sure to add any additional config for optimizing your tables based on your warehouse.

---

# Using different sessionisation
> Use custom session identifiers in Snowplow dbt packages including daily aggregates and alternative sessionization logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-different-sessionisation/

Using a different sessionisation is a rare use case for our packages, but can be done in 2 different ways. The use case for each approach is often different so we treat them differently on this page.

## Tweaking the session identifiers

If you only need to slightly alter the identifiers, such as using an entity value where it exists instead of just the default fields, you can do this by following the details in the [custom identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/) page to alter this via the existing variables of our packages.

## Using the incremental logic for all custom models

The more large-scale use case is when you wish to take advantage of our [incremental sessionisation logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), but have no need for any of the other derived tables in our packages. A common use case for this is when you want to do daily aggregates and reporting, in this case your "session" identifier is actually the day of the event.

For this purpose, rather than disable all the models you don't need, it would be easier to use the `base` macros and tables at the root of your project, or stand up a standalone project. For this purpose we have the [dbt-template](https://github.com/snowplow-incubator/dbt-template) repository which has all the core logic of our packages set up, including all the variables and models.

Once you have forked/copied this project, you will need to replace all instances of `<YOUR_PACKAGE_NAME>` across the package with the name you want for this project. This includes file names, not just a find and replace in the files.

You can then make changes to the variables as required, for example the following changes would give you a daily "sessionisation" to allow you to easily do incremental daily reporting:

```diff
...
vars:
- <YOUR PACKAGE NAME>:
+ my_package:
    ...
    snowplow__app_id: []
-   # snowplow__session_sql: 'e.domain_sessionid'
+   snowplow__session_sql: "DATE(e.derived_tstamp)"
    # snowplow__user_sql: 'e.domain_userid'
    # snowplow__custom_sql: ''
    ...
...
```

---

# Using multi-valued entities
> Extract and use multi-valued entities in Snowplow dbt custom models with warehouse-specific unnesting techniques.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/

When building [custom derived models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/new-derived-model/) you may find yourself needing to use a multi-valued entity, i.e. an entity with multiple entries in the array. These fields should not be extracted as part of the [event this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table because it would lead to duplicate rows, and so are only extracted as part of the custom model where you are either keeping the rows at that level or aggregating back up.

In this page we will just showcase the SQL required, including some of our specialist macros, to extract out these entities into a 1-row per value table, for you to then use in any specific model.

## SQL to extract each value

**Snowflake:**

To make use of a multi-valued entity in BigQuery, you can make use of the `LATERAL FLATTEN` function.

```jinja2
SELECT
    ...
    r.value:myField::varchar,  -- field will be variant type so important to cast
    ...
FROM
    {{ ref('snowplow_<package_name>_base_events_this_run') }} t,
    LATERAL FLATTEN(input => t.contexts_my_entity_1) r
```

**BigQuery:**

To make use of a multi-valued entity in BigQuery, you can make use of the `unnest` function. Note that you will need to do this for each version of the context.

```jinja2
SELECT
    ...
    my_ent.my_field AS my_field,
    ...
FROM
    {{ ref('snowplow_<package_name>_base_events_this_run') }}
LEFT JOIN
    unnest(contexts_my_entity_1_0_0) AS my_ent -- left join to avoid discarding events without values in this entity
```

**Databricks:**

To make use of a multi-valued entity in BigQuery, you can make use of the `LATERAL VIEW EXPLODE` function.

```jinja2
SELECT
    ...
    my_ent.my_field,
    ...
FROM
    {{ ref('snowplow_<package_name>_base_events_this_run') }}
    LATERAL VIEW EXPLODE(contexts_my_entity_1) AS my_ent
```

**Redshift:**

As the fields in Redshift are part of a different table, we need to join that table onto the events this run model. We do this in 3 steps:

1. Get the limits for the current run to reduce the scan on the entity table
2. Use a CTE and macro to get the records from the entity table, adding fields needed to deduplicate
3. Join this onto events this run, using a specific set of conditions to manage duplicates but keep _genuine_ same-valued records.

We use the [`snowplow_utils.get_sde_or_context` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/utils/get_sde_or_context.sql), which takes five arguments:

- `schema`: the schema in your warehouse that the table is in
- `identifier`: the table name for the entity
- `lower_limit`: the lower limit of the event time to scan
- `upper_limit`: the upper limit of the event time to scan
- `prefix`: the prefix to apply to all column names
- `single_entity`: if it is a single-valued entity

In this case, we want to make sure to set `single_entity` to false as we are explicitly using a multi-valued entity.

```jinja2
{# Get the limits for the run to reduce the scan on the entity table#}
{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref('snowplow_<package_name>_base_sessions_this_run'),
                                                                            'start_tstamp',
                                                                            'end_tstamp') %}

{# Use the macro as part of a CTE which will select the relevant data, prefix the columns, and add the de-dupe fields #}
with {{ snowplow_utils.get_sde_or_context(var('snowplow__atomic_schema', 'atomic'), 'contexts_my_entity_1', lower_limit, upper_limit, 'my_ent', single_entity = false) }},

select
    a.*,
    b.my_ent_my_field
from {{ ref('snowplow_<package_name>_base_events_this_run') }} a
left join contexts_my_entity_1 b on
    a.event_id = b.yauaa__id  -- match the event id between the tables, note the dobule underscore
    and a.collector_tstamp = b.yauaa__tstamp -- match the collector timestamp between the tables, note the double underscore
    and mod(b.yauaa__index, a.event_id_dedupe_count) = 0 -- ensure one version of each potentially duplicated entity in context
```

***

This table can then be used to do any further analysis required, either in the same model or for use in another model.

---

# High volume optimizations
> Optimize Snowplow dbt packages for high-volume data processing with reduced columns, ephemeral models, and tuned incremental logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/

For users with very high data volumes (>100M daily events) you may find that, even with our [optimized upserts](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) and [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), the package is still slow in any given run or the processing cost is high. There are a few specific things you can do to help optimize the package even further, which require various levels of effort on your part. In general we have taken the decision to not do these things as part of the "normal" deployment of our packages as there is a trade-off for each one and in the vast majority of use cases they aren't required.

## Tune the incremental logic parameters

Our [incremental sessionization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default will look back and reprocess 6 hours of data; if you are confident there is no delay in data loading, or if you are using the `load_tstamp` as the `snowplow__session_timestamp` then there is much less need to have such a big lookback window, decreasing this to 1 hour will greatly reduce volumes of data for regular (\~hourly) package runs.

Decreasing the backfill limit days will only impact on backfill runs so once models are up to date this will have little impact. Decreasing the upsert lookback days, or the session lookback days can have benefits but come at the risk of duplicates making it into the manifest or derived tables so do this at your own risk.

## Reduce the number of columns written + ephemeral this run

> **Danger:** These configurations have not been fully tested and leave you at risk of issues occurring when we make upgrades to our packages. Continue at your own risk.

### Derived Tables

In general we try to only write the columns into [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/), and therefore derived tables, that are populated and relevant to that table. However, there may by many columns in these tables that you don't require and as as cloud warehouses are usually [columnar warehouses](https://en.wikipedia.org/wiki/Column-oriented_DBMS) this leads to more compute required to write these columns, and more storage to keep and maintain the metadata for them.

Rather than edit the this run tables directly, as this could leave you out of sync with our updates, we recommend changing the materialization of all this run models (except the events and base sessions) to be `ephemeral`, this stops the incremental data being first written as a table before being merged into the derived table. Doing this may not be the best option if you re-use any of these tables in multiple custom models as it will have to run them each time.

Next instead of selecting all columns from your now ephemeral this run models, disable our derived model and build your own to explicitly select just the columns that you require. See the [adding fields to derived table](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/adding-fields-to-derived-table/) example for how to do this.

Your derived table will now be writing and storing fewer columns, and the this run tables will optionally not be written at all.

### Events this run

The [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) by design contains every column that your atomic events table does including all self-describing events and entities (except in Redshift where these must be specified by a variable to be joined on). This can mean a very large amount of columns and data that you never use have to be read and written in every single run.

In general, it is a bad idea to alter the events this run table, as it can lead to issues elsewhere in the package, but it is also one of the largest places to reduce the amount of read/written data in a given run. The easiest thing to do in this case is to disable the events this run table from the package and duplicate the model into your top level project. Note the model must have the exact same name as the one you disabled, and you will need to duplicate many of the variables from the package into your project yaml to ensure they are available in the correct scope.

Nearly all our packages will have the same structure for this model; a macro is called, this is used within a CTE, and then we do some further SQL on this CTE. In your case, you could select only the columns required for **all** downstream models explicitly within this model

#### Example from unified

**Project yaml**

Note that any variables used in the model need to be made available in your project scope

```yaml
models:
  snowplow_unified:
    base:
      scratch:
        snowplow_unified_base_events_this_run:
          +enabled: false

vars:
  my_project_name:
    snowplow__session_timestamp: ...
    ...
```

**Details**

snowplow\_unified\_base\_events\_this\_run (copy)

We have removed the vast majority of the code from the original model code. The changes are the addition of the `default_cte` and then the selection of specific columns below.

```jinja2
...
with base_query as (
  {{ base_events_query }}
)

default_cte as (
  select
    *
    -- extract commonly used contexts / sdes (prefixed)
    {{ snowplow_unified.get_web_page_context_fields() }}
    {{ snowplow_unified.get_iab_context_fields() }}
    {{ snowplow_unified.get_ua_context_fields() }}
    {{ snowplow_unified.get_yauaa_context_fields() }}
    {{ snowplow_unified.get_browser_context_fields() }}
    {{ snowplow_unified.get_screen_view_event_fields() }}
    {{ snowplow_unified.get_session_context_fields() }}
    {{ snowplow_unified.get_mobile_context_fields() }}
    {{ snowplow_unified.get_geo_context_fields() }}
    {{ snowplow_unified.get_app_context_fields() }}
    {{ snowplow_unified.get_screen_context_fields() }}
    {{ snowplow_unified.get_deep_link_context_fields() }}
    {{ snowplow_unified.get_app_error_event_fields() }}
    {{ snowplow_unified.get_screen_summary_context_fields() }}


  {% if var('snowplow__enable_consent', false) -%}
    {{ snowplow_unified.get_consent_event_fields() }}
    {{ snowplow_unified.get_cmp_visible_event_fields() }}
  {% endif -%}

  {% if var('snowplow__enable_cwv', false) -%}
    {{ snowplow_unified.get_cwv_fields() }}
  {% endif -%}
  from base_query
)

select
  ... -- your specific columns here
from
  default_cte
```

> **Tip:** In the case of the unified package, there are two event this run tables, normally we recommend editing the latter one via the macro that creates it, but in this case you need to edit the upstream one to remove the columns as early as possible.

---

# Custom dbt models
> Create, update, and remove custom models in Snowplow dbt packages with incremental, drop+recompute, and this run model types.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/

The Snowplow packages are designed to be easily customized or extended within your own dbt project by building your own custom models. The standard models we provide (base, page/screen views, sessions and users) are not designed to be modified by you.

In general there are 3 types of custom models you may end up building:

1. **Drop+recompute/view models**: these models build from the derived tables but require a Drop+recompute each time they are built. This could include things like daily aggregations or rolling views.
2. **Incremental models**: these models build from the existing (or custom) [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) tables(s) and are incrementally upserted to each run. This could include things like a custom sessions table with additional user information. These are the most common types of custom models.
3. **This run models**: these models build from the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table that contains all columns for events being (re)processed in the run. These can be used to feed custom incremental models.

Alternatively, if you wish to only use the events this run logic and build everything on top of this yourself, you may be better looking at the [using different sessionisation example](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-different-sessionisation/) for how to set up a new package from scratch.

The best way to learn is by doing, so check out the [Examples](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/) section for specific custom models.

## Guidelines & Best Practice

The standard models carry out the heavy lifting in establishing an incremental structure and providing the core logic for the most common aggregation use cases. It also allows custom models to be plugged in without impeding the maintenance of standard models.

The following best practices should be followed to ensure that updates and bug fixes to the model can be rolled out with minimal complication:

- Custom models should not modify any of the tables generated by the Snowplow packages i.e. the scratch, derived or manifest tables.
- The logic for custom SQL should be idempotent, and restart-safe - i.e. it should be written in such a way that a failure mid-way, or a re-run of the model will not change the deterministic output.
- Redshift/Postgres users: be careful about joining mutli-valued entities to any events table. Deduplication logic needs to be applied to avoid many-to-many joins. See the [using multi-valued entities example](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) for how to manage this.

In short, the standard models can be treated as the source code for a distinct piece of software, and custom models can be treated as self-maintained, additive plugins - in much the same way as a Java package may permit one to leverage public classes in their own API, and provide an entry point for custom programs to run, but will not permit one to modify the original API.

The [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) and derived tables are considered part of the 'public' class of tables in this model structure, and so we can give assurances that non-breaking releases won't alter them. The other tables may be used in custom SQL, but their logic and structure may change from release to release, or they may be removed.

## Custom model usage

An overview of the detail provided below is available in this summary table:

| Model Type           | Materialization             | Requires Tagging | Requires backfilling | Requires specific refreshing | Use is run with new events | Use `snowplow_optimize` |
| -------------------- | --------------------------- | ---------------- | -------------------- | ---------------------------- | -------------------------- | ----------------------- |
| Drop+recompute/ view | `table`/`view`              | ❌                | ❌                    | ❌                            | ❌                          | ❌                       |
| Incremental          | `incremental`               | ✅                | ✅                    | ✅                            | ✅                          | ✅                       |
| This Run             | `table`/`view`/ `ephemeral` | ✅                | ❌                    | ❌                            | ❌                          | ❌                       |

### Tagging Models

Any incremental model, or one that builds from the events this run table, in the package requires tagging so that it can be tracked in the manifest table as part of the [incremental sessionisation logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/). If you do not correctly tag the model you may get an error, or your package may get stuck in State 2.

The model(s) should be tagged with `snowplow_<package>_incremental`; the easiest way to achieve this is to place any custom models of this type in a specific directory in your project and tag the whole folder in your project yaml:

```yml
models:
  your_dbt_project:
    snowplow_<package>_custom_models:
      +tags: snowplow_<package>_incremental #Adds tag to all models in the 'snowplow_<package>_custom_models' directory
```

Note that because Drop+recompute models are not tagged, they will not be run when using the `snowplow_<package>` selector.

### Using `is_run_with_new_events`

Incremental type custom models, those that reference a `_this_run` type table should make use of the `is_run_with_new_events` macro to only process the table when new events are available in the current run.

```jinja2
select
  ...
from {{ ref('snowplow_<package>_<table>_this_run') }}
where {{ snowplow_utils.is_run_with_new_events('snowplow_<package>') }} --returns false if run doesn't contain new events.
```

### Using `snowplow_optimize`

To make use of our [efficient upsert](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) you should add `snowplow_optimize=true` to your custom model config for incremental models, setting an appropriate `upsert_date_key` and `unique_id`.

```jinja2
{{
  config(
    materialized='incremental',
    upsert_date_key='my_date_field',
    unique_key='unique_id_field',
    snowplow_optimize=true,
    ...
  )
}}

select
  ...
```

### Backfilling Custom Models

See our page on [backfilling](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/).

### Refreshing Custom Models

See our page on [full or partial refreshes](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/).

### Retiring Custom Models

If you want to retire a custom model, you should delete the models from your project or [disable](https://docs.getdbt.com/reference/resource-configs/enabled#disable-a-model-in-a-package-in-order-to-use-your-own-version-of-the-model) the models.

There is no need to remove the models from the `snowplow_<package>_incremental_manifest` manifest table. The packages identifies **enabled** models tagged with `snowplow_<package>_incremental` within your project and selects these models from the manifest in order to calculate the state of the package as described above.

> **Danger:** Do **NOT** just use `--exclude` on the retired models from your job in production. Currently the packages is unable to identify which models are due to be executed in a given run. As a result, if you exclude a model the package will get stuck in [State 3](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-3-models-out-of-sync) and continue to attempt to sync your excluded with the remaining models.

## Tips for developing custom models

> **Note:** Most of these tips apply only to Incremental type custom models

While developing custom models you may benefit from the following:

1. Minimizing the amount of data being processed to reduce cost & run time.
2. Use recent events from your events table to ensure you have all the latest contexts and event types available.
3. BigQuery only: Automatic handling of evolving schemas for custom contexts and unstructured events.

### 1. Reducing Costs

By setting `snowplow__backfill_limit_days` to 1 in your `dbt_project.yml` file you will only process a days worth of data per run.

We have provided the `get_value_by_target` macro to dynamically switch the backfill limit depending on your environment i.e. dev vs. prod, with your environment determined by your target name:

```yml
vars:
  snowplow_<package>:
    snowplow__backfill_limit_days: "{{ snowplow_utils.get_value_by_target(
                                            dev_value=1,
                                            default_value=30,
                                            dev_target_name='dev') }}"
```

### 2. Using Recent Data

This can be achieved by setting `snowplow__start_date` to a recent date. If a custom model only models newly tracked events, you should set this to the date that data started to be produced instead of your initial pipeline date. To dynamically change the start date depending on your environment, you can use the following:

```yml
vars:
  snowplow_<package>:
    snowplow__start_date: "{{ snowplow_utils.get_value_by_target(
                                      dev_value=snowplow_utils.n_timedeltas_ago(1, 'weeks'),
                                      default_value='2020-01-01',
                                      dev_target_name='dev') }}"
```

### 3. Handling of schema evolution

> **Info:** As your schemas for such custom contexts and unstructured events evolve, multiple versions of the same column will be created in your events table e.g. `custom_context_1_0_0`, `custom_context_1_0_1`. These columns contain nested fields i.e. are of a datatype `RECORD`. When modeling Snowplow data it can be useful to combine or coalesce each nested field across all versions of the column for a continuous view over time.
>
> The snowplow-utils package provides the [combine\_column\_versions](https://github.com/snowplow/dbt-snowplow-utils#combine_column_versions-source) macro, which will automatically coalesce the fields within every version of the specified column. This mitigates the need for you to update your models every time a new column version is created.
>
> Please refer to the [snowplow-utils](https://github.com/snowplow/dbt-snowplow-utils) docs for the full documentation on these macros.

---

# Custom Attribution implementations
> Customize the Attribution dbt package with custom conversion periods, sources, and attribution overview logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/custom-implementations/

> **Warning:** Make sure you upgrade to [v.0.4.0](https://github.com/snowplow/dbt-snowplow-unified/releases/tag/0.4.0) of Unified Package if you would like to use the v0.2.0 of Attribution Package

This page describes how to customize the Attribution model.

## Customizing the Attribution Overview

The Attribution Overview, though enabled by default, is abstracted away into its own macro: `attribution_overview()` encouraging the package users to have a think about customizing the logic as it is heavily generalized by nature. Let's break down what is happening there by default so that the users can understand it better and be able to make customizations that fits their analysis best.

When analyzing marketing attribution it is always going to be a specific analysis with respect to time: there is always a conversion period, which sets the main timeframe of the analysis, during which conversions happen.

By default, the model expects a spend source and the model aggregates that (it takes the sum of total cost by spend and by channel and campaign). However, associating the spend data with the attribution data is taken dynamically by default based on the timestamp of the conversion relative to the spend. The view by default takes into account spend data that happened before conversion but no less than 90 days before conversion. This is where it makes sense add custom logic, adjusting the period to filter on the investment period relative to the conversion period in question depending on business requirements.

```sql
    from spend_with_unique_keys s
    inner join {{ ref('snowplow_attribution_campaign_attributions') }} c
    on c.campaign = s.campaign and s.spend_tstamp < cv_tstamp
    and s.spend_tstamp > {{ snowplow_utils.timestamp_add('day', -90, 'cv_tstamp') }}
```

Next the model is aggregating data from both the channel and campaign attributions table by conversion, summing up all the different kinds of attribution (first touch, last touch etc.)

Here comes the second time based filter which you might want to adjust in case you run your package incrementally as default. If you adjusted the `snowplow__conversion_window_start_date` and `snowplow__conversion_window_end_date` and ran a specific conversion period, then you can leave it as per default, but for incremental package runs you might want to change the `snowplow__conversion_window_days` variable to extend the analysis period (by default it means it will process last 30 number of days since the last path being available in the paths source (most likely the last page view from the snowplow\_unified.views table)).

```jinja2
  {% if not var('snowplow__conversion_window_start_date') == '' and not var('snowplow__conversion_window_end_date') == '' %}
    cv_tstamp >= '{{ var("snowplow__conversion_window_start_date") }}' and cv_tstamp < '{{ var("snowplow__conversion_window_end_date") }}'
  {% else %}
    cv_tstamp >= {{ snowplow_utils.timestamp_add('day', -var("snowplow__conversion_window_days"), last_processed_cv_tstamp) }}
  {% endif%}
```

Finally, the model takes the aggregated conversion level data and does a series of unions to calculate metrics by attribution type (first touch, last touch etc.) Here again you might want to adjust what specific metrics you are interested in, add your own etc.

## Running the Attribution Analysis with a specific conversion period only

By default the Attribution package is designed to be run incrementally.

In case you only need ad hoc analysis studying only a specific conversion window and perhaps looking at a larger lookback period to gather a broader user journey, for cost saving purposes you might want to define a set period every time the model is run (e.g. on a monthly basis).

In that case, you can simply overwrite both the `snowplow__conversion_window_start_date` and `snowplow__conversion_window_end_date` variables to decide on the period. In this case the default `snowplow__conversion_window_days` variable will be disregarded by the package.

Bear in mind that it needs a full-refresh every time you process the package:

```bash
 dbt run --select snowplow_attribution --full-refresh --vars 'snowplow__allow_refresh: true'
```

## Bringing your own sources - customizing the paths\_to\_conversions() macro

There may be cases where you opt for not relying on the snowplow-unified based source tables. The main thing to care about is that you have a joined user\_id in both the conversion and path source.

There are two ways to go about this:

#### 1. Align your data sources in a way to fit the package logic

- create a view on top of your source data to make sure you align the expected input fields

- have the common user\_id field called `stitched_user_id` in both your path and conversions source view

- make sure to set the variable `snowplow__conversion_stitching` to True (it means the package will rely on the `stitched_user_id` fields for the joins)

#### 2. Align the package logic to work with your data sources

The `paths_to_conversions()` macro to your package is provided in the form of a dispatch macro that you can simply overwrite by adding a new macro with the same name in your dbt project.

The default structure of the macro is the following:

```sql
--
with paths as (
  ...
)

, conversions as (
  ...
)

, string_aggs as (
  ...
)

, arrays as (
  ...
)

{{ transform_paths('conversions', 'arrays') }}

select ...
```

You would most probably need to touch the `paths` and `conversions` cte to adjust how your data is getting processed incrementally from your custom sources. The `string_aggs` cte creates the individual paths the user travelled through to get to the conversion. These path strings are then converted to `arrays` in the subsequent cte.

After that there is a separate macro called `transform_paths` that is being called to provide the sql for the path transformations. It is quite complex, we would not be recommending to overwrite these.

After that the macro just selects the final paths\_to\_conversion data that will be used for the incremental update for a given run for this derived table.

---

# Snowplow Attribution dbt package
> Marketing attribution analysis with Snowplow dbt package supporting first-touch, last-touch, position-based, and linear attribution models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/

> **Info:** You will need Unified [version 0.4.0](https://github.com/snowplow/dbt-snowplow-unified/releases/tag/0.4.0) to use Attribution version 0.2.0.

​ **The package source code can be found in the [snowplow/dbt-snowplow-attribution repo](https://github.com/snowplow/dbt-snowplow-attribution), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-attribution/#/overview/snowplow_attribution).** ​

> **Tip:** Although there is no direct dependency, by default the attribution package is dependent on the `snowplow_unified_views` as well as the `snowplow_unified_conversions` table created by the [snowplow\_unified](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) dbt package. As long as the field names are matched you can use other source tables, too.

## Overview

​ The Snowplow Attribution dbt package produces a set of incremental derived tables to provide a basis for in-depth **Marketing Attribution Analysis** on an ongoing basis. It allows you to attribute the value of a conversion to one or more channels and campaigns depending on the conversion pathway. As a result, it becomes possible to determine the revenue per pathway (channel or campaign), as well as ROAS (Return On Ad Spend - the amount of revenue that is earned for every dollar spent on advertising) once you have cost data for each marketing channel or campaign.

This package consists of a series of dbt models that produce the following tables:

- `snowplow_attribution_paths_to_conversion`: Customer id and the paths the customer has followed that have lead to conversion
- `snowplow_attribution_campaign_attributions`: By campaign path and conversion level incremental table that attributes the conversion value based on various algorithms
- `snowplow_attribution_channel_attributions`: By channel path and conversion level incremental table that attributes the conversion value based on various algorithms
- `snowplow_attribution_overview`: The user defined report view (potentially showing ROAS)
- `snowplow_attribution_path_summary`: For each unique path, a summary of associated conversions, optionally non-conversions and revenue
- `snowplow_attribution_paths_to_non_conversion`: Customer id and the paths the customer has followed that have not lead to conversion. Optional drop and recompute table, disabled by default.

In the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section you will find a step-by-step guide on how to operate the package as a whole.

![Package flow](/assets/images/attribution_package_flow_light-d9ba3bb672a244629720bba7291a9895.png)

### Attribution Models

The package currently offers 4 different attribution models all of which are calculated by default into both the incremental base tables and the report tables. If you would like to filter on them for reporting you can do so with `var('snowplow__attribution_list')`. Please note, however, that they will still be available for the incremental base tables.

- `first_touch`: Assigns 100% attribution to the first channel in each path.
- `last_touch`: Assigns 100% attribution to the last channel in each path.
- `position_based`: The first and last channels get 40% of the credit each, with the remaining channels getting the leftover 20% distributed evenly.
- `linear`: Assigns attribution evenly between all channels on the path. ​

![Data processing model for the attribution package](/assets/images/attribution_models_light-9fcb879888bac68c8da798cde1c97e9d.png)

## What is Marketing Attribution?

Marketing attribution determines which marketing tactics are contributing to sales or conversions by analysing the marketing touchpoints a consumer encounters on their journey to purchase. The aim is to determine which channels and marketing campaigns had the greatest impact on the decision to convert. There are many popular attribution models used by marketers which give insight into customers' behaviors, more specifically the pathways they took to purchase the product or service. This allows marketing teams to improve ROAS by changing marketing strategies and campaigns.

## Benefits of using our package

The package was implemented with a **glassbox philosophy** in mind, making it very **transparent** for data consumers how the data is transformed. It is processed **all in SQL** inside your warehouse, there are no external dependencies e.g. extra Python packages or black-box ML algorithms.

It is also very **flexible**, there are no limitations on the touchpoints (it works on all channels or campaigns). You can use ad spend data from any 3rd party data sources. Although there are no direct dependencies on data sources either, it is recommended to be used together with the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) dbt package for a seamless setup without the need to overwrite default dbt macros.

It also does the heavy-lifting for you and provides you with **incremental data models** reducing unnecessary costs even if you would like to regularly analyze the data based on a shifting time window in your reports (e.g. last 30 days).

## How to do Marketing Attribution with this package?

The purpose of this package is to allow an incremental, efficient way to do marketing attribution without having to read all your data every time.

In the below guide we will walk you through the data transformation process step-by-step in order for you to see how the source data changes downstream. This will give you and your team a transparent and easy-to-understand way to see how this package will lead you to valuable insights.

We also provide the **[Marketing Attribution Data Model Pack](/docs/modeling-your-data/visualization/attribution-modeling/)** specifically to help your analysis by visualizing the output in the form of interactive dashboards as well as letting you capture datasets for comparison. It works in tandem with the package and will auto-update daily in case your package has been processed since then.

## Sources you are going to need

### 1. Conversion source

You will also need a table where the conversion events are stored. If you use the snowplow\_unified model and configure conversions to be modelled you will have this information in your `derived.snowplow_unified_conversions` table by default.

In case you store your conversion events elsewhere in the data warehouse (e.g. because those are transactional data that you decide you do not want to process through Snowplow), you can mutate that source table into the format of the conversions table (expand the Conversion formatting requirements section below for a list of all the conversion table columns) generated by the unified package, store it as a sql view, and refer to that in the variable `snowplow__conversions_source` (in the format of `schema.view_name`).

You could also union multiple sources (e.g. one transactional event data for automatic conversion events not dictated by user actions and then the rest, ending up in the unified\_conversions table). Bear in mind though, that the incremental logic is dictated based on the last processed conversion in your conversion source, you need to make sure those tables are synced before the model is run.

Please bear in mind, if you would like to calculate the attribution for conversion events not processed by Snowplow, you would need to make sure these events contain a user\_id type field which the package can rely on to find previous website visits for it to be able to attribute them to these conversions. Otherwise, your overall revenue will be more than the total attributed revenue.

**Conversion formatting requirements**

Fields to use (including the name) when creating the view:

- `cv_id`: a unique identifier for each conversion, can be a dummy UUID
- `event_id`: a unique identifier for the conversion event, can be dummy UUID
- `session_identifier` (OPTIONAL): can be left out if it is the only source or NULL for unions with the unified.conversions table
- `user_identifier`: the user id field to be used for identifying pre conversion website visits
- `user_id`: the user id field to be used for identifying pre-conversion website visits
- `stitched_user_id`: the user id field to be used for identifying pre conversion website visits (this will be used if conversion\_stiching is enabled)
- `cv_value`: the conversion amount
- `cv_tstamp`: a timestamp field when the conversion took place
- `dvce_created_tstamp` (OPTIONAL): a timestamp field, can be left out if it is the only source or the same as the cv\_tstamp for unions with the unified.conversions table
- `cv_type`: you can name the types of conversions to differentiate them, if needed, it should not be left NULL/blank
- `cv_tstamp_date`: (DATABRICKS/SPARK ONLY): the date of the `cv_tstamp` value

### 2. Path source

You will also need a source table to track your user journey / path with fields to be able to classify your marketing channels. The perfect table for this is your `derived.snowplow_unified_views` table (default):



Alternatively, you could use the `derived.snowplow_unified_sessions` table as well, but bear in mind that this will mean only the first channel/campaign will be counted within a session, and you will have to make sure the correct field reference is used in the `snowplow_attribution_paths_to_conversion` table by overwriting the `paths_to_conversion()` macro in your project (e.g `first_page_urlhost` instead of `page_urlhost`).

As for campaigns,

> **Tip:** To fully finish the config you might need to overwrite the `channel_classification()` macro. In case your classification logic for attribution analysis needs to be the same as the one already configured in the snowplow\_unified model you can simply leave the default macro which refers to that field. See the [channel\_group\_query macro](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/) for more details.

### 3. Channel spend information (optional, but recommended)

You most likely have a warehouse with marketing (ad) spend information by channel and date, something like this:



To make it flexible to use what you already have, we suggest creating a view on top of the table you have, rename the fields that the model will use and add that view reference in `var('snowplow__spend_source')`. For more details on how to do this check out our [Quick Start Guide](/tutorials/attribution/intro/).

### 4. User mapping source

In order to be able to attribute the value of a conversion to one or more channels and campaigns it is essential to have a shared user identifier field between the conversion events and the preceding website visits. If the Unified dbt package is used for the conversion and conversion path source, there are two out-of-the box stitching options to use:

1. By enabling BOTH the `snowplow__view_stitching` the `snowplow__conversion_stitching` in the unified package, both sources will get the latest `stitched_user_id` field based on logged in user activity (see details [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/)). By setting the `snowplow__conversion_stitching` variable in the attribution package to true, the package will consider that field to take as a basis for both the conversion and the path source. Please note that this could potentially become a costly operation due to the volume of the snowplow\_unified\_views table, as after each run all the id fields will be updated by the latest mapping.

2. Alternatively (default from v.0.3.0 onwards), the package will rely on the `snowplow__user_mapping_source` defaulted to the `derived.snowplow_unified_user_mapping` table which will get used to take the latest logged in business user\_id field per user\_identifier, if available, otherwise it will keep the user\_identifier value for both the conversions and conversion path source before the two are joined.

3. When using custom sources or solutions, this logic could be overwritten in the dbt project using the `snowplow_attribution_paths_to_conversion()` dispatch macro. In case the optional `snowplow_attribution_paths_to_non_conversion` table is also in use, and it needs custom stitching logic, it is advised to disable it and create a custom model with the desired stitching logic.

Another option is if you alias your unique user id field as `stitched_user_id` in both your custom path and conversion source and set the `snowplow__conversion_stitching` variable to true, then the macro doesn't need to be altered but it will rely on these fields when linking conversions to paths.

### One-off setup

Once you have the sources ready you need to adjust the package settings that fit your business logic. This has to be done once, otherwise your data might be misaligned due to the incremental logic or you would need to run a full-refresh:

**Decide on your sessionization logic**

By default, Snowplow only considers the first pageview of a session important from an attribution point of view and disregards campaign information from subsequent page\_views. Google Analytics, on the other hand separates session as soon as campaign information is given. Set `var('consider_intrasession_channels')` variable to false in case you would like to follow Snowplow's logic, not GA's. If you opt for this calculation consider changing the `var('snowplow__conversion_path_source')` to `{{ target.schema }}.derived.snowplow_unified_sessions` for performance benefits.

Use the variable `snowplow__conversion_hosts` to restrict which hosts to take into account for conversions.

**Filter unwanted / wanted channels & campaigns**

You can specify a list of channels for the variable `snowplow__channels_to_exclude` to exclude them from analysis (if kept empty all channels are kept). For example, users may want to exclude the 'Direct' channel from the analysis.

You can also do the opposite, filter on certain channels to include in your analysis. You can do so by specifying them in the list captured within the variable `snowplow__channels_to_include`.

You can do either for campaigns, too, with the `snowplow__channels_to_exclude` and `snowplow__channels_to_include` variables.

#### **Transform paths**

In order to reduce unnecessarily long paths you can apply a number of path transformations that are created as part of user defined functions automatically in your warehouse by the package.

In order to apply these transformations, all you have to do is to define them in the `snowplow__path_transforms` variable as a dictionary. In case of `remove_if_last_and_not_all` and `remove_if_not_all` transformations, the transformation name is the key and a non-empty array is the value.

For other transformations (`exposure_path`, `first_path`, `unique_path`), no additional parameter is required, you can just use `null` as values. For more details on how to do this, check out the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/) E.g.: `{'exposure_path': null, 'remove_if_last_and_not_all': ['channel_to_remove_1', 'campaign_to_remove_1', 'campaign_to_remove_2']}` Please note that the transformations are applied on both campaign and channel paths equally.

**Path transform options**

Paths to conversion are often similar, but not identical. As such, path transforms reduce unnecessary complexity in similar paths before running the attribution algorithm. The following transformations are available: ​

1. **`exposure_path (default)`**: the same events in succession are reduced to one: `A → A → B` becomes `A → B`, a compromise between first and unique

2. **`unique_path`**: all events in a path are treated as unique (no reduction of complexity). Best for smaller datasets (small lookback window) without a lot of retargeting

3. **`first_path`**: keep only the first occurrence of any event: `A → B → A` becomes `A → B`, best for brand awareness marketing

4. **`remove_if_last_and_not_all`**: requires a channel to be added as a parameter, which gets removed from the latest paths unless it removes the whole path as it is trying to reach a non-matching channel parameter: E.g target element: `A` path: `A → B → A → A` becomes `A → B`

A common requirement is to take the **`last non-direct clicks`** only into account during attribution analysis. If you use the `remove_if_last_and_not_all` path transform with `direct` as the parameter to remove it from the end of the path (unless they are all direct) the `last-touch` attribution results should show the outcome you need. You can also filter out the direct channels altogether with the `snowplow__channels_to_exclude` variable.

5. **`remove_if_not_all`**: requires a channel to be added as a parameter, which gets removed from the path altogether unless it would result in the whole path's removal: E.g target element: `A` path: `A → B → A → A` becomes `B`

Apart from this, you can also restrict how far in time (`var('snowplow_path_lookback_days')`) and steps (`var('snowplow_path_lookback_steps')`) you want to allow your path to go from the actual conversion event.

> **Warning:** Redshift users starting from 0.6.0 are only allowed to have one path transformation (e.g. either `exposure_path` or `first_path`). If using `remove_if_last_and_not_all` or `remove_if_not_all`, only single-item arrays are allowed. In case of other supported targets there are no such limitations, multiple path transformations can be applied, if neeeded.

#### **Other, macro based setup**

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__channel_classification() %}
>     case when (...)
> {% endmacro %}
> ```

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-attribution/blob/main/macros/channel_classification.sql) macro is used to classify each marketing touchpoint from your path source deciding which marketing channel it was driven by. Be default it expects an already classified field called `default_channel_group`. In case the Attribution package is used together with the Unified Digital package, it is advisable to classify your pageviews and sessions upstream using the `channel_group_guery()` macro. For an in-depth guide on how to achive this check the [channel group query](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/) macro documentation.

#### `attribution_overview` macro

> Defines the sql for the view called attribution overview which provides the main report calculating the return on advertising spend (ROAS). In order to do that you would need a marketing spend table as an additional source which will contain the spend information by channel and or campaign with a timestamp to filter on the period. If you are happy with the logic of the macro, you can just define your spend table in the `snowplow__spend_source` variable and let it run.

#### `paths_to_conversion` macro

> Macro to allow flexibility for users to modify the definition of the paths\_to\_conversion incremental table. By default the incremental table uses the `snowplow_unified_views` and `snowplow_unified_conversions` source tables but this macro allows for flexibility around that. You can also restrict which conversion events to take

#### **Adding tests (Optional)**

Please note that the Unified data model allows nulls on `user_identifier` field, but the Attribution package does not, by default we filter out nulls within the `paths_to_conversion` macro when joining on the `path_source` and also on the `conversion_source` using the `conversion_clause` variable, which by default includes the null filter. We encourage users to add their own tests in the appropriate level (e.g. in the derived.snowplow\_unified\_views level) to make sure you don't accidentally exclude those events without user\_identifier that are null (e.g. due to a tracking issue).

## Output

### Incremental data models to prepare for attribution analysis:

1. The **`derived.snowplow_attribution_paths_to_conversion`** model will aggregate the paths the customer has followed that have lead to conversion based on the path transformation and other limitations such as the path\_lookback\_step or path\_lookback\_days variable i.e. it combines the path and conversion source tables to produce an outcome. It looks like this:

| customer\_id | cv\_tstamp       | revenue | channel\_path                              | channel\_transformed\_path                 | campaign\_path                                    | campaign\_transformed\_path |
| ------------ | ---------------- | ------- | ------------------------------------------ | ------------------------------------------ | ------------------------------------------------- | --------------------------- |
| user\_id1    | 2022-06-11 15:33 | 20.42   | Direct                                     | Direct                                     | camp1 > camp2                                     | camp1 > camp2               |
| user\_id2    | 2022-07-30 11:55 | 24      | Direct > Direct                            | Direct                                     | camp1                                             | camp1                       |
| user\_id3    | 2022-06-08 20:18 | 50      | Direct > Direct                            | Direct                                     | camp2 > camp1                                     | camp2 > camp1               |
| user\_id1    | 2022-07-25 07:52 | 140     | Organic\_Search > Direct > Organic\_Search | Organic\_Search > Direct > Organic\_Search | Campaign 2 > Campaign 2 > Campaign 1 > Campaign 1 | Campaign 2 > Campaign 1     |

2. The **`derived.snowplow_attribution_channel_attributions`** unnests the paths from paths\_to\_conversion into their separate rows and calculates the attribution amount for that specific path step for each of the sql based attribution models:

| composite\_key        | event\_id | customer\_id | cv\_tstamp              | cv\_total\_revenue | channel\_transformed\_path       | channel         | source\_index | path\_length | first\_touch\_attribution | last\_touch\_attribution | linear\_attribution | position\_based\_attribution |
| --------------------- | --------- | ------------ | ----------------------- | ------------------ | -------------------------------- | --------------- | ------------- | ------------ | ------------------------- | ------------------------ | ------------------- | ---------------------------- |
| id1\_Video0           | event\_1  | user\_id1    | 2023-07-07 13:05:55.000 | 200                | Video                            | Video           | 0             | 1            | 200                       | 200                      | 200                 | 200                          |
| id2\_Display\_Other0  | event\_2  | user\_id2    | 2023-07-19 04:27:51.000 | 66.5               | Display\_Other > Organic\_Search | Display\_Other  | 0             | 2            | 66.5                      | 0                        | 33.25               | 33.25                        |
| id3\_Organic\_Search1 | event\_2  | user\_id2    | 2023-07-19 04:27:51.000 | 66.5               | Display\_Other > Organic\_Search | Organic\_Search | 1             | 2            | 0                         | 66.5                     | 33.25               | 33.25                        |

3. The **`derived.snowplow_attribution_campaign_attributions`** does the same, only for campaigns not channels.

### Drop and recompute reporting tables/views:

1. The **`derived.snowplow_attribution_path_summary`** shows the campaign/channel paths and the associated conversions (and optionally non-conversions, if the `path_to_non_conversions` table is enabled through its related variable `enable_path_to_non_conversions`)

| transformed\_path                     | conversions | non\_conversions | revenue |
| ------------------------------------- | ----------- | ---------------- | ------- |
| Direct                                | 3           | 25               | 94.42   |
| Organic\_Search                       | 2           | 26               | 206.5   |
| Referral > Direct                     | 0           | 1                | 0       |
| Video                                 | 1           | 2                | 200     |
| Organic\_Search > Paid\_Search\_Other | 0           | 2                | 0       |

2. The view called **`derived.snowplow_attribution_overview`** is tied to a dispatch macro of the same name which lets you overwrite it in your project, if needed. Given you specify your `var('snowplow__spend_source')` it will calculate the ROAS for you for each channel and campaign:

| path\_type | attribution\_type | touch\_point    | in\_n\_conversion\_paths | attributed\_conversions | min\_cv\_tstamp         | max\_cv\_tstamp         | spend   | sum\_cv\_total\_revenue | attributed\_revenue | roas     |
| ---------- | ----------------- | --------------- | ------------------------ | ----------------------- | ----------------------- | ----------------------- | ------- | ----------------------- | ------------------- | -------- |
| campaign   | first\_touch      | Campaign 2      | 2                        | 1                       | 2023-07-19 04:27:51.000 | 2023-07-25 07:52:34.000 | 100,000 | 206.5                   | 206.5               | 0.002065 |
| campaign   | last\_touch       | Campaign 1      | 3                        | 1                       | 2023-07-07 13:05:55.000 | 2023-07-30 11:55:24.000 | 100,000 | 364                     | 364                 | 0.00364  |
| channel    | first\_touch      | Display\_Other  | 1                        | 1                       | 2023-07-19 04:27:51.000 | 2023-07-19 04:27:51.000 | 100,000 | 66.5                    | 66.5                | 0.000665 |
| channel    | first\_touch      | Organic\_Search | 2                        | 0                       | 2023-07-19 04:27:51.000 | 2023-07-25 07:52:34.000 | 100,000 | 206.5                   | 0                   | 0        |

### Manifest table

We have included a manifest table to log information about the setup variables each time the **`paths_to_conversion`** incremental table runs to help prevent and debug issues.

---

# Keep cohesion between the Unified and Attribution packages
> Run Unified Digital and Attribution dbt packages together with proper lineage and source references for cohesive analysis.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/keeping-cohesion-with-unified/

In case there are data issues (e.g. you may want to reprocess the analysis based on the latest user stitching status) the Attribution package produced data models can be reprocessed fully without the need to refresh the Unified package, but not the other way round as it is used as one of the primary sources to rely on when creating the incremental tables. If the underlying data source changed which impacted the views or conversions source data, you would need to refresh the attribution package as well.

There is, however, no dependency between the packages when it comes to syncing them. You may want to run your Unified package once an hour for the more business critical tables to get updated, while you can process the Attribution package once a month, if that's what makes sense for your business. There is no concept of backfill limits or partials reprocessing for the Attribution package, regardless of the last run, the package will reprocess all the data since the last processed conversion event that became available in the source tables.

### Running both Unified and Attribution dbt packages from the same project

It might be practical to run both the Unified Digital and Attribution dbt packages from the same project.

When specifying the sources, make sure you change the default source references to `ref('...')` instead of hard hardcoding the `schema.table_name` for these variables. This allows dbt to establish a proper lineage graph to process things in order:

```yml
snowplow__conversion_path_source: "{{ ref('snowplow_unified_views') }}"
snowplow__conversions_source: "{{ ref('snowplow_unified_conversions') }}"
snowplow__user_mapping_source: "{{ ref('snowplow_unified_user_mapping') }}"
```

Keep in mind that each model keeps its own state. This means that you can, for example, set a different `start_date` for the attribution model if you add it at a later point.

```yml
vars:
  snowplow_unified:
    snowplow__start_date: '2024-06-03'
  snowplow_attribution:
    snowplow__attribution_start_date: '2024-09-03'
```

---

# Snowplow Ecommerce dbt package
> Transform raw ecommerce event data into derived tables for carts, checkouts, products, transactions, and sessions.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-ecommerce repo](https://github.com/snowplow/dbt-snowplow-ecommerce), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-ecommerce/#!/overview/snowplow_ecommerce).**

The package contains a fully incremental model that transforms raw ecommerce event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) or [Mobile trackers](/docs/sources/mobile-trackers/) into a set of derived tables based around the following ecommerce data objects: carts, checkouts, products and transactions.

The Snowplow ecommerce data model aggregates Snowplow's out of the box `snowplow_ecommerce_action` events to create a set of derived tables based on the ecommerce data objects - carts, checkouts, products, transactions, users - that contain many useful dimensions as well as calculated measures.

## Overview

The package contains multiple staging models however the mart models are as follows:

- Base: Performs the incremental logic, outputting the table `snowplow_ecommerce_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model, and is the foundation for all other models generated.
- Carts: Parses the cart interactions that occur to provide handy filters and aggregations, which helps identify what happened to carts on a session level to extract, for example, abandoned carts with ease. The Carts module outputs the tables `snowplow_ecommerce_cart_interactions`.
- Checkouts: Parses checkout steps that occur to provide handy filters and aggregations to help identify which checkout steps were walked through, and what details were entered in each of these steps. This lends itself well to a funnel analysis. The Checkouts module outputs the `snowplow_ecommerce_checkout_interactions` table.
- Products: Parses product view and list information to provide insights into which products were being viewed, what details were being shown to the end user and how the user then interacted with these products. The Products module outputs the `snowplow_ecommerce_product_interactions` table.
- Transactions: Parses transaction actions to provide insights into which transactions occurred, how much revenue was generated from them, and other insights leveraging the many properties of the transaction ecommerce context. The Transactions module outputs the `snowplow_ecommerce_transaction_interactions` table.
- Sessions: Aggregates all other data into a sessions table which leverages the `domain_sessionid`, outputting the `snowplow_ecommerce_sessions` table.

## Mixing web and mobile events

The package makes no distinction between events tracked from the web and those tracked from a mobile application, so long as you are tracking `snowplow_ecommerce_action` events and from allowed `app_id`s.

The `sessionId` from the `client_session` context, and the `id` from the `mobile_screen` context, overwrite the `domain_sessionid` and `page_view_id` fields respectively in our intermediate and derived tables, for events where they are populated.

The configuration is different depending on what event sources you have:

- Web events only: the package will work out the box
- Mobile events only: the `snowplow__enable_mobile_events` package variable must be set to `true`, and the [`webPage` context](/docs/sources/web-trackers/tracker-setup/initialization-options/) must be available in your warehouse (even though it will not be populated for these mobile events)
- Both web and mobile events: the `snowplow__enable_mobile_events` package variable must be set to `true`

---

# Snowplow Media Player dbt package
> Transform raw media player event data into derived tables for media plays, ad views, and media stats across HTML5, YouTube, and Vimeo.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-media-player repo](https://github.com/snowplow/dbt-snowplow-media-player), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-media-player/#!/overview/snowplow_media_player).**

The package contains a fully incremental model that transforms raw media player event data into derived tables for easier querying. It can support media events tracked using the following tracking implementations on Web and mobile:

- on Web using plugins for our [JavaScript trackers](/docs/sources/web-trackers/):

  - [media plugin](/docs/sources/web-trackers/tracking-events/media/) that can be used to track events from any media player.
  - [HTML5 media tracking plugin](/docs/sources/web-trackers/tracking-events/media/html5/).
  - [YouTube tracking plugin](/docs/sources/web-trackers/tracking-events/media/youtube/).
  - [Vimeo tracking plugin](/docs/sources/web-trackers/tracking-events/media/vimeo/).

- [media tracking APIs on our iOS and Android trackers](/docs/sources/mobile-trackers/tracking-events/media-tracking/) for mobile apps.

- [media tracking APIs on our Roku tracker](/docs/sources/roku-tracker/media-tracking/) for [Roku](https://www.roku.com/) devices

**Version 1 and version 2 of the media event and context entity schemas**

There are two versions of schemas for media events that our trackers may use to track media events. This has an effect on the information provided by the media package. In contrast with v1, the v2 schemas contain information about the media playback that is computed directly on the tracker and is more accurate (e.g., play time, buffering time). They also introduce new schemas for tracking ads during media playback.

1. v1 media schemas (used by the HTML5 and YouTube plugin for JavaScript tracker and the Roku tracker):

   - [media-player event schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player_event/jsonschema/1-0-0) used for all media events.

   - [media-player context v1 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0).

   - Depending on the plugin / intention there are player-specific contexts:

     - in case of embedded YouTube tracking: Have the [YouTube specific context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) enabled.
     - in case of HTML5 audio or video tracking: Have the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) enabled.
     - in case of HTML5 video tracking: Have the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) enabled.

2. v2 media schemas (used by the media and Vimeo plugins for the JavaScript trackers, the Roku tracker, and the mobile trackers):

   - [per-event media event schemas](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.media).
   - [media-player context v2 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0).
   - optional [media-session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0).
   - optional [media-ad](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) and [ad break](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) context schema.

> **Note:** Support for the version 2 schemas (as used by the media, Vimeo JS plugin, Roku tracker or the mobile trackers) has been added in version 0.6 of the media player package. Older package versions only support version 1 schemas.

## Overview

The package contains multiple staging models however the mart models are as follows:

- **Base:** Performs the incremental logic, outputting the table `snowplow_media_player_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model, and is the foundation for all other models generated.

- **Media base:** Summarizes the key media player events and metrics of each media element on a media session level (or media\_id and page/screen view if media session context is not tracked, which is essentially the same), identified by the play\_id key, which is considered as a base aggregation level for media events.

  - The media base module outputs the `snowplow_media_player_base` table.
  - It also produces a `_pivot_base` table to calculate the percent\_progress boundaries and weights that are used to calculate the total play\_time and other related media fields.

- **Media plays:** Removes impressions from the media base table while keeping the same structure as the `snowplow_media_player_base` table. Removing impressions means that only media sessions (identified by the `play_id` key) in which the user played the content are kept.
  - The media plays module outputs the `snowplow_media_player_plays_by_pageview` view.

- **Media stats:** Aggregates the media base table to individual `media_identifier` level, calculating the main KPIs and overall video/audio metrics.

  - It uses the native dbt incremental materialization on a pageview basis after a set time window passed. This is to prevent complex and expensive queries due to metrics which need to take the whole page\_view events into calculation. This way the metrics will only be calculated once per pageview / media, after no new events are expected.
  - The media stats module outputs the `snowplow_media_player_media_stats` table.

- **Media ad views:** Summarizes metadata and KPIs for each ad view within a media playback. Each ad a user viewed will result in a single row with information about the progress reached, whether the ad was skipped or clicked.
  - The media ad views module outputs the `snowplow_media_player_media_ad_views` table.

- **Media ads:** Aggregates all ad views to produce a summary of the KPIs for each ad played within each media content. Produces total counts of views, skips, clicks and progress reached. It also counts the metrics in terms of unique users (e.g., number of users who clicked the ad).
  - It outputs the `snowplow_media_player_media_ads` table.

## Identifiers

The package generates robust identifiers for use in the incremental logic and keys for the derived tables.

| Identifier           | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `session_identifier` | The session identifier as defined in your dbt project variables which is is used for the package's [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/). If not set, this defaults to the `mediaSessionId` from the media session entity if enabled, else to the page/screen view id. This ensures all media events for a play spanning multiple `domain_sessionid`s are modelled. The more traditional session identifier `domain_sessionid` can be extracted from the `domain_sessionid_array` field. |
| `user_identifier`    | The user identifier as defined in your dbt project variables. If not set, this defaults to be `domain_userid` for web or the `userId` from the client session entity for mobile.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `media_identifier`   | Used to identify individual media elements/content. It is generated from the `player_id`, `media_label`, `media_type` and `media_player_type` fields.                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `play_id`            | The unique identifier for each individual play of media content. This is the `mediaSessionId` from the media session entity if enabled. Otherwise this uses the page/screen view identifier together with the `media_identifer` to generate a unique play id.                                                                                                                                                                                                                                                                                                                         |
| `media_ad_id`        | Generated identifier that identifies an ad (identified using the `ad_id`) played with a specific media (identified using the `media_identifier`) and on a specific platform (based on the `platform` property).                                                                                                                                                                                                                                                                                                                                                                       |
| `media_ad_view_id`   | The unique identifier for each individual ad view. It is generated from the `play_id`, `ad_break_id` and `media_ad_id` fields.                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

## Mixing web and mobile events

The package makes no distinction between events tracked from the web and those tracked from a mobile application, so long as you are tracking media events and from allowed `app_id`s. The `sessionId` from the `client_session` context, and the `id` from the `mobile_screen` context, overwrite the `domain_sessionid_array` and `page_view_id` fields respectively in our intermediate and derived tables, for events where they are populated. If you are just using web events, the package will work out the box. If you are using a mix of web and mobile events, you will need to set the `snowplow__enable_mobile_events` package variable to `true` and events will be processed from both sources. If you are only tracking mobile events, you can set the `snowplow__enable_web_events` to `false`.

## Custom models

There are two custom models included in the package which could potentially be used in downstream models:

1. the `snowplow_media_player_session_stats` table, which aggregates the `snowplow_media_player_base` table on a session level

2. the `snowplow_media_player_user_stats` table, which aggregates the `snowplow_media_player_session_stats` to user level

By default these are disabled, but you can enable them in the project's `profiles.yml`, if needed.

```yml
  models:
    snowplow_media_player:
      custom:
        enabled: true
```

Just like in case of all our packages, users are encouraged to use the Media Player model and its incremental logic to design their own custom models / modules. The `snowplow_media_player_base_events_this_run` table is designed with this in mind, where a couple of potentially useful fields are generated that the Media Player model does not use downstream but they nonetheless have the potential to be incorporated into users custom models. For more information on creating custom models, [visit the guide here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/).

---

# Snowplow Normalize dbt package
> Flatten and normalize self-describing events and entities into dedicated tables for downstream ETL and analysis tools.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/

> **Note:** Normalize in this context means [database normalization](https://en.wikipedia.org/wiki/Database_normalization), as these models produce flatter data, not statistical normalization.

**The package source code can be found in the [snowplow/dbt-snowplow-normalize repo](https://github.com/snowplow/dbt-snowplow-normalize), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-normalize/#/overview/snowplow_normalize).**

The package provides [macros](https://docs.getdbt.com/docs/build/jinja-macros) and a python script that is used to generate your normalized events, filtered events, and users table for use within downstream ETL tools such as Census. See the [Model Design](#model-design) section for further details on these tables.

The package only includes the base incremental scratch model and does not have any derived models, instead it generates models in your project as if they were custom models you had built on top of the [Snowplow incremental tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), using the `_this_run` table as the base for new events to process each run. See the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) section for the variables that apply to the incremental model.

> **Note:** The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__session_timestamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.

## Model Design

![Data processing model for the normalize package](/assets/images/normalized_model_light.drawio-3782b7eca528cbfd3ea13194b8189e03.png)

The package outputs models for 3 types of models: normalized events, a filtered events model, and a users model (different to the users table generated by the `web` or `mobile` packages).

### Normalized Event Models

For each `event_names` listed, a model is generated for records with matching event type(s) and with the specified columns from the `atomic.events` table, including flattened (1 level) versions of your [self-describing event](/docs/fundamentals/events/#self-describing-events) or [context](/docs/fundamentals/entities/) columns. The table names are versioned based on the same versioning that is used to handle schema evolution with your data warehouse tables (see [Updating Your Models](#updating-your-models) section for potential issues with evolving schemas). In the case of multiple event types for a table you will need to provide the name and version of the table yourself. These can be combined into other models you create, or be used directly in downstream tools. The model file itself consists of lists of variables and a macro call.

For example, if you have 3 `event_names` listed as `['page_view']`, `['page_ping']`, and `['link_click', 'deep_link_click']` then 3 models will be generated, each containing only those respective events from the atomic events table.

### Filtered Events Model

A single model is built that provides `event_id`, `snowplow__partition_tstamp` (defaults to `collector_tstamp`), and the name of the Normalized Event Model that the event was processed into, it does not include records for events that were not of an event type in your configuration. The model file itself is a series of `UNION` statements.

### Users Model

The users model provides a more traditional view of a Users table than that presented in the other Snowplow dbt packages. This model has one row per non-null `user_id` (or other identifier column you specify), and takes the latest (based on `collector_tstamp`) values from the specified contexts to ensure you always have the latest version of the information that you choose to collect about your users. This is designed to be immediately usable in downstream tools. The model file itself consists of lists of variables and a macro call.

## Package Overview

This package consists of two macros, a python script, and some example configuration files to help you get started, as well as the incremental models to build these normalized tables from:

- `normalize_events` _(macro)_: This macro does the heavy lifting of the package, taking a series of inputs to generate the SQL required to normalize the `atomic.events` table and flatten any [self-describing event](/docs/fundamentals/events/#self-describing-events) or [context](/docs/fundamentals/entities/) columns. While you can use this macro manually it is recommended to create the models that use it by using the script provided.

- `users_table` _(macro)_: This macro takes a series of inputs to generate the SQL that will produce your users table, using the user identifier column you define, and any custom contexts from your events table. The identifier will be always be renamed to `user_id` in the output.

- `snowplow_normalize_model_gen.py` _(script)_: This script uses an input configuration to generate the models based on a provided configuration file. See the [operation](#operation) section for more information.

  ```yml
  usage: snowplow_normalize_model_gen.py [-h] [--version] [-v] [--dryRun] [--configHelp] [--cleanUp] config

  Produce dbt model files for normalizing your Snowplow events table into 1 table per event type.

  positional arguments:
  config         relative path to your configuration file

  optional arguments:
  -h, --help     show this help message and exit
  --version      show program's version number and exit
  -v, --verbose  verbose flag for the running of the tool
  --dryRun       flag for a dry run (does not write/delete any files)
  --configHelp   prints information relating to the structure of the config file
  --cleanUp      delete any models not present in your config and exit (no models will be generated)
  ```

- `example_normalize_config.json`: This file is an example of an input to the python script, showing all options and valid values. For additional information about the file structure run `python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py --configHelp` in your project root.

- `example_resolver_config.json`: This file is an example [Iglu Resolver](/docs/api-reference/iglu/iglu-resolver/) configuration. It supports custom iglu servers with API keys, but does not currently support accessing embedded registries. For more information please see the Resolver docs.

- `models/base/`: Models relating to the incremental nature of the package, processing only new events (and those covered by the lookback window).

## Operation

> **Tip:** If you are using dbt Cloud then you will need to run the model-generating script locally, as there is no way to do this in cloud, and then copy your models up to your Cloud environment. You'll still need to add the package to your `packages.yml` file in your project.
>
> You will need to have [python installed](https://www.python.org/downloads/), and basic experience using the terminal and a text editor, but there is no need to install dbt. If you clone or download the [package repo](https://github.com/snowplow/dbt-snowplow-normalize) you can follow the instruction below, but remove the `dbt_packages/snowplow_normalize` from the front of paths when you see it.

In general, it should only be required to run the script in this package once to begin with, then only as/when you need to add new models based on new events or alter the contexts attached to existing ones. While it is possible to manually set the values and use the macros, it is not recommended due to the time it would take and the likelihood of making mistakes.

The script should always be run from the **root** of your dbt project (the same level your `dbt_project.yml` file is at).

### Install python packages

> **Warning:** Python versions between 3.7 and 3.10 (inclusive) are currently supported

The script only requires 2 additional packages (`jsonschema` and `requests`) that are not built into python by default, you can install these by running the below command, or by installing them by your preferred method.

```bash
pip install -r dbt_packages/snowplow_normalize/utils/requirements.txt
```

### Configuration File

> **Warning:** There may be changes to the config file between versions, this page will always contain the latest information but if you are using an older version you should call the python script with the `--configHelp` flag.

The configuration file is used to provide the information needed to generate the SQL models listed above. We use the [schemas](/docs/fundamentals/schemas/) that are defined in your event tracking to ensure full alignment with the data and reduce the amount of information you need to provide.

> **Info:** The config file is a JSON file which can be viewed by running the python script with the `--configHelp` flag. The config file can be located anywhere in your project, but it must have the following structure. Note that you must provide **at least** one of `event_columns`, `self_describing_event_schemas` or `context_schemas` for each event listed.

**Field Description:**

- `config` _(required - object)_:

  - `resolver_file_path` _(required - string)_: Relative path to your resolver config json, or `"default"` to use [iglucentral](/docs/api-reference/iglu/iglu-repositories/iglu-central/) only
  - `filtered_events_table_name` _(optional - string)_: Name of filtered events table, if not provided it will not be generated
  - `users_table_name` _(optional - string)_: Name of users table, default `events_users` if user schema(s) provided
  - `validate_schemas` _(optional - boolean)_: If you want to validate schemas loaded from each iglu registry or not, default `true`
  - `overwrite` _(optional - boolean)_: Overwrite existing model files or not, default `true`
  - `models_folder` _(optional - string)_: Folder under `models/` to place the models, default `snowplow_normalized_events`
  - `models_prefix` _(optional - string)_: The prefix to apply (with `_`) to all normalize model table names, if a `table_name` isn't provided, default `snowplow`

- `events` _(required - array)_:

  - `event_1` _(required - object)_:

    - `event_names` _(required - array)_: Name(s) of the event type(s), value of the `event_name` column in your warehouse
    - `event_columns` _(optional (>=1 of) - array)_: Array of strings of flat column names from the events table to include to include in the model
    - `self_describing_event_schemas` _(optional (>=1 of) - array)_: `iglu:com.` type url(s) for the self-describing event(s) to include in the model
    - `self_describing_event_aliases` _(optional - array)_: array of strings of prefixes to the column alias for self describing events
    - `context_schemas` _(optional (>=1 of) - array)_: Array of strings of `iglu:com.` type url(s) for the context/entities to include in the model
    - `context_aliases` _(optional - array)_: Array of strings of prefixes to the column alias for context/entities
    - `table_name` _(optional if only 1 event name, otherwise required- string)_: Name of the model, default is the `event_name`
    - `version` _(optional if only 1 event name, otherwise required - string - length 1)_: Version number to append to table name. If (only one)`self_describing_event_schema` is provided the major version number from that is used, default `1`

  - `event_2` _(optional - object)_

  - ...

  - `event_n` _(optional - object)_

- `users` _(optional - if not provided will not generate users model - object)_

  - `user_id` _(optional - if not provided will use default `user_id` field - object)_

    - `id_column` _(required - string)_: Name of column or attribute in the schema that defines your user\_id, will be converted to a string in Snowflake
    - `id_self_describing_event_schema`_(optional - string)_: `iglu:com.` type url for the self-describing event schema that your user\_id column is in, used over `id_context_schema` if both provided
    - `id_context_schema`_(optional - string)_:`iglu:com.` type url for the context schema that your user\_id column is in
    - `alias`: _(optional - string)_: Alias to apply to the id column

  - `user_contexts` _(optional (>=1 of) - array)_: Array of strings of `iglu:com.` type url(s) for the context/entities to add to your users table as columns, if not provided will not generate users model

  - `user_columns` _(optional (>=1 of) - array)_: Array of strings of flat column names from the events table to include in the model

**Example Config File:**

```json
{
    "config":{
        "resolver_file_path": "utils/example_resolver_config.json",
        "filtered_events_table_name":"snowplow_events_normalized",
        "users_table_name":"snowplow_context_users",
        "validate_schemas": true,
        "overwrite": true,
        "models_folder": "snowplow_normalized_events",
        "models_prefix": "snowplow"
    },
    "events":[
        {
            "event_names": ["page_view", "page_ping"],
            "event_columns": [
                "domain_userid",
                "app_id",
                "load_tstamp",
                "page_url"
            ],
            "context_schemas":[
                "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
                "iglu:nl.basjes/yauaa_context/jsonschema/1-0-4"
            ],
            "context_aliases": ["page", "yauaa"],
            "table_name": "snowplow_page_events",
            "version": "1"
        },
        {
            "event_names": ["link_click"],
            "event_columns": [
                "domain_userid",
                "app_id",
                "refr_urlpath"
            ],
            "self_describing_event_schemas": ["iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1"],
            "self_describing_event_aliases": ["click"],
            "context_schemas":[
                "iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0"
            ]
        }
    ],
    "users":{
        "user_id": {
            "id_column": "userId",
            "id_self_describing_event_schema": "iglu:com.google.analytics.measurement-protocol/user/jsonschema/1-0-0",
            "id_context_schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",
            "alias": "true_user_id"
        },
        "user_contexts": ["iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0",
            "iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0"],
        "user_columns": [
            "domain_userid",
            "network_userid",
            "app_id"
        ]
    }
}
```

***

An example configuration can be found in the `utils/example_normalize_config.json` file within the package.

> **Tip:** You should keep your configuration file, and your resolver file if you have one, at your project level and not inside the the snowplow-normalize package to avoid them being overwritten/deleted when the package is updated.

### Producing your models

To produce your models you need to run

`python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py path/to/your/config.json`

(or equivalent path in Windows) from the root of your dbt project. This will produce one `.sql` file for each of the event name specified in the `events` part of your configuration, one `.sql` file for the combined filtered events table if a name was provided, and one `.sql` file for your users table if schema(s) were provided. These files will be in your `models` folder in the sub-folder specified in your config.

> **Info:** Custom error messages have been added to the script to try and catch any issues and provide suggested resolutions to any issues such as invalid configurations or failing validation of schemas. If you persist in getting errors when validating schemas you believe you be correct, you can disable this validation by setting `validate_schemas` to `false` in your config.

### Adding new models

To add new models for new types of events, simply add them to your config and run the model again. Note that `overwrite` must be set to `true` for your filtered events table model to include these new events, and that this will overwrite any manual changes you made to the other models generated by this script.

### Updating your models

> **Warning:** Adding or removing columns from your dbt model, either manually or by changes caused by a different schema version, without specifying the `on_schema_change` model configuration or directly altering the tables in your database, is very likely to result in unintended outcomes - either your new columns will not appear or dbt may return an error. Please see the [dbt docs](https://docs.getdbt.com/docs/build/incremental-models#what-if-the-columns-of-my-incremental-model-change) for what to do in this situation. If you need to backfill new columns regardless you can perform a rerun of the entire model following the instructions [here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#refresh-only-some-models-in-a-package) _(note using the `--full-refresh` flag will not work in this case due to the use of the Snowplow incremental logic)_.

If you wish to update your models, such as adding a new context, removing a column, or updating to a new schema version, you must update your config file and run the script again, ensuring the `overwrite` value is set to `true`. This will remove any custom changes you made to the model file itself and you will need to re-add these.

All events models will be updated including the filtered events table, it is not possible at this time to update just a subset of models.

> **Info:** As your schemas for custom contexts and unstructured events evolve, multiple versions of the same column may be created in your events table (depending on the type of data loader being used) e.g. custom\_context\_1\_0\_0, custom\_context\_1\_0\_1. When modeling Snowplow data it can be useful to combine or coalesce each nested field across all versions of the column for a continuous view over time.
>
> The snowplow-normalize package handles this for you, all you have to do is make sure you add all the versions in the relevant config file parameter (`self_describing_event_schemas` / `context_schemas`):
>
> ```json
> "self_describing_event_schemas": ["iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-0-0", "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-1-0"]
> ```
>
> The columns will then appear as part of the derived model configuration and will then be coalesced when compiled.
>
> ```jinja2
> {%- set sde_cols = ['UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_GEOLOCATION_CONTEXT_1_0_0', 'UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_GEOLOCATION_CONTEXT_1_1_0'] -%}
> ```

#### Removing specific columns

If you want to not select all columns from your self describing event or context then you can manually delete them from the lists in each model file, ensuring to delete the matching record in the `*_types` list as well. The same warning applies from [Updating your models](#updating-your-models), and any overwriting run of the script will remove these changes.

### Removing models

You can remove all models that don't exist in your config by running the script with the `--cleanUp` flag. This will scan the `models_folder` folder provided in your config and list all models in this folder that don't match those listed in your config file; you will then be asked to confirm deletion of these. If you may wish to re-enable these models at a later date you can [disable them in your project](https://docs.getdbt.com/reference/resource-configs/enabled) instead.

> **Warning:** If you have other models in the same sub-folder that were not generated via this package then this will attempt to delete those files.

---

# App errors module for Unified Digital
> Track and analyze application errors with the App errors module for the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/app-errors-module/

Assuming your tracker is capturing `application_error` events, the module can be enabled by configuring the `dbt_project.yml` file:

```yaml

vars:
snowplow_unified:
  snowplow__enable_app_errors: true
  
```

---

# Consent module for Unified Digital
> Model user consent preferences and GDPR compliance with the Consent module for the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/

This custom module is built as an extension of the dbt-snowplow-unified package.

It transforms raw `consent_preferences` and `cmp_visible` event data into derived tables for easier querying. These events are generated by the **Enhanced Consent plugin** of the [JavaScript tracker](/docs/sources/web-trackers/).

> **Info:** For the incremental logic to work within the module you **must** use at least `RDB Loader v4.0.0`, as the custom module relies on the additional `load_tstamp` field for dbt native incrementalisation.
>
> Whenever a new consent version is added to be tracked, the model expects an `allow_all` event in order to attribute the events to the full list of latest consent scopes. It is advisable to send a test event of that kind straight after deployment so that the model can process the data accurately.

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw consent tracking events:

- `snowplow_unified_consent_log`: Snowplow incremental table showing the audit trail of consent and Consent Management Platform (cmp) events.
- `snowplow_unified_consent_users`: Incremental table of user consent tracking stats.
- `snowplow_unified_consent_totals`: Summary of the latest consent status, per consent version.
- `snowplow_unified_cconsent_scope_status`: Aggregate of current number of users consented to each consent scope.
- `snowplow_unified_cmp_stats`: Used for modeling cmp\_visible events and related metrics.
- `snowplow_unified_consent_versions`: Incremental table used to keep track of each consent version and its validity.

## Operation

It is assumed that the Snowplow unified package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_consent` variable most conveniently set in your dbt\_project.yml file:

```yaml

vars:
snowplow_unified:
  snowplow__enable_consent: true
  
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_unified` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_unified` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Model conversion events with the Unified Digital package
> Model and track conversion events with the Unified Digital dbt package for attribution and funnel analysis.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/

Conversion events are a type of event that's important to your business, be that a transaction, a sign up to a newsletter, or the view of a specific page. Whatever type of event matters to you, so long as it can be determined from a single event record, you'll be able to model and aggregate these conversions at a session level with our package.

Because we know that each user may have a different concept of what a conversion means to them, for example it could be a specific type of event such as a `page_view` on a specific url, a self-describing event such as `sign_up`, or even a bespoke `conversion` event type with some attached context, the way we model conversions is also flexible and allows you to change your definition over time. This allows you to track multiple types of conversions, with varying logic, even retrospectively.

There are two ways that you can model conversions in the Unified package:

1. In the **sessions table only**
2. In the **optional conversions module**

## 1. Enabling conversions in the sessions table only

By configuring the conversion specifications through the `snowplow__conversion_events` variable you can enable conversion events to be modelled in the derived sessions table.

> **Warning:** Because this is part of the sessions table within the Unified dbt package, we still expect your sessions to contain at least one `page_view`, a `page_ping` or a `screen_view` event, and the events must all have a `session_identifier` to be included in the `base_events_this_run_table`. Without a `session_identifier` the event will not be visible to the model, and without a `page_view`, a `page_ping` or a `screen_view` in the session there will be no session record for the model to attach the conversions to.

### Conversion Columns

For every type of conversion you provide a configuration for, the package will add up to 6 columns to the `snowplow_unified_sessions` table, depending on what you provide in your configuration. All columns start with `cv_{name}_` based on the name of your conversion and will be placed at the end of the table.

| Column                       | Description                                                                                  |
| ---------------------------- | -------------------------------------------------------------------------------------------- |
| `cv_{name}_volume`           | An integer column with the total number of conversions in the session                        |
| `cv_{name}_values`           | An array of values for each conversion, if provided in the configuration                     |
| `cv_{name}_total`            | The total of the values for each conversion, if provided in the configuration                |
| `cv_{name}_events`           | An array of `event_id`s for each conversion, if `list_events` is `true` in the configuration |
| `cv_{name}_first_conversion` | The `derived_tstamp` of the first conversion event                                           |
| `cv_{name}_converted`        | A boolean for if there was a conversion in the session                                       |

> **Tip:** If you only want to take into account the first conversion in a session then use the `_converted` column and the first element of the `_values` array in any downstream analysis.

Finally, if `snowplow__total_all_conversions` is set to `true` two additional columns will be added that total the volume and value across all conversions.

| Column           | Description                                  |
| ---------------- | -------------------------------------------- |
| `cv__all_volume` | The sum of volume across all conversions     |
| `cv__all_total`  | The sum of total value across all conversion |

> **Tip:** If you are using these columns make sure that all your conversion values are in the same units e.g. $ or stock volume.

### The conversion configuration dictionary

The `snowplow__conversion_events` variable in our project takes a list of dictionaries to determine what events count as a conversion. These dictionaries are expected to have certain keys and form what we call a conversion configuration. The keys are:

| Key             | Required | Description                                                                                                                                                         | Example value                                              |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `name`          | Yes      | The name of the conversion, to be used in the column names for this conversion                                                                                      | `sign_up`                                                  |
| `condition`     | Yes      | The (valid sql) condition that identifies if the event is a conversion. Must return true or false, and will be used in a `case when ...` statement.                 | `event_name = 'page_view' and page_url like '%signed_up%'` |
| `value`         | No       | The field name or sql to select the value associated with the conversion. If not provided the `cv_{name}_value` and `cv_{name}_total` fields will not be generated. | `tr_total_base/100`                                        |
| `default_value` | No       | The default value to use when a conversion is identified but the value returned is `null`. The type should match value. Default `0`                                 | `1`                                                        |
| `list_events`   | No       | A boolean to determine whether to add the `cv_{name}_events` column to the output.                                                                                  | `true`                                                     |

> **Tip:** Because the sessions table builds from our `_events_this_run` table, and both the `condition` and `value` fields will accept any valid sql that can go in a `case when...` and `select` block respectively, this means you can use any fields in a `contexts_` or `unstruct_` column in addition to the default `atomic.events` fields. However, it will not accept dbt code so you need to extract the relevant field yourself. You can see an example of this below.
>
> For Redshift and Postgres we have added the variable `snowplow__entities_or_sdes` that you can use to bring in all the fields from any self describing events or entities. For more details on this check out the [Custom entities & Events section](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/#custom-entities--events).

#### Example configurations

**All ids of page views on a particular url**

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events:
      [
        {
          "name": "contact_page_view",
          "condition": "event_name = 'page_view' and page_url like '%contact-us%",
          "list_events": true
        }
      ]
```

**Using a self describing event name**

For some self-describing event with a name of `sign_up`, where we do not want to attribute a value:

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events:
      [
        {
          "name": "transact",
          "condition": "event_name = 'sign_up'",
        }
      ]
```

**Using a self-describing event and a context name**

Using our [Snowplow e-commerce tracking](/docs/sources/web-trackers/tracking-events/ecommerce/):

**Snowflake:**

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events:
      [
        {
          "name": "transact",
          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1:type::varchar = 'transaction'",
          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0]:revenue::decimal(22,2)",
          "default_value":0
        }
      ]
```

**BigQuery:**

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events:
      [
        {
          "name": "transact",
          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1_0_0.type = 'transaction'",
          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1_0_0[SAFE_OFFSET(0)].revenue",
          "default_value":0
        }
      ]
```

**Databricks:**

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events:
      [
        {
          "name": "transact",
          "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1.type = 'transaction'",
          "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0].revenue",
          "default_value":0
        }
      ]
```

**Redshift:**

```yml
vars:
  snowplow_unified:
    snowplow__entities_or_sdes:
      [
        {
          'schema': 'com_snowplowanalytics_snowplow_ecommerce_transaction_1',
          'prefix': 'trans_entity',
          'alias': 'tr',
          'single_entity': true
        },
        {
          'schema': 'com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1',
          'prefix': 'trans_event',
          'alias': 'trev',
          'single_entity': true
        }
      ]
    snowplow__conversion_events:
      [
        {
          "name": "transact",
          "condition": "event_name = 'snowplow_ecommerce_action' and trans_event_type = 'transaction'",
          "value":"trans_entity_revenue"
        }
      ]
```

***

**Defining multiple conversions**

When defining multiple conversion types just follow the (list of dictionaries) format:

```yml
    snowplow__conversion_events:
      [
        {
          "name": "...",
          "condition": "...",
          ...(add more definitions optionally)
        },
        {
          "name": "...",
          "condition": "...",
          ...(add more definitions optionally)
        }
      ]
```

### Configuration Generator

You can use the below generator to generate the conversion events variable, make sure you combine this with existing conversions if you have any already. You can also use the full config generator on the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/) to help you generate all your package variables.

#### Project Variable:

```yaml
vars:
  snowplow_unified:
    snowplow__conversion_events: null
```

## 2. Enabling conversions in the Conversions Module

If you need a conversions source table for your downstream model (e.g. for the snowplow-attribution package), you can enable the optional Conversions module by setting the `snowplow__enable_conversions` variable to `true` to model that for you. You will also need to make sure you define your conversion events using the `snowplow__conversion_events` variable in case you have not already done so for the sessions table.

This would produce an incremental conversions table where you will see the most important fields related to your conversion events based on your definitions. If you defined multiple conversion types, you will see them all in one table.

| event\_id | session\_identifier | user\_identifier     | user\_id | cv\_id | cv\_value | cv\_tstamp              | dvce\_created\_tstamp   | cv\_type     |
| --------- | ------------------- | -------------------- | -------- | ------ | --------- | ----------------------- | ----------------------- | ------------ |
| event\_1  | session\_1          | f000170187170673177  | user\_1  | cv\_1  | 50.00     | 2023-06-08 20:18:32.000 | 2023-06-08 20:18:32.000 | transactions |
| event\_2  | session\_2          | f0009028775170427694 | user\_2  | cv\_2  | 20.42     | 2023-06-11 15:33:03.000 | 2023-06-11 15:33:03.000 | transactions |
| event\_3  | session\_3          | f0008284662789123943 | user\_3  | cv\_3  | 200.00    | 2023-07-07 13:05:55.000 | 2023-07-07 13:05:55.000 | transactions |

> **Note:** On Databricks targets, this table should also have a `cv_tstamp_date` column; this is used as the partition key for the table and should be the `DATE()` of the `cv_tstamp` value.

You can also apply user-stitching to your conversions table by setting the `snowplow__conversion_stitching` variable to `true`. In this case a new field called `stitched_user_id` will be added to the table. For more details please refer to our guide on [Identity Stitching](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/).

---

# Core web vitals module for Unified Digital
> Track Core Web Vitals metrics including LCP, FID, CLS, INP, and TTFB with the Core Web Vitals module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/

This custom module is built as an extension of the dbt-snowplow-unified package, it transforms raw `web vitals` event data into derived tables for easier querying. These events are generated by the **Snowplow Web Vitals plugin** (@snowplow/browser-plugin-web-vitals) of the [JavaScript tracker](/docs/sources/web-trackers/).

To enable this optional module, the unified package must be correctly configured. Please refer to the [quickstart guides](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for a full breakdown of how to set it up.

## Prerequisites

In order to use this module you would need to:

1. Track core web vitals using the web `Snowplow Web Vitals plugin`, which populates the column/table `unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1`
2. Have the [yauaa enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled (for device type and device class), which populates `contexts_nl_basjes_yauaa_context_1`
3. (Ideally, but not necessarily) Have the [spiders and bots](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichment enabled, which populates `contexts_com_iab_snowplow_spiders_and_robots_1`

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw web vitals events:

- `snowplow_unified_web_vitals`: Incremental table used as a base for storing core web vital events (first event per page view).
- `snowplow_unified_web_vital_measurements`: Drop and recompute table to use for visualizations that takes core web vital measurements at the user specified percentile point (defaulted to 75).

## Operation

It is assumed that the Snowplow unified package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_cwv` variable in your `dbt_project.yml` file:

```yaml

vars:
snowplow_unified:
  snowplow__enable_cwv: true
  
```

### Override the module specific macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- The `core_web_vital_page_groups()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_page_groups.sql) macro is used to let the user classify their urls to specific page groups. It returns the sql to provide the classification (expected in the form of case when statements).

- The `core_web_vital_results_query()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_results_query.sql) macro is used to let the user classify the thresholds to be applied for the measurements. It returns the sql to provide the logic for the evaluation based on user defined thresholds (expected in the form of case when statements).

Please make sure you set the results you would like the measurements to pass to **`good`** or align it with the `macro_core_web_vital_pass_query()` macro.

- The `core_web_vital_pass_query()` [(source)](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/core_web_vital_pass_query.sql)

```sql
case when lcp_result = 'good' and fid_result = 'good' and cls_result = 'good' then 1 else 0 end passed
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_unified` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_unified` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Snowplow Unified Digital dbt package
> Transform raw web and mobile event data into derived tables for views, sessions, users, and conversions with the Unified Digital dbt package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-unified repo](https://github.com/snowplow/dbt-snowplow-unified), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-unified/#!/overview/snowplow_unified).**

The package contains a fully incremental model that transforms raw web and mobile event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) and the [Snowplow iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) or [android trackers](/docs/sources/mobile-trackers/previous-versions/android-tracker/) into a series of derived tables of varying levels of aggregation.

The Snowplow Unified Digital Model aggregates Snowplow's out of the box page view, screen view, and page ping events to create a set of derived tables - views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![Unified Digital Model data flow](/assets/images/unified-process-light.drawio-a3ee55188faf2cad375b15273928e694.png)

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_unified_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model. From this table it also generates the `snowplow_unified_events_this_run` table, which coalesces multiple fields to combine web and mobile fields coming from different contexts / sdes.
- Views: Aggregates event level data to page view / screen view level, `view_id`, outputting the table `snowplow_unified_views`.
- Sessions: Aggregates event level data to a session level, `session_identifier`, outputting the table `snowplow_unified_sessions`. Includes other events but requires at least one `page_view`, `screen_view` or `page_ping` event in the session.
- Users: Aggregates session level data to a users level, `user_identifier`, outputting the table `snowplow_unified_users`.
- User Mapping: Provides a mapping between user identifiers, `user_identifier` and `user_id`, outputting the table `snowplow_unified_user_mapping`. This can be used for session stitching.

### Supported Entities

While using any entity in our packages is possible thanks to [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/), a large set of common web and mobile entities are built into the processing of the package to add to your derived tables.

| Entity                                                                                 | Type   | Enabled via Variable                                                                                                                                            |
| -------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [YAUAA](/docs/events/ootb-data/device-and-browser/#user-agent-parsing)                 | web    | `snowplow__enable_yauaa`                                                                                                                                        |
| [IAB](/docs/events/ootb-data/device-and-browser/#spiders-and-robots)                   | web    | `snowplow__enable_iab`                                                                                                                                          |
| [UA](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)           | web    | `snowplow__enable_ua`                                                                                                                                           |
| [Browser](/docs/events/ootb-data/device-and-browser/#browser-entity)                   | web    | `snowplow__enable_browser_context`, `snowplow__enable_browser_context_2` (depending on schema versions tracked, when both are enabled the values are coalesced) |
| [Mobile](/docs/events/ootb-data/device-and-browser/#mobile-entity)                     | mobile | `snowplow__enable_mobile_context`                                                                                                                               |
| [Geolocation](/docs/events/ootb-data/geolocation/#geolocation-entity)                  | mobile | `snowplow__enable_geolocation_context`                                                                                                                          |
| [Application](/docs/events/ootb-data/app-information/#entity-definitions)              | mobile | `snowplow__enable_application_context`                                                                                                                          |
| [Screen](/docs/events/ootb-data/page-and-screen-view-events/#screen-entity)            | mobile | `snowplow__enable_screen_context`                                                                                                                               |
| [Deep links](/docs/events/ootb-data/links-and-referrers/#deep-links-for-mobile)        | mobile | `snowplow__enable_deep_link_context`                                                                                                                            |
| [Screen summary](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) | mobile | `snowplow__enable_screen_summary_context`                                                                                                                       |

### Optional Modules

| Module                                                                                                                            | Enabled via Variable           |
| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [Consent reporting](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/)       | `snowplow__enable_consent`     |
| [Core web vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/) | `snowplow__enable_cwv`         |
| [App errors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/app-errors-module/)           | `snowplow__enable_app_errors`  |
| [Conversions](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/)                | `snowplow__enable_conversions` |

## Engaged vs. Absolute Time

At a view- and session-level we provide two measures of time; **absolute**, how long a user had the page open, and **engaged**, how much of that time the user was on the page. Engaged time is often a large predictor of a customer conversion, such as a purchase or a sign-up, whatever that may be in your domain.

Calculating absolute time is simple, it's the difference between the `derived_tstamp` of the first and last (page view or page ping) events within that page view/session.

### Web Calculation

The calculation for engaged time on web is more complicated, it is derived based on page pings which means if the user isn't active on your content, the engaged time does not increase. Let's consider a single page view example of reading an article; partway through the reader may see something they don't understand, so they open a new tab and look this up. They might stumble upon a Wikipedia page on it, they go down a rabbit hole and 10 minutes later they make it back to your site to finish the article. In this case there will be a gap for those 10 minutes in the page pings in the events data.

To adjust for these gaps we calculate engaged time as the time to trigger each ping (your heartbeat) times the number of pings (ignoring the first one), and add to that the time delay to the first ping (your minimum visit length). The formula is:

$$ engaged\_time=t\_{heartbeat}\times (n\_{distinct\_pings} -1) + t\_{min\_visit\_length} $$

and the below shows an example visually for a single page view.

![Page views and pings showing gaps to highlight the difference between absolute and engaged time](/assets/images/engaged_time_light.drawio-7db13a561e8e35da0c9ec1a0d605a344.png)

At a session level, this calculation is slightly more involved, as it needs to happen per page view and account for stray page pings (see below), but the underlying idea is the same.

### Mobile Calculation

For Mobile we use the `screen_summary` entity from the [mobile trackers](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking) for the engaged time. Check out the [mobile engagement demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/) for a live view of this.

## Stray Page Pings (Web only)

Stray Page Pings are pings within a session that do not have a corresponding `page_view` event within **the same session**. The most common cause of these is someone returning to a tab after their session has timed out but not refreshing the page. The `page_view` event exists in some other session, but there is no guarantee that both these sessions will be processed in the same run, which could lead to different results. Depending on your site content and user behavior the prevalence of sessions with stray page pings could vary greatly. For example with long-form content we have seen around 10% of all sessions contain only stray page pings (i.e. no `page_view` events).

We take different approaches to adjust for these stray pings at the page view and sessions levels, which can lead to differences between the two tables, but each is as accurate as we can currently make it.

### Sessions

As all our processing ensures full sessions are reprocessed, our sessions level table includes all stray page ping events, as well as all other view and ping events. We adjust the start time down based on your minimum visit length if the session starts with a page ping, and we include sessions that contain only (stray) pings. We also count page views based on the number of unique `page_view_ids` you have (from the `web_page` context) rather than using absolute `page_view` events to include these stray pings, and account for stray pings in the engaged time. Overall this is a more accurate view of a session and treats the stray pings as if they had a corresponding `page_view` event in the same session, even when they did not.

The result of this is you may see misalignment between sessions and if you tried to recalculate them based directly off the page views table; this is because we discard stray pings during page view processing as discussed below, so the values (`views`, `engaged_time_in_s`, and `absolute_time_in_s`) in the sessions table may be higher, but are more accurate at a session level.

![Stray page ping sessionisation](/assets/images/stray_sessions_light.drawio-fa6d6267e82193eda817072a25e62d82.png)

### Page Views

For page views, because we cannot guarantee the sessions with the `page_view` event and all subsequent `page_ping` events are processed within the same run, we choose to discard all stray page pings. Without doing this it could be possible that you would get different results from different run configurations.

**Without enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_old-light.drawio-99737733b79738a26736f5958f49625f.png)

**With enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_new-light.drawio-2cc5a036f12bec0cdc82533701523a75.png)



> **Info:** Currently we do not process these discarded stray page pings in any way, meaning that engaged time and scroll depth in these cases may be under representative of the true value. Due to session level reprocessing this remains a complicated issue to resolve, but please [let us know](https://github.com/snowplow/dbt-snowplow-unified/issues) if you would like to help solve this!

---

# How to override macros in the Unified Digital package
> Override macros in the Unified Digital dbt package to customize channel classification, conversion definitions, and event processing.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/overridable-macros/

The Unified Digital package includes several macros you can override to customize the package behavior for your specific requirements. This page documents the available macros and their purposes.

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

## Unify fields query

The `unify_fields_query` macro is used to populate the `snowplow_unified_events_this_run` table; groups together fields that come from multiple entities or SDEs depending on whether the platform is mobile or web. Should be overwritten if there is other grouping or complex SQL you wish to run on the `events this run` table.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/unify_fields_query.sql).

## Filter bots

The `filter_bots` macro defines the filter to remove bot events from events processed by the package. The filter is of the form `and <condition>` and is used throughout the package to filter out bots from all models.

The macro takes an optional `table_alias` argument, which is the table alias to prefix the column name with, if any. Default `none`.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/filter_bots.sql).

## Channel group query

The `channel_group_query` macro defines the channel a user arrived at using various fields and populates the `default_channel_group` field in the `views` and `sessions` tables. Must be a valid SQL `select` object, such as a complete `case when` statement. Used as part of the `platform_independent_fields` macro.

The defaults can be altered by overriding the macro in your project to generate your expected channels. Default values will not consider any custom marketing parameters you may have. Your override will most likely be a long list of case statements where `mkt_source` and `mkt_medium` fields are used for the classification. You can also rely on the `source_category` field that you will get as the package automatically joins the seed file generated table `snowplow_unified_dim_ga4_source_categories` where this macro gets used.

If you enable the `snowplow__use_refr_if_mkt_null` variable, the `src_field` will become a `coalesce(mkt_source, refr_source)`, and it will become `coalesce(mkt_medium, refr_medium)` for the `medium_field` used as a reference throughout the original definition.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/channel_group_query.sql).

## Engaged session

The `engaged_session` macro defines whether a session was engaged and populates the `is_engaged` field. Must return `true` or `false` and be a valid SQL `select` object, such as a complete `case when` statement. Used in the `sessions_this_run` table.

The default definition considers a session engaged if it has at least 2 views, or at least 2 heartbeats of engaged time, or had a conversion event.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/engaged_session.sql).

## Content group query

The `content_group_query` macro defines the content groups by classifying the page URLs for views. Must be a valid SQL `select` object, such as a complete `case when` statement. Used in the `views_this_run` table.

The macro code is on [GitHub](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/content_group_query.sql).

---

# Snowplow Utils dbt package
> Core macros and helper functions for Snowplow dbt packages including incremental logic, optimized upserts, and utility macros.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/

> **Info:** The models, functionality, and variables described below are only available from `snowplow-utils v0.15.0` and above, as earlier packages do not utilize these variables.

**The package source code can be found in the [snowplow/dbt-snowplow-utils repo](https://github.com/snowplow/dbt-snowplow-utils).**

The package contains a handful of useful macros that can help speed up development, as well as some handy macros to rebuild the `base` level tables that each Snowplow dbt package uses internally. We highly recommend looking at the package [readme](https://github.com/snowplow/dbt-snowplow-utils?tab=readme-ov-file#macros) for a list of all macros in the package that may be of use in any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you may build.

> **Info:** This `base` functionality and the process flow described below is only available from `snowplow-utils v0.15.0` and above.

The process flow of these macros and the tables generated is shown below:

![Utils Package data flow](/assets/images/utils-process-light.drawio-aa8ea9da878b2ebfd2b7ec86a91cb75a.png)

You can utilize these macros to create customized versions of the `events_this_run` tables. The pre-built Snowplow dbt packages can then incorporate these tables in addition to your own models. This approach offers several benefits:

Firstly, it allows you to generate custom logic that can be leveraged by the standard dbt packages. This means you can take advantage of the existing functionality while incorporating your own specialized requirements.

Secondly, it enables you to develop custom logic on top of the incremental framework already provided by Snowplow. This ensures that you only process the events that are necessary for your use case, without having to handle the incremental logic yourself. The framework takes care of sessionizing the data, simplifying your workflow.

## Overview

This model consists of a series of macros that generate models, together culminating in the `base_events_this_run` table. This table is built in other projects, including the other Snowplow packages, which can then be leveraged to get insights into your Snowplow data. The models that are built in these other projects are:

- `snowplow_base_quarantined_sessions.sql`: This model contains a list of all quarantined sessions, where a session is quarantined if it's total duration exceeds the maximum specified in the `snowplow__max_session_days` variable which has a default value of 3.
- `snowplow_incremental_manifest.sql`: This model is a manifest which contains a list of all Snowplow related models that are enabled, and keeps track of the last timestamp of successfully parsed data for each model.
- `snowplow_base_new_event_limits.sql`: This model keeps track of the upper and lower limit of events that are going to be considered, which is referenced downstream in other models to limit table scans.
- `snowplow_base_sessions_lifecycle_manifest.sql`: This model maintains a session lifecycle manifest, outlining for each session (based on the specified `session_identifier` and `user_identifier`) the start and end timestamp of the session
- `snowplow_base_sessions_this_run.sql`: This model identifies which sessions are relevant to be processed for the current run, and picks only those sessions from the session lifecycle manifest outlined above.
- `snowplow_base_events_this_run.sql`: This model finally takes the output of the base sessions table and extracts all relevant events from the sessions identified in the base sessions table, along with any custom contexts specified. This table can then be leveraged down the line by other Snowplow dbt packages or by a user's custom models.

With these macros, you can specify which columns get used as timestamps for incremental data processing, and which identifiers are used on a `session` and `user` level. By default, the `session_tstamp` is the `collector_tstamp`, the `session_identifier` is `domain_sessionid`, and the `user_identifier` is `domain_userid`. For a more in-depth explanation into how you can customize these values, you can read the [Utils quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/#6-setting-up-the-sessions-lifecycle-manifest-macro) docs.

---

# Snowplow dbt packages
> Overview of Snowplow dbt packages including Unified Digital, Attribution, Ecommerce, Media Player, Normalize, and Utils.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/

Snowplow provides several dbt packages to help you get value from your data.

| Package name                                                                                                 | Purpose                                                                            |
| ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)   | Marketing attribution analysis                                                     |
| [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/)   | Transform raw web and mobile event data into derived tables                        |
| [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) | Transform raw media player event data into derived tables                          |
| [Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/)       | Transform raw ecommerce event data into derived tables                             |
| [Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/)       | Normalize events, filtered events, and users table for use in downstream ETL tools |
| [Utils](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/)               | Useful macros                                                                      |

---

# Snowplow Fractribution dbt package
> Legacy Fractribution dbt package for marketing attribution analysis. Superseded by the Attribution package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/

**The package source code can be found in the [snowplow/dbt-snowplow-fractribution repo](https://github.com/snowplow/dbt-snowplow-fractribution), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-fractribution/#/overview/snowplow_fractribution).** ​

> **Warning:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

The Fractribution package is dependent on the `snowplow_web_page_views` model created by the [snowplow\_web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/) dbt package. Run [snowplow\_web](https://github.com/snowplow/dbt-snowplow-web) if you do not have data in the `snowplow_web_page_views` table for the period of time you will run fractribution for.

## Overview

​ The Snowplow Fractribution dbt package is a tool to perform Attribution Modeling. The name itself comes from `Fractional attribution`, which allows you to attribute the value of a conversion to one or more channels depending on the conversion pathway. As a result, it becomes possible to determine the revenue per channel, as well as ROAS (Return On Ad Spend - the amount of revenue that is earned for every dollar spent on advertising) once you have cost data for each marketing channel. ​

> **Tip:** While the throughout this package we use the term _channel_, it is entirely possible to define this in terms of something else e.g. a campaign, and define the spend and classification in the `channel_spend` and `channel_classification` macros respectively.

This package consists of a series of dbt models that produce the following tables:

- `snowplow_fractribution_channel_counts`: Number of sessions grouped by channel, campaign, source and medium
- `snowplow_fractribution_channel_spend`: Spend on each channel, used in ROAS calculations
- `snowplow_fractribution_conversions_by_customer_id`: Conversion revenue for each conversion, along with the associated `customerid`
- `snowplow_fractribution_path_summary`: Summary of different path combinations and associated conversion/non-conversions
- `snowplow_fractribution_paths_to_conversion`: Path combinations leading to conversion
- `snowplow_fractribution_paths_to_non_conversion`: Path combinations leading to non-conversion
- `snowplow_fractribution_sessions_by_customer_id`: Channel information by session timestamp, where an event timestamp is considered as the session start

Once the models are generated, the next step is to run a python script which is included in the package to run the attribution calculations. You can run this locally, via a docker image, or using Snowpark if you are on Snowflake. In the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section you will find a step-by-step guide on how to operate the package as a whole.

This script will generate and populate the following three additional tables:

- `snowplow_fractribution_channel_attribution`: The main output table that shows conversions, revenue, spend and ROAS per channel
- `snowplow_fractribution_path_summary_with_channels`: The conversion and revenue attribution per channel (used to create the report table)
- `snowplow_fractribution_report_table`: An intermediate table that shows, for each unique path, a summary of conversions, non conversions and revenue, as well as which channels were assigned a contribution ​

### Intra-session Channels

​ In Google Analytics (Universal Analytics) a new session is started if a campaign source changes (referrer of campaign tagged URL). Snowplow utilizes activity based sessionization, rather than campaign based sessionization. Setting `consider_intrasession_channels` to `false` will take only the campaign information from the first page view in a given Snowplow session, and not give credit to other channels in the converting session if they occur after the initial page view.

### Filter unwanted / wanted channels

You can specify a list of channels for the variable `snowplow__channels_to_exclude` to exclude them from analysis (if kept empty all channels are kept). For example, users may want to exclude the 'Direct' channel from the analysis.

You can also do the opposite, filter on certain channels to include in your analysis. You can do so by specifying them in the list captured within the variable `snowplow__channels_to_include`. ​

### Path Transform Options

​ Paths to conversion are often similar, but not identical. As such, path transforms reduce unnecessary complexity in similar paths before running the attribution algorithm. ​

1. `exposure (default)`: the same events in succession are reduced to one: `A → A → B` becomes `A → B`, a compromise between first and unique
2. `unique`: all events in a path are treated as unique (no reduction of complexity). Best for smaller datasets (small lookback window) without a lot of retargeting
3. `first`: keep only the first occurrence of any event: `A → B → A` becomes `A → B`, best for brand awareness marketing
4. `frequency`: keep a count of the events’ frequency: `A → A → B` becomes `A(2) → B`, best when there is a lot of retargeting
5. `remove_if_last_and_not_all`: requires a channel to be added as a parameter, which gets removed from the latest paths unless it removes the whole path as it is trying to reach a non-matching channel parameter: E.g target element: `A` path: `A → B → A → A` becomes `A → B`
6. `remove_if_not_all`: requires a channel to be added as a parameter, which gets removed from the path altogether unless it would result in the whole path's removal: E.g target element: `A` path: `A → B → A → A` becomes `B` ​

### Attribution Models

The package currently offers 5 different attribution models, that can be chosen by setting the `attribution_model` flag when calling the python script.

- `shapley` (default): Takes the weighted average of the marginal contributions of each channel to a conversion
- `first_touch`: Assigns 100% attribution to the first channel in each path.
- `last_touch`: Assigns 100% attribution to the last channel in each path.
- `position_based`: The first and last channels get 40% of the credit each, with the remaining channels getting the leftover 20% distributed evenly.
- `linear`: Assigns attribution evenly between all channels on the path. ​

![Data processing model for the fractibution package](/assets/images/attribution_models_light-6421813fd5263308643dee86dc1f23e9.png)

### Package Macros

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__conversion_value() %}
>     tr_total/100
> {% endmacro %}
> ```

#### `conversion_clause` macro

> The [`conversion_clause`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_clause.sql) macro specifies how to filter Snowplow events to only succesfful conversion events. How this is filtered will depend on your definition of a conversion. The default is filtering to events where `tr_total > 0`, but this could instead filter on `event_name = 'checkout'`, for example. If you are using the e-commerce model, you will still need to set this for the fractribution code to run (even though _all_ events are conversions in the e-commerce model), in this case change it to `transaction_revenue > 0`.

#### `conversion_value` macro

> The [`conversion_value`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_value.sql) macro specifies either a single column or a calculated value that represents the value associated with that conversion. The default is `tr_total`, but revenue or a calculation using revenue and discount\_amount from the default e-commerce schema, for example, could similarly be used.

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_classification.sql) macro is used to perform channel classifications. This can be altered to generate your expected channels if they differ from the channels generated in the default macro. It is highly recommended that you examine and configure this macro when using your own data, as the ROAS calculations and attribution calculations will run against these channel definitions, and the default values will not consider any custom marketing parameters.

#### `channel_spend` macro

> The [`channel_spend`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_spend.sql) macro is used to query the spend by channels. It requires a user supplied SQL script to extract the total ad spend by channel.
>
> Required output format:
>
> - `channel`: STRING NOT NULL
> - `spend`: FLOAT64 _(or other warehouse equivalent)_ (Use the same monetary units as conversion revenue, and NULL if unknown.)

### Environment Variables

For the script or docker container to run you need to set some environment variables that can be accessed to connect to your warehouse. These variables vary by warehouse. If you are using Docker, it can be easier to set these in a file that you reference when you run the container. See the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for detailed running information.

**BigQuery:**

- `project_id`: Project id of your BigQuery warehouse
- `bigquery_dataset`: Dataset for your derived tables For the python script only (for Docker you mount this as a volume at run time):
- `google_application_credentials`: Google Service Account JSON file

**Databricks:**

- `databricks_schema`: Schema for your derived tables
- `databricks_server_hostname`: Databricks server hostname
- `databricks_http_path`: Databricks compute resources URL
- `databricks_token`: Personal Access Token

**Snowflake:**

- `snowflake_account`: Snowflake account ID
- `snowflake_user`: Snowflake username
- `snowflake_password`: Snowflake password
- `snowflake_user_role`: Snowflake role
- `snowflake_warehouse`: Snowflake warehouse
- `snowflake_database`: Snowflake database
- `snowflake_schema`: Schema for your derived tables

**Redshift:**

- `redshift_host`: Redshift host url
- `redshift_database`: Redshift database
- `redshift_port`: Redshift port (likely `5439`)
- `redshift_user`: Redshift user
- `redshift_password`: Redshift password
- `redshift_schema`: Redshift schema

***

All warehouses require the following if you are using the docker container:

- `conversion_window_start_date`: The start date of your conversion window, in the format of YYYY-MM-DD
- `conversion_window_end_date`: The end date of your conversion window, in the format of YYYY-MM-DD
- `attribution_model`: The [attribution model method](#attribution-models)
- `warehouse`: The warehouse to run on, one of `snowflake`, `bigquery`, `redshift`, or `databricks`

​

### Differences to Google's Fractribution

​ There are some changes from [Google's](https://github.com/google/fractribution) Fractribution code that have been noted below. ​

- Temporary UDFs have been converted to persistent / permanent UDFs
- Some temporary tables converted to permanent tables
- Users without a user\_id are treated as 'anonymous' ('f') users and the domain\_userid is used to identify these sessions
- Users with a user\_id are treated as identified ('u') users
- Templating is now run almost entirely within dbt rather than the custom SQL / Jinja templating in the original Fractribution project
- Channel changes and contributions within a session can be considered using the `consider_intrasession_channels` variable

---

# App errors in Mobile package
> App errors module for the legacy Mobile dbt package. Superseded by the Unified Digital App errors module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/app-errors-module/

Exception tracking captures any unhandled exceptions within the application. The exception tracking is enabled by default, and it allows the tracker to intercept critical exceptions in the app. Exceptions can crash the app so it's likely that the event will be sent after the restart of the app. Being a critical situation we can't be 100% sure that all the exception stacktraces are reliably stored for sending before the crash of the app.

## App Error Columns

| Column                | Description                                                                 |
| --------------------- | --------------------------------------------------------------------------- |
| `programmingLanguage` | The name of the programming language used in which the app error occured.   |
| `message`             | The error message that the application showed when the app error occurred.  |
| `threadName`          | The name of the process that ran the thread when the app error occurred.    |
| `threadId`            | The ID of the thread in which the app error occurred.                       |
| `stackTrace`          | The full stack trace that was presented when the app error occured.         |
| `causeStackTrace`     | The cause of the stack trace that was presented when the app error occured. |
| `lineNumber`          | The line number in the code where the app error occured.                    |
| `className`           | The name of the class where the app error occurred.                         |
| `exceptionName`       | The name of the exception encountered in the app error.                     |
| `isFatal`             | A boolean to describe whether the app error was fatal or not.               |
| `lineColumn`          | The line number in the code where the app error occured.                    |
| `fileName`            | The name of the file where the app error occurred.                          |

---

# Snowplow Mobile dbt package
> Legacy Mobile dbt package for mobile event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

**The package source code can be found in the [snowplow/dbt-snowplow-mobile repo](https://github.com/snowplow/dbt-snowplow-mobile), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-mobile/#!/overview/snowplow_mobile).**

The package contains a fully incremental model that transforms raw mobile event data generated by the [Snowplow iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) or [android trackers](/docs/sources/mobile-trackers/previous-versions/android-tracker/) into a series of derived tables of varying levels of aggregation.

The Snowplow mobile data model aggregates Snowplow's out-of-the-box mobile events to create a set of derived tables - screen views, sessions, and users. These contain many useful dimensions, as well as calculated measures such as screen views per session.

![Mobile Package data flow](/assets/images/mobile-process-light.drawio-de145671692376e99ef55abe866f03dc.png)

| snowplow-mobile version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| ----------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- |
| 1.0.0                   | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.7.2                   | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.6.3                   | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.5.5                   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.2.0                   | >=0.20.0 to <1.0.0 | ✅        | ❌          | ✅        | ✅         | ✅        |

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_mobile_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model.
- Screen Views: Aggregates event level data to a screen view level, on `screen_view_id`, outputting the table `snowplow_mobile_screen_views`.
- Sessions: Aggregates screen view level data to a session level, on `session_id`, outputting the table `snowplow_mobile_sessions`.
- Users: Aggregates session level data to a users level, on `device_user_id`, outputting the table `snowplow_mobile_users`.
- User Mapping: Provides a mapping between user identifiers, `device_user_id` and `user_id`, outputting the table `snowplow_mobile_user_mapping`. This can be used for session stitching.

## Contexts

The following contexts can be enabled depending on your tracker configuration:

- Mobile context
- Geolocation context
- Application context
- Screen context

By default they are disabled. They can be enabled by configuring the `dbt_project.yml` file and setting the appropriate `snowplow__enable_{context_type}_context` variable to `true`.

## Optional Modules

Currently the App Errors module, used for crash reporting, is the only optional module. More will be added in the future as the tracker's functionality expands.

Assuming your tracker is capturing `application_error` events, the module can be enabled by configuring the `dbt_project.yml` file:

```yaml

vars:
snowplow_mobile:
  snowplow__enable_app_errors_module: true
  
```

---

# Consent module for the Web package
> Consent module for the legacy Web dbt package. Superseded by the Unified Digital Consent module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/consent-module/

This custom module is built as an extension of the dbt-snowplow-web package.

It transforms raw `consent_preferences` and `cmp_visible` event data into derived tables for easier querying. These events are generated by the **Enhanced Consent plugin** of the [JavaScript tracker](/docs/sources/web-trackers/).

> **Info:** For the incremental logic to work within the module you **must** use at least `RDB Loader v4.0.0`, as the custom module relies on the additional `load_tstamp` field for dbt native incrementalisation.
>
> Whenever a new consent version is added to be tracked, the model expects an `allow_all` event in order to attribute the events to the full list of latest consent scopes. It is advisable to send a test event of that kind straight after deployment so that the model can process the data accurately.

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw consent tracking events:

- `snowplow_web_consent_log`: Snowplow incremental table showing the audit trail of consent and Consent Management Platform (cmp) events.
- `snowplow_web_consent_users`: Incremental table of user consent tracking stats.
- `snowplow_web_consent_totals`: Summary of the latest consent status, per consent version.
- `snowplow_web_cconsent_scope_status`: Aggregate of current number of users consented to each consent scope.
- `snowplow_web_cmp_stats`: Used for modeling cmp\_visible events and related metrics.
- `snowplow_web_consent_versions`: Incremental table used to keep track of each consent version and its validity.

## Operation

It is assumed that the Snowplow web package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_consent` variable most conveniently set in your dbt\_project.yml file:

```yaml

vars:
snowplow_web:
  snowplow__enable_consent: true
  
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_web` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_web` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Modeling conversions with the Web package
> Conversion modeling for the legacy Web dbt package. Superseded by the Unified Digital Conversions module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/

Conversion events are a type of event that's important to your business, be that a transaction, a sign up to a newsletter, or the view of a specific page. Whatever type of event matters to you, so long as it can be determined from a single event record, you'll be able to model and aggregate these conversions at a session level with our package.

Because we know that each user may have a different concept of what a conversion means to them, for example it could be a specific type of event such as a `page_view` on a specific url, a self-describing event such as `sign_up`, or even a bespoke `conversion` event type with some attached context, the way we model conversions is also flexible and allows you to change your definition over time. This allows you to track multiple types of conversions, with varying logic, even retrospectively.

> **Warning:** Because this is part of the sessions table within the web package, we still expect your sessions to contain at least one `page_view` or `page_ping` event, and the events must all have a `domain_sessionid` to be included in the `base_events_this_run_table`. Without a `domain_sessionid` the event will not be visible to the model, and without a `page_view` or `page_ping` in the session there will be no session record for the model to attach the conversions to.

## Conversion Columns

For every type of conversion you provide a configuration for, the package will add up to 6 columns to the `snowplow_web_sessions` table, depending on what you provide in your configuration. All columns start with `cv_{name}_` based on the name of your conversion and will be placed at the end of the table.

| Column                       | Description                                                                                  |
| ---------------------------- | -------------------------------------------------------------------------------------------- |
| `cv_{name}_volume`           | An integer column with the total number of conversions in the session                        |
| `cv_{name}_values`           | An array of values for each conversion, if provided in the configuration                     |
| `cv_{name}_total`            | The total of the values for each conversion, if provided in the configuration                |
| `cv_{name}_events`           | An array of `event_id`s for each conversion, if `list_events` is `true` in the configuration |
| `cv_{name}_first_conversion` | The `derived_tstamp` of the first conversion event                                           |
| `cv_{name}_converted`        | A boolean for if there was a conversion in the session                                       |

> **Tip:** If you only want to take into account the first conversion in a session then use the `_converted` column and the first element of the `_values` array in any downstream analysis.

Finally, if `snowplow__total_all_conversions` is set to `true` two additional columns will be added that total the volume and value across all conversions.

| Column           | Description                                  |
| ---------------- | -------------------------------------------- |
| `cv__all_volume` | The sum of volume across all conversions     |
| `cv__all_total`  | The sum of total value across all conversion |

> **Tip:** If you are using these columns make sure that all your conversion values are in the same units e.g. $ or stock volume.

## The conversion configuration dictionary

The `snowplow__conversion_events` variable in our project takes a list of dictionaries to determine what events count as a conversion. These dictionaries are expected to have certain keys and form what we call a conversion configuration. The keys are:

| Key             | Required | Description                                                                                                                                                         | Example value                                              |
| --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `name`          | Yes      | The name of the conversion, to be used in the column names for this conversion                                                                                      | `sign_up`                                                  |
| `condition`     | Yes      | The (valid sql) condition that identifies if the event is a conversion. Must return true or false, and will be used in a `case when ...` statement.                 | `event_name = 'page_view' and page_url like '%signed_up%'` |
| `value`         | No       | The field name or sql to select the value associated with the conversion. If not provided the `cv_{name}_value` and `cv_{name}_total` fields will not be generated. | `tr_total_base/100`                                        |
| `default_value` | No       | The default value to use when a conversion is identified but the value returned is `null`. The type should match value. Default `0`                                 | `1`                                                        |
| `list_events`   | No       | A boolean to determine whether to add the `cv_{name}_events` column to the output.                                                                                  | `true`                                                     |

> **Tip:** Because the sessions table builds from our `_events_this_run` table, and both the `condition` and `value` fields will accept any valid sql that can go in a `case when...` and `select` block respectively, this means you can use any fields in a `contexts_` or `unstruct_` column in addition to the default `atomic.events` fields. However, it will not accept dbt code so you need to extract the relevant field yourself. You can see an example of this below.
>
> For Redshift and Postgres users currently you are limited to just the fields added by the IAB, UA, and YAUAA contexts without modifying the models yourself, however we plan to support adding arbitrary self-describing event and context fields to our base table in the future.

### Example configurations

**All ids of page views on a particular url**

```json
    {
    "name": "contact_page_view",
    "condition": "event_name = 'page_view' and page_url like '%contact-us%",
    "list_events": true
    }
```

**Using a self describing event name**

For some self-describing event with a name of `sign_up`, where we do not want to attribute a value:

```json
    {
    "name": "transact",
    "condition": "event_name = 'sign_up'",
    }
```

**Using a self-describing event and a context name**

Using our [Snowplow e-commerce tracking](/docs/sources/web-trackers/tracking-events/ecommerce/):

**Snowflake:**

```json
    {
    "name": "transact",
    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1:type::varchar = 'transaction'",
    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0]:revenue::decimal(22,2)",
    "default_value":0
    }
```

**BigQuery:**

```json
    {
    "name": "transact",
    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1_0_0.type = 'transaction'",
    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1_0_0[SAFE_OFFSET(0)].revenue",
    "default_value":0
    }
```

**Databricks:**

```json
    {
    "name": "transact",
    "condition": "UNSTRUCT_EVENT_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_SNOWPLOW_ECOMMERCE_ACTION_1.type = 'transaction'",
    "value": "CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_ECOMMERCE_TRANSACTION_1[0].revenue",
    "default_value":0
    }
```

***

## Configuration Generator

You can use the full config generator on the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/) to help you generate your conversion event definitions.

---

# Core web vitals module for the Web package
> Core Web Vitals module for the legacy Web dbt package. Superseded by the Unified Digital Core Web Vitals module.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/core-web-vitals-module/

This custom module is built as an extension of the dbt-snowplow-web package, it transforms raw `web vitals` event data into derived tables for easier querying. These events are generated by the **Snowplow Web Vitals plugin** (@snowplow/browser-plugin-web-vitals) of the [JavaScript tracker](/docs/sources/web-trackers/).

To enable this optional module, the web package must be correctly configured. Please refer to the [quickstart guides](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) for a full breakdown of how to set it up.

## Prerequisites

In order to use this module you would need to:

1. Track core web vitals using the web `Snowplow Web Vitals plugin`, which populates the column/table `unstruct_event_com_snowplowanalytics_snowplow_web_vitals_1`
2. Have the [yauaa enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled (for device type and device class), which populates `contexts_nl_basjes_yauaa_context_1`
3. (Ideally, but not necessarily) Have the [spiders and bots](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichment enabled, which populates `contexts_com_iab_snowplow_spiders_and_robots_1`

## Overview

This custom module consists of a series of dbt models which produce the following aggregated models from the raw web vitals events:

- `snowplow_web_web_vitals`: Incremental table used as a base for storing core web vital events (first event per page view).
- `snowplow_web_web_vital_measurements`: Drop and recompute table to use for visualizations that takes core web vital measurements at the user specified percentile point (defaulted to 75).

## Operation

It is assumed that the Snowplow web package is already installed and configured as per the [Quick Start](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) instructions.

### Enable the module

You can enable the custom module through the `snowplow__enable_cwv` variable in your `dbt_project.yml` file:

```yaml

vars:
snowplow_web:
  snowplow__enable_cwv: true
  
```

### Override the module specific macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- The `core_web_vital_page_groups()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_page_groups.sql) macro is used to let the user classify their urls to specific page groups. It returns the sql to provide the classification (expected in the form of case when statements).

- The `core_web_vital_results_query()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_results_query.sql) macro is used to let the user classify the thresholds to be applied for the measurements. It returns the sql to provide the logic for the evaluation based on user defined thresholds (expected in the form of case when statements).

Please make sure you set the results you would like the measurements to pass to **`good`** or align it with the `macro_core_web_vital_pass_query()` macro.

- The `core_web_vital_pass_query()` [(source)](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/core_web_vital_pass_query.sql)

```sql
case when lcp_result = 'good' and fid_result = 'good' and cls_result = 'good' then 1 else 0 end passed
```

### Run the module

If you have previously run the model without this optional module enabled, you can simply enable the module and run `dbt run --selector snowplow_web` as many times as needed for this module to catch up with your other data. If you only wish to process this from a specific date, be sure to change your `snowplow__start_date`, or refer to the [Custom module](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models) section for a detailed guide on how to achieve this the most efficient way.

If you haven't run the web package before, then you can run it using `dbt run --selector snowplow_web` either through your CLI, within dbt Cloud, or for Snowplow CDI customers you can use Snowplow Console. In this situation, all models will start in-sync as no events have been processed.

---

# Snowplow Web dbt package
> Legacy Web dbt package for web event data modeling. Superseded by the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/

> **Info:** This package has been superseded by the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) that combines data from both web and mobile sources. For more information, see the Unified Digital page.

**The package source code can be found in the [snowplow/dbt-snowplow-web repo](https://github.com/snowplow/dbt-snowplow-web), and the docs for the [model design here](https://snowplow.github.io/dbt-snowplow-web/#!/overview/snowplow_web).**

The package contains a fully incremental model that transforms raw web event data generated by the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) into a series of derived tables of varying levels of aggregation.

The Snowplow web data model aggregates Snowplow's out of the box page view and page ping events to create a set of derived tables - page views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![Web Package data flow](/assets/images/web-process-light.drawio-a2b276daa0d6a441c3851bc3bb5db941.png)

| snowplow-web version | dbt versions        | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| -------------------- | ------------------- | -------- | ---------- | -------- | --------- | -------- |
| 1.0.1                | >=1.6.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.16.2               | >=1.5.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.15.2               | >=1.4.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅\*      |
| 0.13.3\*\*           | >=1.3.0 to <2.0.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.11.0               | >=1.0.0 to <1.3.0   | ✅        | ✅          | ✅        | ✅         | ✅        |
| 0.5.1                | >=0.20.0 to <1.0.0  | ✅        | ❌          | ✅        | ✅         | ✅        |
| 0.4.1                | >=0.18.0 to <0.20.0 | ✅        | ❌          | ✅        | ✅         | ❌        |

\* Since version 0.15.0 of `snowplow_web` at least version 15.0 of Postgres is required, otherwise you will need to [overwrite](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/) the `default_channel_group` macro to not use the `regexp_like` function.

\*\* From version v0.13.0 onwards we use the `load_tstamp` field so you must be using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v4.0.0 and above, or [BigQuery Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v1.0.0 and above. If you do not have this field because you are not using these versions, or you are using the Postgres loader, you will need to set `snowplow__enable_load_tstamp` to `false` in your `dbt_project.yml` and will not be able to use the consent models.

## Overview

This model consists of a series of modules, each producing a table which serves as the input to the next module. The 'standard' modules are:

- Base: Performs the incremental logic, outputting the table `snowplow_web_base_events_this_run` which contains a de-duped data set of all events required for the current run of the model.
- Page Views: Aggregates event level data to a page view level, `page_view_id`, outputting the table `snowplow_web_page_views`.
- Sessions: Aggregates event level data to a session level, `domain_sessionid`, outputting the table `snowplow_web_sessions`. Includes other events but requires at least one `page_view` or `page_ping` event in the session.
- Users: Aggregates session level data to a users level, `domain_userid`, outputting the table `snowplow_web_users`.
- User Mapping: Provides a mapping between user identifiers, `domain_userid` and `user_id`, outputting the table `snowplow_web_user_mapping`. This can be used for session stitching.

## Optional Modules

See the subsections for information on the optional modules.

## Overridable Macros

> **Tip:** For information about overriding our macros, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/)

- `filter_bots(table_alias)`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/filter_bots.sql): used to define the filter to remove bot events from events processed by the package. Of the form `and <condition>`. Used throughout the package to filter out bots from all models.
- `channel_group_query()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/channel_group_query.sql): defines the channel a user arrived at using various fields, populates the `default_channel_group` field. Must be a valid sql `select` object e.g. a complete `case when` statement. Used in `sessions_this_run` table.
- `engaged_session()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/engaged_session.sql): defines if a session was engaged or not, populates the `is_engaged` field. Must return `true` or `false` and be a valid sql `select` object e.g. a complete `case when` statement. Used in `sessions_this_run` table.
- `content_group_query()`[source](https://github.com/snowplow/dbt-snowplow-web/blob/main/macros/channel_group_query.sql): defines the content groups by classifying the page urls for page views. Must be a valid sql `select` object e.g. a complete `case when` statement. Used in `page_views_this_run` table.

## Engaged vs. Absolute Time

At a page view- and session-level we provide two measures of time; **absolute**, how long a user had the page open, and **engaged**, how much of that time the user was on the page. Engaged time is often a large predictor of a customer conversion, such as a purchase or a sign-up, whatever that may be in your domain.

Calculating absolute time is simple, it's the difference between the `derived_tstamp` of the first and last (page view or page ping) events within that page view/session.

The calculation for engaged time is more complicated, it is derived based on page pings which means if the user isn't active on your content, the engaged time does not increase. Let's consider a single page view example of reading an article; partway through the reader may see something they don't understand, so they open a new tab and look this up. They might stumble upon a Wikipedia page on it, they go down a rabbit hole and 10 minutes later they make it back to your site to finish the article. In this case there will be a gap for those 10 minutes in the page pings in the events data.

To adjust for these gaps we calculate engaged time as the time to trigger each ping (your heartbeat) times the number of pings (ignoring the first one), and add to that the time delay to the first ping (your minimum visit length). The formula is:

$$ engaged\_time=t\_{heartbeat}\times (n\_{distinct\_pings} -1) + t\_{min\_visit\_length} $$

and the below shows an example visually for a single page view.

![Page views and pings showing gaps to highlight the difference between absolute and engaged time](/assets/images/engaged_time_light.drawio-7db13a561e8e35da0c9ec1a0d605a344.png)

At a session level, this calculation is slightly more involved, as it needs to happen per page view and account for [stray page pings](#stray-page-pings), but the underlying idea is the same.

## Stray Page Pings

Stray Page Pings are pings within a session that do not have a corresponding `page_view` event within **the same session**. The most common cause of these is someone returning to a tab after their session has timed out but not refreshing the page. The `page_view` event exists in some other session, but there is no guarantee that both these sessions will be processed in the same run, which could lead to different results. Depending on your site content and user behavior the prevalence of sessions with stray page pings could vary greatly. For example with long-form content we have seen around 10% of all sessions contain only stray page pings (i.e. no `page_view` events).

We take different approaches to adjust for these stray pings at the page view and sessions levels, which can lead to differences between the two tables, but each is as accurate as we can currently make it.

### Sessions

As all our processing ensures full sessions are reprocessed, our sessions level table includes all stray page ping events, as well as all other view and ping events. We adjust the start time down based on your minimum visit length if the session starts with a page ping, and we include sessions that contain only (stray) pings. We also count page views based on the number of unique `page_view_ids` you have (from the `web_page` context) rather than using absolute `page_view` events to include these stray pings, and account for stray pings in the engaged time. Overall this is a more accurate view of a session and treats the stray pings as if they had a corresponding `page_view` event in the same session, even when they did not.

The result of this is you may see misalignment between sessions and if you tried to recalculate them based directly off the page views table; this is because we discard stray pings during page view processing as discussed below, so the values (`page_views`, `engaged_time_in_s`, and `absolute_time_in_s`) in the sessions table may be higher, but are more accurate at a session level.

![Stray page ping sessionisation](/assets/images/stray_sessions_light.drawio-fa6d6267e82193eda817072a25e62d82.png)

### Page Views

For page views, because we cannot guarantee the sessions with the `page_view` event and all subsequent `page_ping` events are processed within the same run, we choose to discard all stray page pings. Without doing this it could be possible that you would get different results from different run configurations.

**Without enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_old-light.drawio-99737733b79738a26736f5958f49625f.png)

**With enforcing within-session view**

![Stray page ping page views](/assets/images/stray_views_new-light.drawio-2cc5a036f12bec0cdc82533701523a75.png)



> **Info:** Currently we do not process these discarded stray page pings in any way, meaning that engaged time and scroll depth in these cases may be under representative of the true value. Due to session level reprocessing this remains a complicated issue to resolve, but please [let us know](https://github.com/snowplow/dbt-snowplow-web/issues) if you would like to help solve this!

---

# Overview of legacy dbt models
> Legacy Snowplow dbt packages including Web, Mobile, and Fractribution models superseded by newer packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/

> **Warning:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Run modules asynchronously
> Run dbt package modules asynchronously at different intervals using session identifiers and selective model execution.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/async-runs/

You may wish to run the modules asynchronously, for instance run the views module hourly but the sessions and users modules daily. You would assume this could be achieved using e.g.:

```bash
dbt run --select +snowplow_web.page_views
```

However, due to the models being listed in the [manifest table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) we would need to know which models are included in this at run time; this is not currently possible. Instead all models from the standard and custom modules are selected from the manifest table and the package will attempt to synchronize all models. This makes the above command unsuitable for asynchronous runs.

However you can leverage dbt's `ls` command in conjunction with shell substitution to explicitly state what models to run, allowing a subset of models to be selected from the manifest and thus run independently.

For example to run just the page views module asynchronously:

```bash
dbt run --select +snowplow_unified.views --vars "{'models_to_run': '$(dbt ls --select +snowplow_unified.views --output name | tail -n +4)'}"
```

> **Note:** The `| tail -n +4` simply removes the dbt logging front matter from the output

---

# Backfill your data
> Steps for backfilling the data models after initialization, adding custom models, or following a full refresh.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/

When you first start using our packages, when you add [new custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/), or following on from a [full refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/), you may have a large period of data that needs to be processed before your models are _caught up_ with your live data and the package does a [standard run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-4-standard-run).

As described in [incremental processing](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default our packages are set up to avoid processing large periods of data in one go, this helps to reduce the workload on your cloud, as well as reducing the impact of any issues in the data causing a single run to fail.

When backfilling your data, it is recommended to increase the `snowplow__backfill_limit_days` variable to as high as you feel comfortable with to reduce the number of runs that need to complete to get your processed data up to date. In general increasing this to cover 3-6 months is a good balance between speed and number of runs.

We also recommend doing this process manually, as until your data is up to date, no new data will be processed, so it may take many runs for your data to catch up with live tracking. Running locally you can do many runs back-to-back rather than wait for your scheduled runs to take place. You may also wish to warn any users of your data this is planned as it can lead to delays, and an initial full-refresh will remove any existing data from the derived tables.

> **Tip:** If you have added a new model it will be processed from the value in your `snowplow__start_date` variable, if you don't need the model to start from this far back then you should change this for the first run so the new model only runs from a more recent date instead.

## Blue-Green style Deployment

As the process of backfilling data may take a long time depending on your data volumes, and during this time no new data will be processed, it may be preferable to backfill your model(s) in a staging environment and then deploy these derived tables into your production schema at a later date.

> **Warning:** This process involves manually editing the manifest table, which is dangerous and should be done only with understanding of our [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) and [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/).

Let's assume you have one schema named `prod` that is used by a regularly scheduled run of a project that uses our package(s), and you want to introduce a new custom model backfilled with all your historic Snowplow data. We are going to first process this model in a `staging` schema before migrating it across.

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    %% This comment is here to trigger a git change when spaces at the end of lines are removed, do not do that on this page it is required for the axis formatting
    axisFormat
    tickInterval 1day
    section New Model
    Unprocessed :crit, n1, 2020-01-01, 5d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

### Step 1: Create and process new model only

In a local environment, assuming you are using version control for your project, create a new branch and create your model in this branch. Next ensure you are using a profile/target and/or configure the package models so that they will write to the `staging` schema (or be prefixed with `staging` depending on your setup) instead of `prod`.

Now you will need to run the below script to run only what is needed for your new model (see [model selection](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) for more details on why). We've also increased the backfill limit days so this needs to be run fewer times.

```bash
dbt run --select +my_new_model --vars "{'models_to_run': '$(dbt ls --select  +my_new_model --output name | tail -n +4)', 'snowplow__backfill_limit_days': 90}"
```

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 1day
    section New Model
    Unprocessed :crit, n1, after n2, 4d
    Processed (in staging) :active, n2, 2020-01-01, 1d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

Once you have run this enough times that the data is up-to-date, which you can identify either by looking at the print out in the logs or by querying the manifest table, it's time to move this data to your `prod` schema.

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 1day
    section New Model
    Processed (in staging) :active, n2, 2020-01-01, 5d
    section Other Models
    Processed :active, a1, 2020-01-01, 5d
```

### Step 2: Migrate the new table

The exact SQL you need to run will vary by warehouse, and some warehouses may offer the ability to do this in their UI. It may be possible to move your table via an `ALTER TABLE` statement, however for simplicity you can recreate your new table in your `prod` schema via a `CREATE TABLE` statement e.g.

```sql
CREATE TABLE prod.my_new_model as (select * from staging.my_new_model);
```

Make sure to recreate any additional features as necessary e.g. partition keys.

### Step 3: Update the manifest record

> **Danger:** Ensure your project is not running while you perform this step as it could cause a conflict in the manifest table.

If you were to now add this model to the project being run in `prod`, because the manifest table in `prod` (or more likely, `prod_snowplow_manifest`) has no record of this model, it will process it again from the start (as it will be in [state 2](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-2-new-model-introduced)). To avoid this you need to add a record for this model into the manifest table.

Again, some UIs may allow you to edit a table, but an `INSERT` statement is recommended e.g.

```sql
INSERT INTO PROD.SNOWPLOW_<package>_INCREMENTAL_MANIFEST (MODEL, LAST_SUCCESS) VALUES
('my_new_model', timestamp '2023-10-11 10:11:12');
```

Where you can find the exact timestamp you need to use in your `staging` manifest table.

### Step 4: Migrate the model

Finally, with the data in the `prod` schema, and the record of it in the manifest table, you can merge your branch into `main` (or whatever branch you run your `prod` from), ensuring that the schema setting for your model matches. On the next run the package will pick up this new model, see it in the manifest, and run it alongside the rest of the models without needing to start from scratch.

---

# Tests for dbt packages
> Use dbt tests to validate data quality in Snowplow packages with test selection tags for scratch and derived models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/dbt-tests/

[Tests](https://docs.getdbt.com/docs/build/tests) are a useful feature in dbt to ensure that your data meets some expectations of it, such as certain fields should not be null. We provide a suite of tests with our packages to help test the output tables for any issues, however you could and should add your own as no one knows the required quality checks of your data better than you.

## Test Selection

The packages contain tests for both the scratch and derived models. Depending on your use case you might not want to run all tests in production, for example to save costs. There are several tags included in the packages to help select subsets of tests. Tags:

- `this_run`: Any model with the `_this_run` suffix
- `scratch`: Any model in the scratch sub directories.
- `derived`: Any of the derived models i.e. page views, sessions and users.
- `primary-key`: Any test on the primary keys of all models in this package.

For example if your derived tables are very large you may want to run the full test suite on the `this_run` tables, which act as the input for the derived tables, but only primary key schema tests on the derived tables to ensure no duplicates. If using such a set up, we would also recommend including the `page/screen_view_in_session_value` data test for the page/screen views derived tables.

This is our recommended approach to testing and can be implemented using the selector flag (see [YAML selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section for more details) as follows:

```bash
dbt test --selector snowplow_<package>_lean_tests
```

This is equivalent to:

```bash
dbt test --select snowplow_<package>,tag:this_run # Full tests on _this_run models
dbt test --select snowplow_<package>,tag:manifest # Full tests on manifest models
dbt test --select snowplow_<package>,tag:primary-key,tag:derived # Primary key tests only on derived tables.
dbt test --select snowplow_<package>,tag:derived,test_type:data  # Include the page/screen_view_in_session_value data test
```

Alternatively, if you wanted to run all available tests both in the Snowplow package, and your custom modules:

```bash
dbt test --selector snowplow_<package>
```

## Test Configuration

There are a number of configuration options on dbt tests, which are documented in the [dbt docs](https://docs.getdbt.com/reference/test-configs). You may want to apply some of these on the tests within the Snowplow dbt packages to ensure they're applicable to your data and use cases. The best way to achieve this is by editing the configuration within your project's root `dbt_project.yml` file.

For example, you may want to apply a severity threshold on all tests within the `snowplow_web` package. You can do this by updating your `dbt_project.yml` configuration as below:

```yml
# dbt_project.yml
# Example config applied to all tests in a package
...
tests:
  snowplow_web:
    +warn_if: ">10"
    +error_if: ">100"
```

Alternatively, you could apply a configuration to a specific, single instance of a test by calling out its resource path directly:

```yml
# dbt_project.yml
# Example config applied to a specific test
...
tests:
  snowplow_web:
    page_views:
      not_null_snowplow_web_page_views_domain_sessionid:
        +where: "app_id != '<app_id_to_exclude_from_test>'"
```

---

# Debug data modeling issues
> Guide for debugging data modeling issues including execution errors, missing data, and unexpected results.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/debugging/

There will inevitably be times when you run into dbt errors or the output is not what you expected (e.g. a session is missing from the sessions table). This guide will try and help you get started with the most common issues and how to handle them, to help you get unblocked faster and avoid having to open a customer ticket to ask for help.

## Scenario 1. The model errors out during execution

### What is the specific error message?

The first thing to look for is the actual error message. Depending on how you run your package you might have different levels of access: e.g. running the dbt locally will give you access to the whole model run log and compiled sql code within the `target` folder, whereas in Console you will typically only be able to see the model's run log with the error message where it fails.

If you look at the run log, you may need to scroll down the output until you reach the error though (there may be more than one). You will also see which models ran successfully. Look for the full error message, if it starts with `Snowplow Error: ...` it is an error message specifically created by the Snowplow Package you run, and it should give you a context into where and why it is failing. It could also be a warehouse related issue, due to access rights, or temporary resource issues etc.

In case of a Snowplow Console based error message, an error output message such as a `connection refused` related message, or for example an `EOF` error is typically down to some external database process, or a network connection error. These errors often do not need any interaction to be resolved.

If it is a syntax issue, it may give a specific code fragment to help you debug. Note down which model fails and try to execute that particular model in debug mode locally by adding `-d` to your dbt run prompt. Dbt should display a more extensive log message potentially with the full compiled code it tries to run but fails that you can copy and take to your sql editor of choice to debug separately.

### Is this the first time running into this issue? Did the model run properly before?

If the error happens after the initial package setup, it may be related to specific configurations missing. This is especially true in case of the Unified package due to its extensive optional feature support. We added a specific config helper variable: `snowplow__enable_initial_checks` to help with errors resulting from wrong configurations. When enabled it should give you a more specific error log with respect to your config that might help in such cases.

If the model only ran once but not the second time, the issue is most likely related to the incrementalization, in many cases the second run of the model contains extra code related to the timeframe to take into account for incremental updates.

If the package was running fine several times, but all of a sudden stopped working, you may need to check what changes happened since the last run. Did you upgrade to a new dbt version? Can it be related to a snowplow package version change? Was there something implemented from the tracking side? Were there any recent pipeline issues?

## Scenario 2. Output is missing data

The most common scenario that may happen is there are certain session\_identifiers or user\_identifiers missing from the output tables. The reason behind this can be many, we will touch the most common ones.

We process the data based on sessions: for each run we take a look at the new events (with default 6 hours of lookback window) then look at the sessions manifest table and identify how far to look back in time to cover those sessions with new events as a whole. One overlooked common scenario is that in case of pipeline issues, if some data is getting processed by the model, but the rest are only added to the warehouse outside of that default 6 hour lookback window, then the package will leave those events behind unless the same session will have new events coming in, and it needs to be reprocessed as a whole again (unless the data modeling jobs were immediately paused after the pipeline went partially down). In such cases you need partial backfilling (see the [late loaded events](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/#late-loaded-events) section).

The events can also be [late sent](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/#late-sent-events), in which case if they fall out of the limit, they will not get processed.

We also "quarantine" the sessions at times, as however hard we may try to filter out bots, there could be odd outliers that keep sessions alive for days, for which we have the `snowplow__max_session_days` variable. If bots keep sending events for longer than defined in that variable, the package will only process the first of those in a subsequent run and archives the session\_identifier in the [quarantine table](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) after it runs. In such a case it is always best to look for the missing event's session\_identifier (domain\_userid as defaulted for web events) in the table to explain why the event is missing.

Other root causes may be missing timestamp fields (e.g. `dvce_created_tstamp`, `dvce_sent_tstamp` that are needed for the sessionization logic).

If the issue is not related to the session not being processed, but certain fields appear in the views table but not downstream in a more aggregated layer (e.g. the users table) it may simply be due to the nature of the aggregation. There are certain rules to which event per session the packages take a certain field (e.g. first based on collector\_tstamp or derived\_tstamp but only if it is a page\_view event etc.).

To help debug it better, it is best to choose a sample session, select all its events from the events table as a whole, and see based on the compiled model code (when executing dbt locally, you can find it in the target folder) how the joins work to figure out why certain things are not working the way there are "expected". There may be changes that need to take place in tracking or through a custom data model for it to work in a specific use case.

One recommendation when deep-diving into errors is to look at dbt's lineage graph to see the model evolution, and work your way upstream to find out where the data gets filtered out using the compiled sql code from the given run. Once you find the first source when the data starts appearing you know which is the model where it gets filtered out, and you can check the subsequent model's compiled code, then cte by cte you can execute the code. It should not impact the model as those will be select statements, and you will most likely be able to identify the root cause.

> **Info:** You can inspect the lineage graph to verify this if you are unsure by using dbt's built in data lineage feature through dbt docs. We also update that for each of our latest releases on github: <https://snowplow.github.io/dbt-snowplow-unified/#!/overview?g_v=1>

---

# Turn off models
> How to turn off a model in our packages using model configuration in your dbt project yaml file.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/disabling-models/

If you do not require certain modules provided by the package you have the option to disable them using the [model configuration](https://docs.getdbt.com/reference/resource-configs/enabled) in your project yaml. For instance to disable the users module in the `snowplow_unified` package:

```yml
models:
  snowplow_unified:
    users:
      enabled: false
```

For other models, you can identify the path to the models in your `dbt_packages` folder in your project. Note you should disable the entire folder, now just the derived model itself otherwise the [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#other-this-run-tables) tables will still process each run.

> **Tip:** Any dependent modules will also need to be disabled - for instance if you disabled the sessions module in the unified package, you will also have to disable the users module.

---

# Full or partial table refreshes
> Perform full or partial refreshes of Snowplow dbt package models to reprocess data or fix issues.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/

You can choose to refresh all or only some tables.

## Complete refresh of Snowplow package

While you can drop and recompute the incremental tables within this package using the standard `--full-refresh` flag, all [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) are protected from being dropped in production (as defined by a target not matching your `snowplow__dev_target_name`).

Without dropping the manifest during a full refresh, the selected derived incremental tables would be dropped but the processing of events would resume from where the package left off, as captured by the `snowplow_<package_name>_incremental_manifest` table, rather than your `snowplow__start_date`.

In order to drop all the manifest tables and start again set the `snowplow__allow_refresh` var to `true` at run time:

**Other packages:**

```bash
dbt run --select snowplow_<package_name> tag:snowplow_<package>_incremental --full-refresh --vars 'snowplow__allow_refresh: true'
# or using selector flag
dbt run --selector snowplow_<package_name> --full-refresh --vars 'snowplow__allow_refresh: true'
```

**Attribution:**

```bash
dbt run --full-refresh --vars 'snowplow__allow_refresh: true'
```

***

When doing a full refresh of the package, it will begin again from your `snowplow__start_date` and backfill based on the calculations explained in the [Incremental Sessionization Logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) page. Please ensure you trigger enough runs to catch back up with live data, or adjust your variables for these runs accordingly (see our page on [backfilling](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/) for more information).

## Refresh only some models in a package

You may at times wish to refresh only some derived models in a package, either a built in one, or a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/). This may be due to changes you have made to variable, a change in logic, or some other reason. To achieve this:

1. Manually drop the table(s) in your database (you may wish to simply rename them until the back-fill is completed in case of any issues).

2. Remove the models from the manifest table; this can be achieved either by:

   1. _(Recommended)_ using the `models_to_remove` variable at run time

   ```bash
   dbt run --select +snowplow_<package_name>_model_name --vars '{snowplow__start_date: "yyyy-mm-dd", models_to_remove: snowplow_<package>_model_name}'
   ```

   2. _(High Risk)_ manually deleting the record from the `snowplow_<package_name>_incremental_manifest` table.

By removing the model from the manifest the `<package\>` will be in [State 2](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-2-new-model-introduced) and will replay all events.

## Custom period re-processing

There may be times where you need to re-run some or all of the models for a period of data, potentially due to incorrect supplemental data brought into a model, or in case of a mistake in your model code itself. There are two methods to do this: one is simpler but must complete in a single run, the other is more complex but can be executed over multiple runs. Either will work for any custom models, assuming they were built using [Snowplow incremental materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/).

Note that it is not currently possible to re-process a fixed period in the past, it is only possible to re-process from a given date up to the current date of the data.

> **Danger:** Both methods are only suitable if the value of your source data in your `upsert` and date keys have not changed (e.g. `view_id` and `derived_tstamp` for the `snowplow_unified_views` model). If this is not the case, or your custom models are not built using this approach, there is no choice but to run a full refresh of the model.

### Option 1: Altering the look back window

As defined in the [standard run incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#state-4-standard-run), the lower limit for the events to be processed in the models is `max_last_success - snowplow__lookback_window_hours`. If you increase the value of `snowplow__lookback_window_hours` to a number that goes beyond the period you wish to re-run from, then all events from that time will be reprocessed and fed through all models in the package.

For example, if your last run success was `2022-10-30 13:00:00` and you needed to reprocess events from `2022-10-25 02:00:00`, you would set your `snowplow__lookback_window_hours` to `137` (5 days × 24 hours + 11 hours, and 6 hours of an additional buffer the look back window would usually provide). This will reprocess all the events in a single run, which may be larger than the value you have set in `snowplow__backfill_limit_days`. If you wish to avoid going over the backfill limit you have set, please use option 2. Don't forget to change your value back once the run has completed!

![Demonstration of data processing through option 1 approach](/assets/images/data_progress_example1_light.drawio-07bc53a2da644ef13f0119dc545560b2.png)

### Option 2: Manipulating the manifest table

It is possible to update the value in the [manifest tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) so that the `max_last_success` values for the relevant models are set to the date you wish to re-run from. In this case, it may take multiple runs for your data to be completely re-run, based on your `snowplow__backfill_limit_days` value, and you may be left with abnormal data until all runs are completed. In the case where you have aggregations in your models (such as the `snowplow_unified_users` model), or have downstream reports/visualization/ML models based on these tables, we recommend setting the `snowplow__backfill_limit_days` to a large enough value that it can be completed in one run to minimize downstream issues (or to use option 1, where this happens by default).

For example, if your last run success was `2022-10-30 13:00:00` and you needed to reprocess events from `2022-10-25 02:00:00`, you would set the value in your manifest table for the model(s) to `2022-10-25 02:00:00`. This will then process data from that point (minus the `snowplow__lookback_window_hours` buffer) until either the current date, or according to your `snowplow__backfill_limit_days`, whichever yields a smaller time period. This will repeat until the data is fully reprocessed.

![Demonstration of data processing through multiple runs of the option 2 approach](/assets/images/data_progress_example2_light.drawio-7391f248be421c70a9aefa793669b5bc.png)

> **Danger:** Manipulating the values in the manifest tables can cause unexpected outcomes if you don't understand the Snowplow [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/)). Where possible, use option 1.

---

# Operate Snowplow dbt packages
> How to operate our dbt packages including backfilling, model selection, debugging, and asynchronous runs.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/

There are many ways to operate and run our packages, including backfilling or a-sync running of specific tables. The following pages contain guides for how to do this.

## Required Privileges

In addition to the standard [privileges required by dbt](https://docs.getdbt.com/faqs/warehouse/database-privileges), our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have `create schema` privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:

- `<profile_schema>_derived`
- `<profile_schema>_scratch`
- `<profile_schema>_snowplow_manifest`

Alternatively, you can override the output schemas our models write to, see the relevant package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for how to do this.

**Snowflake:**

```sql
  grant create schema on database <database_name> to role <role_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to role <role_name>;
   grant usage on schema <profile_schema>_scratch to role <role_name>;
   grant usage on schema <profile_schema>_manifest to role <role_name>;
```

For more information, please refer to the [Official Guide](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege) on setting up permissions.

**BigQuery:**

Please refer to the [Official Guide](https://cloud.google.com/bigquery/docs/access-control) on setting up permissions.

**Databricks:**

```sql
   -- user with "use catalog" privilege on the catalog
  grant create schema on catalog <catalog_name> to <principal_name>

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to service principal, or group instead of users), please refer to the [Official Guide](https://docs.databricks.com/en/sql/language-manual/security-grant.html) on setting up permissions.

**Redshift:**

```sql
   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to role, or group instead of users), please refer to the [Official Guide](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) on setting up permissions.

**Postgres:**

```sql
   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more information, please refer to the [Official Guide](https://www.postgresql.org/docs/current/sql-createschema.html) on setting up permissions.

***

---

# Run models on data lakes
> How to run our models on lakehouses including Databricks and other data lake platforms.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/lakes/

> **Info:** Running the models on data lakes or lakehouses (using external tables in a warehouse to read directly from a lake) is currently in Early Release state and is not fully supported. Certain features may not work as expected and errors are more likely to occur. Please use this approach at your own risk and raise any issues you find with us.

If you are using the [lake loaders](/docs/api-reference/loaders-storage-targets/lake-loader/) to load your data into a lake storage option, it may be possible to use our data models. In general in this section of the docs we are not going to detail which warehouses support which file formats, or how to set up the respective tables in each warehouse - please see the docs for your appropriate warehouse to see what file formats they support.

## Databricks

At time of writing, `delta` is the preferred file format for Databricks [external tables](https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html). If you create an external table from this lake format in Databricks, you should be able to run the models without any further changes required by simply pointing the model at this table.

## Snowflake

At time of writing, `Iceberg` is the preferred file format for Snowflake [iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg). If you wish to use our models with this, currently only the [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) package supports this, by setting the `snowplow__snowflake_lakeloader` variable to `true`.

Note that compared to the other loaders for Snowflake, that field names in Self-describing events and Entities are converted to `snake_case` format (the other loaders retain the format used in the schema, often `camelCase`). You will need to adjust other variables and inputs accordingly compared to what you may find in the docs.

## Spark

At time of writing, `Iceberg` is the supported file format for Spark external tables. We've tested this using Glue and Thrift as a connection method. If you have your event data in Iceberg format in a lake, you should be able to run the models by pointing the packages to a spark deployment, connected to that lake. For more information on setting up dbt with Spark using Thrift, please refer to the [dbt Spark documentation on Thrift](https://docs.getdbt.com/docs/core/connect-data-platform/spark-setup#thrift).

## Redshift (spectrum)

Currently using Redshift Spectrum tables is not supported for our packages due to [limitations](https://docs.aws.amazon.com/redshift/latest/dg/nested-data-restrictions.html) with the platform.

## BigQuery on GCS

Currently using GCS/BigQuery external tables is not tested but may work, please let us know your experience if you try this.

---

# Model selection
> Select and run specific Snowplow dbt models using tags and selection syntax to keep incremental tables in sync.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/

The Snowplow models in each package are designed to be run as a whole, which ensures all incremental tables are kept in sync. As such, run the model using:

```bash
dbt run --select snowplow_<package> tag:snowplow_<package>_incremental
```

The `snowplow_<package>` selection will execute all nodes within the relevant Snowplow package, while the `tag:snowplow_<package>_incremental` will execute all custom modules that you may have created.

Given the verbose nature of this command we suggest using the YAML selectors we have provided. The equivalent command using the selector flag would be:

```bash
dbt run --selector snowplow_<package>
```

Within the packages we have provided a suite of suggested selectors to run and test the models within the packages. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax).

These are defined in each `selectors.yml` file within the packages, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

---

# Attribution Quickstart
> Quick start guide for the Snowplow Attribution dbt package to analyze marketing attribution and conversion paths.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/attribution/

> **Info:** We are gradually moving tutorials like the QuickStart Guides from our standard Documentation to the **Tutorials & Guides** section. You can now find the latest tutorial on how to get started with the Attribution package [here](/tutorials/attribution/intro/).

---

# Ecommerce Quickstart
> Quick start guide for the Snowplow Ecommerce dbt package to model cart, checkout, product, and transaction data.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/ecommerce/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database, the requirements are:

- A dataset of ecommerce events from the [Snowplow JavaScript tracker](/docs/sources/web-trackers/), or the [iOS/Android](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) trackers must be available in the database.
- Have the [`webPage` context](/docs/sources/web-trackers/tracker-setup/initialization-options/) enabled. (Note if you have only tracked mobile ecommerce events, you will need other events in your warehouse to have used this context as we require the column to exist).
- Have the following ecommerce contexts enabled: `cart`, `checkout_step`, `page` `transaction`, `user`
- Track the ecommerce tracking action events on your website/mobile application

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_ecommerce
  version: 0.9.3
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the ecommerce model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-ecommerce/blob/main/selectors.yml)) within the package, however in order to use these selectors you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_ecommerce:
    snowplow__atomic_schema: schema_with_snowplow_events
    snowplow__database: database_with_snowplow_events
    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events, the `app_id`'s to filter for, and the `event_name` value to filter on. By default the `start_date` is set to `2020-01-01`, all `app_id`'s are selected, and only the `snowplow_ecommerce_action` name is being surfaced. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
vars:
  snowplow_ecommerce:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
    snowplow__ecommerce_event_names: ['snowplow_ecommerce_action', 'my_custom_ecommerce_event']
```

### 5. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>   snowplow_ecommerce:
>     snowplow__derived_tstamp_partitioned: false
> ```

### 6. Removing unused modules

The ecommerce package creates tables that depend on the existence of certain entities that are a part of the [Snowplow ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/) JS plugin. If, for some reason, you have not implemented them and would like to streamline your data modeling not to create empty tables, then you need to add that configuration to your `dbt_project.yml` file. Below you can see an example of what that would look like if you wanted to disable the [cart entity](/docs/events/ootb-data/ecommerce-events/#cart)

#### Disabling the cart module in `dbt_project.yml` (recommended)

```yml

...
vars:
    snowplow_ecommerce:
        snowplow__disable_ecommerce_carts: true
...
models:
    snowplow_ecommerce:
        carts:
            +enabled: false
```

Adding these two configurations to your `dbt_project.yml` will ensure that the carts module is disabled.

#### Disabling the cart module using `dbt run`

If you want to temporarily disable a module, or you just find it easier to use the command line, you can also do this in the command line when executing the `dbt run` command. You will need to run the following command to disable the carts module

```bash
dbt run --exclude carts --select snowplow_ecommerce --vars '\{snowplow__disable_ecommerce_carts: true}'
```

### 7. Enable mobile ecommerce events

Mobile ecommerce events may be processed in the package, if they have a `domain_sessionid` and are in your listed `app_id`s, however to correctly source the mobile session and screen view ids you need to set the following in your `dbt_project.yml`:

```yml
vars:
  snowplow_ecommerce:
    snowplow__enable_mobile_events: true
```

### 8. Optimize your project

There are ways how you can deal with [high volume optimizations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/) at a later stage, if needed, but you can do a lot upfront by selecting carefully which variable to use for `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. This timestamp column should ideally be set to the column your event table is partitioned on. It is defaulted to `collector_tstamp` but depending on your loader it can be the `load_tstamp` as the sensible value to use:

```yml
vars:
  snowplow_ecommerce:
    snowplow__session_timestamp: 'load_tstamp'
```

### 9. Verify your variables using our Config guides (Optional)

If you are unsure whether the default values set are good enough in your case or you would already like to maximize the potential of your models, you can dive deeper into the meaning behind our variables on our [Config](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/) page. It includes a [Config Generator](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/#config-generator) to help you create all your variable configurations, if necessary.

### 10. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_ecommerce
```

---

# Quick start guide for Snowplow dbt packages
> Quick start guides for setting up and running Snowplow dbt packages including database privileges and package installation steps.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/

The following pages contain the steps to get you up and running with our packages. For more detailed information on running the packages see the [individual package pages](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), and the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) pages.

## Database Privileges

In addition to the standard [privileges required by dbt](https://docs.getdbt.com/faqs/warehouse/database-privileges), our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have `create schema` privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:

- `<profile_schema>_derived`
- `<profile_schema>_scratch`
- `<profile_schema>_snowplow_manifest`

Alternatively, you can override the output schemas our models write to, see the relevant package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for how to do this.

**Snowflake:**

```sql
  grant create schema on database <database_name> to role <role_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to role <role_name>;
   grant usage on schema <profile_schema>_scratch to role <role_name>;
   grant usage on schema <profile_schema>_manifest to role <role_name>;
```

For more information, please refer to the [Official Guide](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege) on setting up permissions.

**BigQuery:**

Please refer to the [Official Guide](https://cloud.google.com/bigquery/docs/access-control) on setting up permissions.

**Databricks:**

```sql
   -- user with "use catalog" privilege on the catalog
  grant create schema on catalog <catalog_name> to <principal_name>

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to service principal, or group instead of users), please refer to the [Official Guide](https://docs.databricks.com/en/sql/language-manual/security-grant.html) on setting up permissions.

**Redshift:**

```sql
   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more options (e.g.: granting to role, or group instead of users), please refer to the [Official Guide](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) on setting up permissions.

**Postgres:**

```sql
   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;
```

For more information, please refer to the [Official Guide](https://www.postgresql.org/docs/current/sql-createschema.html) on setting up permissions.

***

## Mixing Variables

> **Warning:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>   snowplow_web:
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>   snowplow_mobile:
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

---

# Fractribution Quickstart
> Quick start guide for the legacy Snowplow Fractribution dbt package for marketing attribution modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/fractribution/

> **Warning:** The Fractribution Package is no longer maintained, please refer to the [Attribution package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) for marketing attribution analysis with Snowplow

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database:

- have `snowplow_web_page_views` derived table available as a source (generated by the [snowplow\_web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/))

- have a table with revenue data by users (`domain_userid`, `user_id`) that serves as another source for the fractribution calculations, you can choose either of the following options:

  - your `atomic.events` table with any [self-describing event](/docs/fundamentals/events/#self-describing-events) that captures revenue data
  - the `snowplow_ecommerce_transaction_interactions` derived table generated by the [snowplow\_ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) package
  - any custom incremental table that is built on top of the `snowplow_web` model that results in an aggregated revenue dataset

- `python` or `docker` installed (or you can use Snowpark on Snowflake)

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_fractribution
  version: 0.3.6
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Set variables

The package has some variables that need to be set before it can be run, you should edit these in your `dbt_project.yml` file. Further customization can be done via the variables listed in the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/fractribution/).

- `snowplow__conversion_window_start_date`: The start date in UTC for the window of conversions to include
- `snowplow__conversion_window_end_date`: The end date in UTC for the window of conversions to include
- `snowplow__conversion_hosts`: `url_hosts` to process
- `snowplow__path_transforms`: A dictionary of path transforms and their arguments (see [Path Transform Options](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/#path-transform-options) section)

```yml
vars:
  snowplow_fractribution:
    snowplow__conversion_window_start_date: '2022-01-01'
    snowplow__conversion_window_end_date: '2023-02-01'
    snowplow__conversion_hosts: ['mysite.com']
    snowplow__path_transforms: \{'exposure_path' : null}
```

> **Tip:** If you are using Snowflake, you can automatically run the python scripts using Snowpark when running the dbt package. This is done using macros that create and run a stored procedure on Snowpark after the dbt models have completed.
>
> To enable this you need to set some additional variables. For example, to enable this and use the `last_touch` attribution model:
>
> ```yml
> vars:
>   snowplow_fractribution:
>     snowplow__run_python_script_in_snowpark: true
>     snowplow__attribution_model_for_snowpark: 'last_touch'
> ```

**Modifying the conversions source**

By default the `snowplow__conversions_source` is your atomic events table. In most cases this is likely to be what you want to use, however you may wish to use the in-built conversions modeling as part of our [web package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/) if you have already defined this, by setting `snowplow__conversions_source` to `"\\{\\{ ref('snowplow_web_sessions') }}\\}"`.

Alternatively, if you are using Redshift/Postgres you may wish to include additional fields from a Self-Describing Event, or an Entity. To do this, you should create a new model in your project, e.g. `models/snowplow/snowplow_joined_events_table.sql` which should have something like the following content:

> _For more information about dealing with duplicates and the macro in this code, make sure to see our [deduplication docs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/)._

```jinja2
with {{ snowplow_utils.get_sde_or_context('atomic',
                                          'my_custom_context',
                                          "'\{\{ get_lookback_date_limits(\"min\") }}\}'",
                                          "'\{\{ get_lookback_date_limits(\"max\") }}\}'",
                                          'my_prefix')}}

select
  events.*,
  b.*
from \\{\\{ source('atomic', 'events') }}\\} as events
  left join nl_basjes_my_prefix_1 b on
    events.event_id = b.my_prefix__id
    and events.collector_tstamp = b.my_prefix__tstamp

where
  -- use the appropriate partition key to filter on in addition to this, add a bit of a buffer if it is not derived_tstamp
  date(derived_tstamp) >= '\{\{ get_lookback_date_limits(\"min\") }}\}'
  and date(derived_tstamp) <= '\{\{ get_lookback_date_limits(\"max\") }}\}'
```

Finally ensure you set the `snowplow__conversions_source` to `"\\{\\{ ref('snowplow_joined_events_table') }}\\}"`

### 3. Configure macros

> **Tip:** While the macro and matching columns/tables use the term _channel_, it is entirely possible to define this in terms of something else e.g. a campaign.

All the below macros are created with the intention to let users modify them to fit their personal use case. If you wish to change this, copy the macro from the macros folder in the `snowplow_fractribution` package (at `[dbt_project_name]/dbt_packages/snowplow_fractribution/macros/conversion_clause.sql`) and add it to the macros folder of your own dbt project where you are free to make any alterations. You will find a detailed guide / illustration with sample code within the individual macros themselves.

> **Tip:** To overwrite these macros correctly with those in your project, ensure you prefix the macro name by `default__` in the definition e.g.
>
> ```jinja2
> {% macro default__conversion_value() %}
>     tr_total/100
> {% endmacro %}
> ```

#### `conversion_clause` macro

> The [`conversion_clause`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_clause.sql) macro specifies how to filter Snowplow events to only succesfful conversion events. How this is filtered will depend on your definition of a conversion. The default is filtering to events where `tr_total > 0`, but this could instead filter on `event_name = 'checkout'`, for example. If you are using the e-commerce model, you will still need to set this for the fractribution code to run (even though _all_ events are conversions in the e-commerce model), in this case change it to `transaction_revenue > 0`.

#### `conversion_value` macro

> The [`conversion_value`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/conversion_value.sql) macro specifies either a single column or a calculated value that represents the value associated with that conversion. The default is `tr_total`, but revenue or a calculation using revenue and discount\_amount from the default e-commerce schema, for example, could similarly be used.

#### `channel_classification` macro

> The [`channel_classification`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_classification.sql) macro is used to perform channel classifications. This can be altered to generate your expected channels if they differ from the channels generated in the default macro. It is highly recommended that you examine and configure this macro when using your own data, as the ROAS calculations and attribution calculations will run against these channel definitions, and the default values will not consider any custom marketing parameters.

#### `channel_spend` macro

> The [`channel_spend`](https://github.com/snowplow/dbt-snowplow-fractribution/blob/main/macros/channel_spend.sql) macro is used to query the spend by channels. It requires a user supplied SQL script to extract the total ad spend by channel.
>
> Required output format:
>
> - `channel`: STRING NOT NULL
> - `spend`: FLOAT64 _(or other warehouse equivalent)_ (Use the same monetary units as conversion revenue, and NULL if unknown.)

### 4. Run the model

Execute the following either through your CLI, within dbt Cloud, or within [Snowplow Console](/docs/modeling-your-data/running-data-models-via-console/)

```yml
dbt run --select snowplow_fractribution
```

### 5. Run the python script to generate the final models

Depending on your setup, please follow the appropriate steps below. All these methods will create the following tables in your warehouse:

- `snowplow_fractribution_path_summary_with_channels`
- `snowplow_fractribution_report_table`
- `snowplow_fractribution_channel_attribution`

**Run on Snowflake using Snowpark**

If you enabled this already, the tables will have already been built as part of step 4. If you wish to just re-run the attribution modeling for some reason you can run the following:

```text
dbt run --select snowplow_fractribution_call_snowpark_macros
```

**Locally run python**

> **Tip:** To run the fractribution script locally in Python, we recommend using a virtual environment such as one in `conda` or `pyenv`.
>
> Example using conda:
>
> ```text
> conda create --name fractribution_env -c https://repo.anaconda.com/pkgs/snowflake python=3.8 absl-py
> conda activate fractribution_env
> ```

#### I. Install packages

You can install the packages using `pip install -r dbt_packages/snowplow_fractribution/utils/requirements.txt` (or the appropriate path from your terminal working directory).

Please note that some of the libraries are adapter specific. These are listed in the `requirements` file, and you can also find the necessary list for each adapter below:

**BigQuery:**

- `absl-py`==`1.2.0`
- `google-cloud-bigquery`==`3.5.0`

**Databricks:**

- `absl-py`==`1.2.0`,
- `databricks-sql-connector`==`2.1.0`
- `pandas`

**Snowflake:**

- `absl-py`==`1.2.0`,
- `snowflake-snowpark-python`==`0.11.0`

**Redshift:**

- `absl-py`==`1.2.0`,
- `redshift_connector`==`2.0.910`

***

**M1 Instructions (for Snowflake only)**

> **Warning:** There is an issue with running Snowpark on M1 chips. A workaround recommended by Snowflake is to set up a virtual environment that uses x86 Python:
>
> ```text
> CONDA_SUBDIR=osx-64 conda create -n fractribution_env python=3.8 absl-py -c https://repo.anaconda.com/pkgs/snowflake
> conda activate fractribution_env
> conda config --env --set subdir osx-64
> ```

#### II. Set the connection parameters in your terminal

**BigQuery:**

```text
export project_id=project_id\
export bigquery_dataset=bigquery_dataset\
export google_application_credentials=google_application_credentials
```

**Databricks:**

```text
export databricks_schema=derived_schema_name\
export databricks_server_hostname=hostname\
export databricks_http_path=http_path\
export databricks_token=token
```

**Snowflake:**

```text
export snowflake_account=my_account\
export snowflake_user=sf_user\
export snowflake_password=password\
export snowflake_user_role=special_role\
export snowflake_warehouse=warehouse_name\
export snowflake_database=database_name\
export snowflake_schema=derived_schema_name
```

**Redshift:**

```text
export redshift_host=redshift_host\
export redshift_database=redshift_database\
export redshift_port=redshift_port\
export redshift_user=redshift_user\
export redshift_password=redshift_password\
export redshift_schema=redshift_schema
```

***

#### III. Run the fractribution script

Run the adapter specific main fractribution script by specifying the conversion window start and end dates, and the attribution model (if you are not using the default `shapley`, see [here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/#attribution-models) for more options). Example:

**BigQuery:**

```text
python main_snowplow_bigquery.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Databricks:**

```text
python main_snowplow_databricks.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Snowflake:**

```text
python main_snowplow_snowflake.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

**Redshift:**

```text
python main_snowplow_redshift.py --conversion_window_start_date '2022-06-03' --conversion_window_end_date '2022-08-01' --attribution_model last_touch
```

***

> **Tip:** To help debug easier we have included logs to be printed each time a table is created in the warehouse, there should be 3 in total (`snowplow_fractribution_path_summary_with_channels`, `snowplow_fractribution_channel_attribution` and `snowplow_fractribution_report_table`). There is also a `--verbose` flag you can include in your shell prompt when executing the python scripts which may give you an even more detailed breakdown depending on your warehouse.

**Run python with docker**

#### I. Pull the docker image ​

You can pull the latest docker image from Docker Hub: `docker pull snowplow/fractribution:latest`. Alternatively, you can pull it based on the package version: `docker pull snowplow/fractribution:0.2.0`

#### II. Set the environment variables

Add the necessary environment variables to an environment file, e.g. `configs.env`. The necessary variables will differ depending on the data warehouse you are using. The easiest way to determine the variables you need to set is to check the Dockerfile in the fractribution dbt package: `dbt-snowplow-fractribution/utils/Dockerfile`. Please note that in case of BigQuery, the `google_application_credentials` env var is not needed for Docker as you mount this as a volume at run time.

Below is an example of the `config.env` file (set up for Snowflake). Note the last 4 variables are named the same across all warehouses. You do not need to specify the attribution model if using the default, `shapley`:

```text
snowflake_account=youraccount.ap-southeast-2
snowflake_user=user
snowflake_password=abc123
snowflake_user_role=DBT
snowflake_warehouse=WH
snowflake_database=snowplow
snowflake_schema=FRACTRIBUTION_DERIVED

conversion_window_start_date=2022-06-03
conversion_window_end_date=2022-08-01
attribution_model=last_touch
warehouse=snowflake
```

#### III. Run the docker container

Run the docker container :

**BigQuery:**

With BigQuery you need to mount your service account keyfile when running the docker image

```text
docker run --rm --env-file /path/to/env/file/configs.env -v /path/to/yourkeyfile.json:/keyfile.json -it snowplow/fractribution:latest
```

**Databricks:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

**Snowflake:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

**Redshift:**

```text
docker run --rm --env-file /path/to/env/file/configs.env -it snowplow/fractribution:latest
```

***

> **Tip:** To help debug easier we have included logs to be printed each time a table is created in the warehouse, there should be 3 in total (`snowplow_fractribution_path_summary_with_channels`, `snowplow_fractribution_channel_attribution` and `snowplow_fractribution_report_table`). In order to run the script with a `--verbose` flag you can set the `verbose_run` variable to "true" in your env-file, which may give you an even more detailed breakdown depending on your warehouse.

---

# Quick start guides for the legacy dbt packages
> Quick start guides for legacy Snowplow dbt packages including Web, Mobile, and Fractribution packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/

> **Warning:** Legacy packages are no longer under active development and are in some cases no longer supported. Please look for a suitable replacement for these packages where possible.

| Package name  | Purpose                                             | Superseded by                                                                                              |
| ------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Web           | Transform raw web event data into derived tables    | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Mobile        | Transform raw mobile event data into derived tables | [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) |
| Fractribution | Marketing attribution analysis                      | [Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) |

---

# Mobile Quickstart
> Quick start guide for the legacy Snowplow Mobile dbt package to model mobile event data into sessions and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/mobile/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a mobile events dataset being available in your database, the requirements are:

- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker version 1.1.0 or later implemented.
- Mobile session context enabled ([ios](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/ios-tracker-1-7-0/#session-context) or [android](/docs/sources/mobile-trackers/previous-versions/android-tracker/android-1-7-0/#session-tracking)).
- Screen view events enabled ([ios](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/ios-tracker-1-7-0/#tracking-features) or [android](/docs/sources/mobile-trackers/previous-versions/android-tracker/android-1-7-0/#tracking-features)).

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_mobile
  version: 1.0.0
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the mobile model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-mobile/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_mobile:
    snowplow__atomic_schema: schema_with_snowplow_events
    snowplow__database: database_with_snowplow_events
    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Enabled desired contexts

The mobile package has the option to join in data from the following 4 Snowplow contexts:

- Mobile context -- Device type, OS, etc.
- Geolocation context -- Device latitude, longitude, bearing, etc.
- Application context -- App version and build
- Screen context -- Screen details associated with mobile event

By default these are **all disabled** in the mobile package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the contexts within the package please modify the following in your `dbt_project.yml` file:

```yml
vars:
  snowplow_mobile:
    snowplow__enable_mobile_context: true
    snowplow__enable_geolocation_context: true
    snowplow__enable_application_context: true
    snowplow__enable_screen_context: true
```

### 5. Enable desired modules

The mobile package has the option to join in data from the following 1 Snowplow module:

- App Errors module -- Details relating to app errors that occur during sessions

By default this module is **disabled** in the mobile package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the module within the package please modify the following in your `dbt_project.yml` file:

```yml
vars:
  snowplow_mobile:
    snowplow__enable_app_errors_module: true
```

### 6. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
vars:
  snowplow_mobile:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
```

### 7. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>   snowplow_mobile:
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>   snowplow_mobile:
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 8. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_mobile
```

---

# Web Quickstart
> Quick start guide for the legacy Snowplow Web dbt package to model web event data into page views, sessions, and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/web/

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web events dataset being available in your database, the requirements are:

- [Snowplow Javascript tracker](/docs/sources/web-trackers/) version 2 or later implemented.
- Web Page context [enabled](/docs/sources/web-trackers/tracker-setup/initialization-options/) (enabled by default in [v3+](/docs/sources/web-trackers/tracker-setup/initialization-options/)).
- [Page view events](/docs/sources/web-trackers/tracking-events/page-views/) implemented.
- From version v0.13.0 onwards you must be using [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v4.0.0 and above, or [BigQuery Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) v1.0.0 and above. If you are not using these versions, or are using the Postgres loader, you will need to set `snowplow__enable_load_tstamp` to `false` in your `dbt_project.yml` and will not be able to use the consent models.

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_web
  version: 1.0.1
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the web model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-web/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_web:
    snowplow__atomic_schema: schema_with_snowplow_events
    snowplow__database: database_with_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Enabled desired contexts

The web package has the option to join in data from the following 3 Snowplow enrichments:

- [IAB enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/)
- [UA Parser enrichment](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)

By default these are **all disabled** in the web package. Assuming you have the enrichments turned on in your Snowplow pipeline, to enable the contexts within the package please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_web:
    snowplow__enable_iab: true
    snowplow__enable_ua: true
    snowplow__enable_yauaa: true
```

### 5. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_web:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
```

### 6. Verify page ping variables

The web package processes page ping events to calculate web page engagement times. If your [tracker configuration](/docs/sources/web-trackers/tracking-events/activity-page-pings/) for `min_visit_length` (default 5) and `heartbeat` (default 10) differs from the defaults provided in this package, you can override by adding to your `dbt_project.yml`:

```yml
vars:
  snowplow_web:
    snowplow__min_visit_length: 5 # Default value
    snowplow__heartbeat: 10 # Default value
```

### 7. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>   snowplow_web:
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>   snowplow_web:
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 8. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package). As this package contains some seed files, you will need to seed these first

```bash
dbt seed --select snowplow_web --full-refresh
dbt run --selector snowplow_web
```

### 9. Enable extras

The package comes with additional modules and functionality that you can enable, for more information see the [consent tracking](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/consent-module/), [conversions](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/conversions/), and [core web vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/core-web-vitals-module/) documentation.

---

# Media Player Quickstart
> Quick start guide for the Snowplow Media Player dbt package to model video and audio engagement data.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/media-player/

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and a web or mobile events dataset being available in your database:

- A dataset of media events must be available in the database. You can collect media events using our plugins for the JavaScript tracker or using the iOS and Android trackers: [Media plugin](/docs/sources/web-trackers/tracking-events/media/), [HTML5 media player plugin](/docs/sources/web-trackers/tracking-events/media/html5/), [YouTube plugin](/docs/sources/web-trackers/tracking-events/media/youtube/), [Vimeo plugin](/docs/sources/web-trackers/tracking-events/media/vimeo/) or the [iOS and Android media APIs](/docs/sources/mobile-trackers/tracking-events/media-tracking/)
- Have the [`webPage` context](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) enabled on Web or the [screen context](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-view-event-and-screen-context-entity) on mobile (enabled by default).
- Enabled session tracking on the tracker (default on mobile).

The model is compatible with all versions of our media tracking APIs. These have evolved over time and may track the media events using two sets of event and contexts schemas:

1. Version 1 media schemas:

   - [media-player event schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player_event/jsonschema/1-0-0) used for all media events.

   - [media-player context v1 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/1-0-0).

   - Depending on the plugin / intention there are player-specific contexts:

     - in case of embedded YouTube tracking: Have the [YouTube specific context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.youtube/youtube/jsonschema/1-0-0) enabled.
     - in case of HTML5 audio or video tracking: Have the [HTML5 media element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/media_element/jsonschema/1-0-0) enabled.
     - in case of HTML5 video tracking: Have the [HTML5 video element context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.whatwg/video_element/jsonschema/1-0-0) enabled.

2. Version 2 media schemas (preferred):

   - [per-event media event schemas](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.media).
   - [media-player context v2 schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/media_player/jsonschema/2-0-0).
   - optional [media-session context schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/session/jsonschema/1-0-0).
   - optional [media-ad](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad/jsonschema/1-0-0) and [ad break](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.media/ad_break/jsonschema/1-0-0) context schema.

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_media_player
  version: 0.9.4
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the media player model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-media-player/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile), in the table labeled `events`. In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_media_player:
    snowplow__atomic_schema: schema_with_snowplow_events
    snowplow__database: database_with_snowplow_events
    snowplow__events_table: table_of_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events, the `app_id`'s to filter for, and the `event_name` value to filter on. By default the `start_date` is set to `2020-01-01`, all `app_id`'s are selected, and all events with the `com.snowplowanalytics.snowplow.media` or the `media_player_event` event name are being surfaced. To change this please add/modify the following in your `dbt_project.yml` file:

```yml
...
vars:
  snowplow_media_player:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
    snowplow__media_event_names: ['media_player_event', 'my_custom_media_event']
```

### 5. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> ...
> vars:
>   snowplow_media_player:
>     snowplow__derived_tstamp_partitioned: false
> ```

### 6. Enable desired contexts and configuration

The media player package creates tables that depend on the existence of certain context entities that are tracked by the media plugins in the Snowplow trackers. Depending on which media plugin or tracking implementation you, you will need to enable the relevant contexts in your `dbt_project.yml`.

#### 6a. Using trackers with support for the version 2 media schemas

This option applied in case you are tracking media events with either the Snowplow Media plugin, Vimeo plugin for JavaScript tracker, or the iOS/Android trackers.

```yaml
...
vars:
  snowplow_media_player:
    # don't use the older version 1 of the media player context schema
    snowplow__enable_media_player_v1: false
    # use the version 2 of the media player context schema
    snowplow__enable_media_player_v2: true
    # use the media session context schema (unless disabled on the tracker)
    snowplow__enable_media_session: true
    # depending on whether you track ads, ad breaks and progress within ads:
    snowplow__enable_media_ad: true
    snowplow__enable_media_ad_break: true
    snowplow__enable_ad_quartile_event: true
    # depending on whether you track events from web or mobile apps:
    snowplow__enable_web_events: true
    snowplow__enable_mobile_events: true
```

#### 6b. Using the HTML5 media tracking plugin for JavaScript tracker

```yaml
...
vars:
  snowplow_media_player:
    # use the version 1 of the media player context schema used by the YouTube plugin
    snowplow__enable_media_player_v1: true
    # don't use the version 2 of the media player context schema as it is not tracked by the plugin
    snowplow__enable_media_player_v2: false
    # don't use the media session context schema as it is not tracked by the plugin
    snowplow__enable_media_session: false
    # set to true if the HTML5 media element context schema is enabled
    snowplow__enable_whatwg_media: true
    # set to true if the HTML5 video element context schema is enabled
    snowplow__enable_whatwg_video: true
```

#### 6c. Using the YouTube tracking plugin for JavaScript tracker

```yaml
...
vars:
  snowplow_media_player:
    # use the version 1 of the media player context schema used by the YouTube plugin
    snowplow__enable_media_player_v1: true
    # don't use the version 2 of the media player context schema as it is not tracked by the plugin
    snowplow__enable_media_player_v2: false
    # don't use the media session context schema as it is not tracked by the plugin
    snowplow__enable_media_session: false
    # set to true if the YouTube context schema is enabled
    snowplow__enable_youtube: true
```

For other variables you can configure please see the [variables](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#variables) section.

### 7. Optimize your project

There are ways how you can deal with [high volume optimizations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/) at a later stage, if needed, but you can do a lot upfront by selecting carefully which variable to use for `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. This timestamp column should ideally be set to the column your event table is partitioned on. It is defaulted to `collector_tstamp` but depending on your loader it can be the `load_tstamp` as the sensible value to use:

```yml
vars:
  snowplow_media_player:
    snowplow__session_timestamp: 'load_tstamp'
```

### 8. Verify your variables using our Config guides (Optional)

If you are unsure whether the default values set are good enough in your case or you would already like to maximize the potential of your models, you can dive deeper into the meaning behind our variables on our [Config](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/) page. It includes a [Config Generator](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/media-player/#config-generator) to help you create all your variable configurations, if necessary.

### 9. Run your model

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_media_player
```

---

# Normalize Quickstart
> Quick start guide for the Snowplow Normalize dbt package to flatten and normalize events and entities for downstream tools.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/normalize/

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed:

- Python 3.7 or later

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_normalize
  version: 0.4.1
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the normalize model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [YAML Selectors](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) section.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-normalize/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

### 3. Check source data

This package will by default assume your Snowplow events data is contained in the `atomic` schema of your [target.database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). In order to change this, please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_normalize:
    snowplow__atomic_schema: schema_with_snowplow_events
    snowplow__database: database_with_snowplow_events
```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.

### 4. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_normalize:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
```

> **Note:** If you have events you are going to normalize with no value for the `dvce_sent_tstamp` field, you need to disable the days late filter by setting the `snowplow__days_late_allowed` variable to `-1`, otherwise these events will not be processed.

### 5. Install additional python packages

The script only requires 2 additional packages (`jsonschema` and `requests`) that are not built into python by default, you can install these by running the below command, or by installing them by your preferred method.

```bash
pip install -r dbt_packages/snowplow_normalize/utils/requirements.txt
```

### 6. Setup the generator configuration file

You can use the example provided in `utils/example_normalize_config.json` to start your configuration file to specify which events, self-describing events, and contexts you wish to include in each table. For more information on this file see the [normalize package docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/).

### 7. Setup your resolver connection file _(optional)_

If you are not using iglu central as your only iglu registry then you will need to set up an [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) file and point to this in your generator config.

### 8. Generate your models

At the root of your dbt project, running `python dbt_packages/snowplow_normalize/utils/snowplow_normalize_model_gen.py path/to/your/config.json` will generate all models specified in your configuration.

### 9. Additional vendor specific configuration

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>   snowplow_normalize:
>     snowplow__derived_tstamp_partitioned: false
> ```

> **Info:** Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>   snowplow_normalize:
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> **A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).**

### 10. Change the default partition timestamp _(optional)_

The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:
  snowplow__partition_tstamp: "collector_tstamp"  # Default value, any change should be a timestamp
```

The purpose of this variable is to adjust the partitioning of the derived tables to use a different timestamp (e.g., derived\_tstamp) that is more suitable for analytics in the next layer.

> **Warning:** If you change `snowplow__partition_tstamp` to a different column (e.g., `load_tstamp`), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.
>
> Example configuration when using a custom partition timestamp:
>
> ```json
> {
>     "events": [
>         {
>             "event_names": ["page_view"],
>             "event_columns": [
>                 "domain_userid",
>                 "load_tstamp",  // Must include your custom partition timestamp here
>                 "app_id"
>             ],
>             // ... rest of configuration
>         }
>     ]
> }
> ```

### 11. Run your model(s)

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/) page for more information on operation of the package):

```bash
dbt run --selector snowplow_normalize
```

---

# Unified Digital Quickstart
> Quick start guide for the Snowplow Unified Digital dbt package to model web and mobile event data into views, sessions, and users.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/unified/

> **Info:** We are gradually moving tutorials like the QuickStart Guides from our standard Documentation to the **Tutorials & Guides** section. You can now find the latest tutorial on how to get started with the Unified Digital package [here](/tutorials/unified-digital/intro/).

---

# Utils Quickstart
> Quick start guide for the Snowplow Utils dbt package with core macros and helper functions for Snowplow dbt packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/

> **Info:** The models, functionality, and variables described below are only available from `snowplow-utils v0.15.0` and above, as earlier packages do not utilize these variables.

The `snowplow-utils` package allows you to create your own custom `snowplow_base_events_this_run` table using macros that generate the required SQL code for you, allowing you to incorporate whatever custom event types, contexts of Snowplow data. Using this package will allow you to leverage the [incremental nature](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of the Snowplow packages, meaning you can more easily build data models using our other packages such as `snowplow-web` and `snowplow-mobile`, as well as build your own completely custom packages without having to do the initial heavy lifting yourself. This can, however, be a bit complicated to set up and so for that purpose we've create this quickstart page to guide you through this process.

> **Info:** It is only recommended that you use this if you are planning on heavily customizing your Snowplow data modeling setup, whilst still taking advantage of the incremental framework that the existing dbt packages offer. If you are going to be heavily leveraging the existing Snowplow packages (e.g. snowplow-unified) then you will not need to leverage this package for the creation of your base tables. Please instead follow the appropriate quickstart guides for the packages you are going to be utilizing instead, such as for [web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/web/) or [mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/legacy/mobile/).

## Requirements

In addition to [dbt](https://github.com/dbt-labs/dbt) being installed and connected to your database you need:

- A dataset of Snowplow events from one of the [Snowplow trackers](/docs/sources/)

## Installation

Make sure to create a new dbt project and import this package via the `packages.yml` as [recommended by dbt](https://docs.getdbt.com/docs/build/packages#how-do-i-add-a-package-to-my-project), or add to an existing top level project. Do not fork the packages themselves.

Check [dbt Hub](https://hub.getdbt.com/snowplow/) for the latest installation instructions, or read the [dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/package-management) for more information on installing packages. If you are using multiple packages you may need to up/downgrade a specific package to ensure compatibility.

```yaml

packages:
- package: snowplow/snowplow_utils
  version: 1.0.0
```

> **Note:** Make sure to run the `dbt deps` command after updating your `packages.yml` to ensure you have the specified version of each package installed in your project.

## Setup

> **Info:** You can largely skip redundant copy + pasting by cloning the following dbt project repository that we have created in GitHub. You can find it [here](https://github.com/snowplow-incubator/dbt-example-project), and this has all of the boilerplate setup for you already. If you want to customize model names or parameter values, you can still follow the quickstart guide below to help you understand how to do that, and what changing each variable will mean for your models. Feel free to skip steps 1 and 2, however.

### 1. Override the dispatch order in your project

To take advantage of the optimized upsert that the Snowplow packages offer you need to ensure that certain macros are called from `snowplow_utils` first before `dbt-core`. This can be achieved by adding the following to the top level of your `dbt_project.yml` file:

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

If you do not do this the package will still work, but the incremental upserts will become more costly over time.

### 2. Creating the `base` module in your dbt project

In your dbt project, create a `base` folder somewhere within your `models` directory. Within this `base` folder, you need to create the 6 `.sql` files shown below:

```ascii
my_dbt_project
├── analyses
│   └── .gitkeep
├── dbt_packages
│   ├── dbt_utils
│   └── snowplow_utils
├── logs
├── macros
├── models
│   └── base
│       ├── snowplow_base_quarantined_sessions.sql
│       ├── snowplow_incremental_manifest.sql
│       ├── snowplow_base_new_event_limits.sql
│       ├── snowplow_base_sessions_lifecycle_manifest.sql
│       ├── snowplow_base_sessions_this_run.sql
│       └── snowplow_base_events_this_run.sql
├── seeds
├── snapshots
├── target
├── tests
├── .gitignore
├── dbt_project.yml
├── packages.yml
├── README.md
└── selectors.yml
```

Once you've created all of these models, you need to call the correct macros in each model to ensure that the correct SQL gets generated for each model. If you'd like to rename any of the models, all you need to do is rename the `.sql` files listed above.

> **Warning:** Please only rename the models with caution and be sure to read the subsequent steps carefully as you may need to modify some of the boilerplate code outlined below to have the macros adapt properly to your naming conventions.

### 3. Setting up the quarantined sessions macro

Within the `snowplow_base_quarantined_sessions.sql` file, you can call the `base_create_snowplow_quarantined_sessions` macro as follows:

```jinja2
{{
  config(
    materialized='incremental'
  )
}}

{% set quarantined_query = snowplow_utils.base_create_snowplow_quarantined_sessions() %}

{{ quarantined_query }}
```

This macro doesn't accept any arguments, and simply generates a table which contains a column named `session_identifier`, containing all session identifiers of sessions that have been quarantined due to exceeding the maximum session length, to avoid long table scans. More information on the sessionization logic and optimization can be found [here](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/)

### 4. Setting up the incremental manifest macro

Next, within the `snowplow_incremental_manifest.sql` file, you can call the `base_create_snowplow_incremental_manifest` macro as follows:

```jinja2
{{
  config(
    materialized='incremental'
  )
}}

{% set incremental_manifest_query = snowplow_utils.base_create_snowplow_incremental_manifest() %}

{{ incremental_manifest_query }}
```

Much like with the quarantined sessions macro, this does not require any arguments and generates the boilerplate for the incremental manifest table that Snowplow leverages.

### 5. Setting up the new event limits macro

For the `snowplow_base_new_event_limits` model, you need to add a few extra macros into the mix, which you can do as follows:

> **Info:** Be sure to specify your `PACKAGE_NAME` when calling the `get_enabled_snowplow_models` macro.

```jinja2
{{
  config(
   post_hook=["\\{\\{snowplow_utils.print_run_limits(this)}}\\}"]
   )
}}

{%- set models_in_run = snowplow_utils.get_enabled_snowplow_models(PACKAGE_NAME, graph_object=none, models_to_run="", base_events_table_name='snowplow_base_events_this_run') -%}

{% set min_last_success,
         max_last_success,
         models_matched_from_manifest,
         has_matched_all_models = snowplow_utils.get_incremental_manifest_status(ref('snowplow_incremental_manifest'),
                                                                                 models_in_run) -%}


{% set run_limits_query = snowplow_utils.get_run_limits(min_last_success,
                                                          max_last_success,
                                                          models_matched_from_manifest,
                                                          has_matched_all_models,
                                                          var("snowplow__start_date")) -%}


{{ run_limits_query }}
```

Here there are a couple of variables that can be used to modify the `new_event_limits` table setup. Firstly, if you chose to name your `snowplow_incremental_manifest` model differently, be sure to reference that properly in the `get_incremental_manifest_status` macro call. If for example, you chose to call the manifest model `incremental_manifest` by naming your file `incremental_manifest.sql` instead of the prescribed `snowplow_incremental_manifest.sql`, then you would reflect that using the following macro call:

> **Info:** The below is just an example of a macro call when using custom naming, this isn't something additional to copy and add if you want to set up your `snowplow_new_event_limits` model!

```jinja2
{% set min_last_success,
         max_last_success,
         models_matched_from_manifest,
         has_matched_all_models = snowplow_utils.get_incremental_manifest_status(ref('incremental_manifest'),
                                                                                 models_in_run) -%}
```

Secondly, you can choose the starting date of when your Snowplow data was first loaded into your data warehouse/lake. This is reflected in the value of the `snowplow__start_date` variable, which you may be familiar with if you've used previous Snowplow dbt packages. This variable is typically defined in your `dbt_project.yml`, and can be defined in that file in the following manner.

```yml

vars:
    # to define it globally, use the following notation
    snowplow__start_date: '2020-01-01'

    # to define the variable locally within the project, use the following nesting
    my_dbt_project:
        snowplow__start_date: '2020-01-01'
```

### 6. Setting up the sessions lifecycle manifest macro

For the `snowplow_base_sessions_lifecycle_manifest` model, you have the following macro call which takes in a lot of parameters to allow for a high level of flexibility in how you can process your Snowplow data:

```jinja2

{% set sessions_lifecycle_manifest_query = snowplow_utils.base_create_snowplow_sessions_lifecycle_manifest(
    session_identifiers=var('snowplow__session_identifiers', '[\\{"schema": "atomic", "field": "domain_sessionid"}]'),
    session_sql=var('snowplow__session_sql', none),
    session_timestamp=var('snowplow__session_timestamp', 'collector_tstamp'),
    user_identifiers=var('snowplow__user_identifiers', '[\\{"schema": "atomic", "field": "domain_userid"}]'),
    user_sql=var('snowplow__user_sql', none),
    quarantined_sessions=var('snowplow__quarantined_sessions', 'snowplow_base_quarantined_sessions'),
    derived_tstamp_partitioned=var('snowplow__derived_tstamp_partitioned', true),
    days_late_allowed=var('snowplow__days_late_allowed', 3),
    max_session_days=var('snowplow__max_session_days', 3),
    app_ids=var('snowplow__app_ids', []),
    snowplow_events_database=var('snowplow__events_schema', none),
    snowplow_events_schema=var('snowplow__events_schema', 'atomic'),
    snowplow_events_table=var('snowplow__events_table', 'events'),
    event_limits_table=var('snowplow__event_limits', 'snowplow_base_new_event_limits'),
    incremental_manifest_table=var('snowplow__incremental_manifest', 'snowplow_incremental_manifest'),
    package_name=var('snowplow__package_name, 'snowplow')
 ) %}

{{ sessions_lifecycle_manifest_query }}
```

To get an in-depth explanation of each variable passed here, please refer to the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) page.

There are several important parameters to consider. The first one is `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. It's recommended to use either `collector_tstamp` or `load_tstamp` as sensible values.

Next, we have `snowplow__session_identifiers` and `snowplow__user_identifiers`, which expect a map object defining the location of session or user identifiers. In this map, the key represents the name of the context or entity where the identifier can be found. If the identifier is a field in the atomic columns, the key can be set to `atomic`. The value specifies the name of the field in either the context/entity or the atomic columns.

> **Warning:** Currently, we only support session and user identifiers found in atomic fields for Redshift/Postgres. We don't support nested level fields for any warehouses, and for BigQuery you will currently need to do the version management yourself. We will be getting around to supporting this extra functionality soon.

By default, `snowplow__session_identifiers` is set to `[\\{"schema": "atomic", "field": "domain_sessionid"}]`, and `snowplow__user_identifiers` is set to `[\\{"schema": "atomic", "field": "domain_userid"}]`. This means that the identifiers for sessions and users are expected to be found in the `domain_sessionid` and `domain_userid` fields, respectively.

If you have more than one session or user identifier, you can specify multiple entries in the map. The order in which you list them determines the precedence that the macro will use to look for these field values, and `COALESCE` them into the common session/user\_identifier field. E.g. if you have the following definition for your `user_identifier`:

**BigQuery, Databricks, & Snowflake:**

```json

[
\{"schema": "my_custom_context", "field": "internal_user_id"},
\{"schema": "atomic", "field": "domain_userid"}
]
```

**Redshift & Postgres:**

```json

[
\{"schema": "my_custom_context", "field": "internal_user_id", "prefix": "mcc", "alias": "mcc_iud"},
\{"schema": "atomic", "field": "domain_userid"}
]
```

For Redshift & Postgres we also introduce the `prefix` and `alias` fields, where `prefix` is the `prefix` that is put in front of each field name in the context, and `alias` is the table alias used upon joining. This can be useful when you are using custom SQL. As an example, using the above configuration we could access the `internal_user_id` field using the following SQL:

```sql
mcc_iud.mcc_internal_user_id as internal_user_id,
```

This could be leveraged in the `snowplow__custom_sql` variable. For more examples, please see [Custom Sessionization & Users](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/).

***

The package will first extract `internal_user_id` from the `my_custom_context` context, and then use something similar to the following SQL statement: `COALESCE(my_custom_context.internal_user_id, events.domain_userid) as user_identifier`. This way, if a user is able to identify themselves through logging in which would populate a context called `my_custom_context`, their `internal_user_id` is used as a `user_identifier`. If, however, this is not the case, then the `user_identifier` field falls back on the value that the `domain_userid` has.

> **Info:** We currently only track one `user_identifier` value per session in the `session_lifecycle_manifest`, which means if a user logs in part-way through a session, we would only keep one of those values.

Further, you can specify some additional configurations here such as in which table/schema the events data sits, what the names are of your `event_limits` and `incremental_manifest` tables, and some parameters around what the maximum session length is. You should once again be familiar with the majority of these variables if you've used another of our dbt packages before.

### 7. Setting up the sessions this run macro

For the `snowplow_base_sessions_this_run` model, you will need to add a post-hook to the configuration of the model as follows:

```jinja2
{{
    config(
        post_hook=["\\{\\{ snowplow_utils.base_quarantine_sessions(var('snowplow__max_session_days', 3), var('snowplow__quarantined_sessions', 'snowplow_base_quarantined_sessions')) }}\\}"]
    )
}}


{% set sessions_query = snowplow_utils.base_create_snowplow_sessions_this_run(
    lifecycle_manifest_table='snowplow_base_sessions_lifecycle_manifest',
    new_event_limits_table='snowplow_base_new_event_limits') %}

{{ sessions_query }}
```

Here the parameters that are called in both macros are only used to direct the macro to the right model names, so again if you've chosen to modify any of the table names then you should adjust the names in the right macros here. For the `base_quarantine_sessions` macro you simply pass the maximum session duration in days, which is taken from the `snowplow__max_session_days` variable, and you specify the name of the `snowplow_base_quarantined_sessions` table, specified by the `snowplow__quarantined_sessions` variable.

For the `base_create_snowplow_sessions_this_run` macro call, you specify the name of the `lifecycle_manifest_table` and the `new_event_limits_table`. The boilerplate contains their default names, and so if you have not customized anything you can simply copy this code into your `snowplow_base_sessions_this_run` model.

### 8. Setting up the events this run macro

For the `snowplow_base_events_this_run` model, you will need to run the following two macros in your model:

```jinja2
{%- set lower_limit, upper_limit = snowplow_utils.return_limits_from_model(ref(var('snowplow__base_sessions', 'snowplow_base_sessions_this_run')),
                                                                          'start_tstamp',
                                                                          'end_tstamp') %}

{% set base_events_query = snowplow_utils.base_create_snowplow_events_this_run(
    sessions_this_run_table=var('snowplow__base_sessions', 'snowplow_base_sessions_this_run'),
    session_identifiers=var('snowplow__session_identifiers', '[\\{"schema": "atomic", "field": "domain_sessionid"}]'),
    session_sql=var('snowplow__session_sql', none),
    session_timestamp=var('snowplow__session_timestamp', 'collector_tstamp'),
    derived_tstamp_partitioned=var('snowplow__derived_tstamp_partitioned', true),
    days_late_allowed=var('snowplow__days_late_allowed', 3),
    max_session_days=var('snowplow__max_session_days', 3),
    app_ids=var('snowplow__app_ids', []),
    snowplow_events_database=var('snowplow__events_schema', none),
    snowplow_events_schema=var('snowplow__events_schema', 'atomic'),
    snowplow_events_table=var('snowplow__events_table', 'events')) %}

{{ base_events_query }}
```

Here you once again have a number of parameters that the macro can take, and to get an in-depth explanation of each variable passed here, please refer to the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/utils/) page. The variables used here are largely either self-explanatory or overlapping with those in the [lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/utils/#6-setting-up-the-sessions-lifecycle-manifest-macro) section, except that you can now specify custom names for your `snowplow_base_sessions_this_run` table through the `snowplow__base_sessions` variable.

### 9. Modify your `dbt_project.yml`

To properly configure your dbt project to utilize and update the manifest tables correctly, you will need to add the following hooks to your `dbt_project.yml`

```yml
# Completely or partially remove models from the manifest during run start.
on-run-start:
  - "\\{\\{ snowplow_utils.snowplow_delete_from_manifest(var('models_to_remove',[]), ref('snowplow_incremental_manifest')) }}\\}"

# Update manifest table with last event consumed per sucessfully executed node/model
on-run-end:
  - "\\{\\{ snowplow_utils.snowplow_incremental_post_hook(package_name='snowplow', incremental_manifest_table_name=var('snowplow__incremental_manifest', 'snowplow_incremental_manifest'), base_events_this_run_table_name='snowplow_base_events_this_run', session_timestamp=var('snowplow__session_timestamp')) }}\\}"
```

The `snowplow_delete_from_manifest` macro is called to remove models from manifest if specified using the `models_to_remove` variable, in case of a partial or full refresh. The `snowplow_incremental_post_hook` is used to update the manifest table with the timestamp of the last event consumed successfully for each Snowplow incremental model - make sure to change the `base_events_this_run_table_name` if you used a different table name.

> **Tip:** The `package_name` variable here is not necessarily the name of your project (although it keeps things simple to make it the same), instead it is what is used to identify your tagged incremental models as they should be tagged with `<package_name>_incremental`.

### 10. Run your models

Now that you've configured all of your macros correctly, you can run your dbt project as normal. If you want to try to only run the models you've just created above, you can issue the following command in your terminal:

```bash
dbt run --select +snowplow_base_events_this_run
```

You will have to change the name of the model if you've customized this earlier.

---

# Model your data with dbt
> Information for our dbt packages including quickstarts, configurations, and building custom models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/

[dbt](https://docs.getdbt.com/) enables analytics engineers to transform data in their warehouses by simply writing select statements. Snowplow has written and maintain a number of dbt packages to model your snowplow data for various purposes and produce derived tables for use in analytics, AI, ML, BI, or reverse ETL tools.

Using Snowplow's dbt packages means you can draw insight and value from your data _quicker_, _easier_, and _cheaper_ than building your own modeling from scratch.

![Snowplow Data Modeling Packages](/assets/images/dbt_packages-light.drawio-9f38a60a3afc9a1eb93d500f6f303803.png)

To set up dbt, Snowplow Self-Hosted users can start with the [dbt User Guide](https://docs.getdbt.com/guides/getting-started) and then we have prepared some [introduction videos](https://www.youtube.com/watch?v=1kd6BJhC4BE) for working with the Snowplow dbt packages.

For Snowplow CDI customers, dbt projects can be configured and scheduled in the console meaning you can [get started](/docs/modeling-your-data/running-data-models-via-console/) running dbt models alongside your Snowplow pipelines.

## Snowplow dbt Packages

> **Info:** Our dbt packages are available under a mix of licenses. For more information about how to get access to these packages, please contact us by requesting a demo if you are new to Snowplow or by reaching out to our support team.

Our dbt packages come with powerful built-in features such as an [optimization to the incremental materialization](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/) to save you cost on warehouse compute resources compared to the standard method, a custom [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) to ensure we process just the required data for each run and keep your models in sync, plus the ability to build your own custom models using both of these!

There are 4 core snowplow dbt packages:

- [Snowplow Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-unified/#!/overview/snowplow_unified)): for modeling your web and mobile data for page and screen views, sessions, users, and consent
- [Snowplow Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-media-player/#!/overview/snowplow_media_player)): for modeling your media elements for play statistics
- [Snowplow E-commerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-ecommerce/#!/overview/snowplow_ecommerce)): for modeling your E-commerce interactions across carts, products, checkouts, and transactions
- [Snowplow Attribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-attribution/#!/overview/attribution)): used for Attribution Modeling with Snowplow

We also have 2 utility packages:

- [Snowplow Normalize](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-normalize-data-model/) package that makes it easy for you to build models that transform your events data into a different structure that may be better suited for downstream consumers
- [Snowplow Utils](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-utils-data-model/) contains all our shared macros and features used across our packages

There are also 3 legacy dbt packages for web, mobile (superseded by unified) and fractribution (superseeded by attribution):

- [Snowplow Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-web/#!/overview/snowplow_web)): for modeling your web data for page views, sessions, users, and consent
- [Snowplow Mobile](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-mobile-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-mobile/#!/overview/snowplow_mobile)): for modeling your mobile app data for screen views, sessions, users, and crashes
- [Snowplow Fractribution](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-fractribution-data-model/) ([dbt model docs](https://snowplow.github.io/dbt-snowplow-fractribution/#!/overview/fractribution)): used for Attribution Modeling with Snowplow

Each package comes with a set of standard models to take your [Snowplow tracker data](/docs/sources/) and produce tables aggregated to different levels, or to perform analysis for you. You can also add your own models on top, see the page on [custom modules](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) for more information on how to do this.

## Package Versions & dbt Compatibility

Starting with v1.0.0, Snowplow dbt packages adopt stricter syntax requirements introduced in dbt Core ≥ 1.10.6 (for example, nested generic test arguments). Earlier dbt versions are not supported from this release onward. Additionally, for Redshift users only, redshift-adapters 1.10 is the minimum requirement going forward.

Snowplow maintains two active release lines:

- v1.x (latest) – recommended for all users, it is where ongoing development occurs
- v0.x (maintenance) – receives critical bug fixes only, where possible

For users upgrading from v0.x to v1.x, see our [migration guides](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/#upgrading-to-100)

### Note on dbt Fusion

While these packages are architecturally compatible with the dbt Fusion engine, Snowplow packages are not officially supported on dbt Fusion at this time. Limited testing may be possible using the [--no-version-check](https://docs.getdbt.com/reference/project-configs/require-dbt-version#fusion-compatibility) flag. Behavior can be fragile, particularly during full-refreshes or when introducing new models, and is under investigation.

The supported data warehouses per version can be seen below:

**Snowplow Unified Digital:**

| snowplow-unified version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ------------------------ | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 1.0.0                    | >=1.10.6 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.4.5                    | >=1.6.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |

**Snowplow Media Player:**

| snowplow-media-player version | snowplow-web version | dbt versions       | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ----------------------------- | -------------------- | ------------------ | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.9.4                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ✅     |
| 0.8.0                         | N/A                  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.5.3                         | >=0.14.0 to <0.16.0  | >=1.4.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.2                         | >=0.13.0 to <0.14.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.4.1                         | >=0.12.0 to <0.13.0  | >=1.3.0 to <2.0.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.3.4                         | >=0.9.0 to <0.12.0   | >=1.0.0 to <1.3.0  | ✅        | ✅          | ✅        | ✅         | ✅        | ❌     |
| 0.1.0                         | >=0.6.0 to <0.7.0    | >=0.20.0 to <1.1.0 | ❌        | ❌          | ✅        | ❌         | ✅        | ❌     |

**Snowplow Normalize:**

| snowplow-normalize version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.4.1                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ✅     |
| 0.3.5                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.3                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.1.0                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

**Snowplow Ecommerce:**

| snowplow-ecommerce version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.9.3                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ✅     |
| 0.8.2                      | >=1.4.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ⚠️       | ❌     |
| 0.3.0                      | >=1.3.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |
| 0.2.1                      | >=1.0.0 to <2.0.0 | ✅        | ✅          | ❌        | ✅         | ❌        | ❌     |

Postgres is technically supported in the models within the package, however one of the contexts’ names is too long to be loaded via the Postgres Loader.

**Snowplow Attribution:**

| snowplow-attribution version | dbt versions      | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| ---------------------------- | ----------------- | -------- | ---------- | -------- | --------- | -------- | ----- |
| 0.6.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ✅     |
| 0.3.0                        | >=1.6.0 to <2.0.0 | ✅        | ✅          | ✅        | ✅         | ❌        | ❌     |

***

## dbt Version Compatibility Checker

You may be using other dbt packages that require you to use a specific version of dbt, or a specific version of the `dbt_utils` package. Or you simply may not wish to upgrade your existing setup.

Here’s a way to check the latest version of our packages you can install for a given setup. Simply enter your dbt version and (optionally) `dbt_utils` version below.

_Please enter a valid dbt version._

---

# Migration guide for Attribution
> Migration guide for upgrading the Snowplow Attribution dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/attribution/

### Upgrading to 0.6.0

- We had to introduce limitations for Redshift users:

  1. users are only allowed to have 1 path transformation defined within the `snowplow__path_transforms` variable
  2. the `snowplow__path_transforms` variable parameters only accept single-item arrays for `remove_if_last_and_not_all` and `remove_if_not_all`

- This is due to Redshift deprecating Python UDFs; the transformations now rely on native SQL, which allows less flexibility for automation.

- Existing users can stay on older versions of the package as long as their UDFs remain in the database.

```yml
vars:
  snowplow_attribution:
    snowplow__path_transforms: {'exposure_path': null}
```

### Upgrading to 0.5.0

- From now on the `snowplow__path_transforms` variable parameters only accept non-empty arrays for `remove_if_last_and_not_all` and `remove_if_not_all` variables instead of strings, please your variable overwrites in your dbt\_project.yml accordingly. Previously you could only remove one specific channel or campaign, now you can do multiple, if needed.

```yml
vars:
  snowplow_attribution:
    snowplow__path_transforms: {'exposure_path': null, 'remove_if_last_and_not_all': ['channel_to_remove_1', 'campaign_to_remove_1, 'campaign_to_remove_2']}
```

---

# Migration guide for Ecommerce
> Migration guide for upgrading the Snowplow Ecommerce dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/ecommerce/

### Upgrading to 0.6.0

Note the minimum `dbt-core` version now required is 1.5.0

Most upgrades and new features should be available automatically after the first run of the new version of the package, however there are a few breaking changes to manage before this first run. To enable any other new optional features or make use of the new configuration options, please see the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/ecommerce/) page.

Both the quarantined sessions and the sessions lifecycle table have has a column renamed, please run the below statements to rename this column in your warehouse if not doing a full-refresh of the package.

```sql
alter table <your_schema>_snowplow_manifest.snowplow_ecommerce_base_sessions_lifecycle_manifest rename column session_id to session_identifier;
alter table <your_schema>_snowplow_manifest.snowplow_ecommerce_base_quarantined_sessions rename column session_id to session_identifier;
```

### Upgrading to 0.5.0

**This version requires a full refresh run if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

To enable modeling mobile events, set `snowplow__enable_mobile_events` to `true`.

### Upgrading to 0.4.2

Two of the derived tables need to be altered for existing Snowflake, Databricks or Redshift users. Please modify the below script to fit your schemas and apply them before running the upgraded package.

The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

**SQL scripts**

**snowflake:**

```sql
alter table (your_schema)_derived.snowplow_ecommerce_cart_interactions alter column cart_total_value type decimal(9,2);
alter table (your_schema)_derived.snowplow_ecommerce_transaction_interactions alter column transaction_revenue type decimal(9,2), transaction_discount_amount type decimal(9,2), transaction_shipping type decimal(9,2), transaction_tax type decimal(9,2);
```

**databricks:**

```sql
create table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new
(
    event_id               STRING,
    page_view_id           STRING,
    domain_sessionid       STRING,
    event_in_session_index INT,
    domain_userid          STRING,
    network_userid         STRING,
    user_id                STRING,
    ecommerce_user_id      STRING,
    derived_tstamp         TIMESTAMP,
    derived_tstamp_date    DATE,
    cart_id                STRING,
    cart_currency          STRING,
    cart_total_value       DECIMAL(9, 2),
    cart_created           BOOLEAN,
    cart_emptied           BOOLEAN,
    cart_transacted        BOOLEAN,
    ecommerce_action_type  STRING
);

insert into (your_schema)_derived.snowplow_ecommerce_cart_interactions_new  select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions;
drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions;
create table (your_schema)_derived.snowplow_ecommerce_cart_interactions select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions_new;
drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new;

create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new
(
    event_id                    STRING,
    page_view_id                STRING,
    domain_sessionid            STRING,
    event_in_session_index      INT,
    domain_userid               STRING,
    network_userid              STRING,
    user_id                     STRING,
    ecommerce_user_id           STRING,
    derived_tstamp              TIMESTAMP,
    derived_tstamp_date         DATE,
    transaction_id              STRING,
    transaction_currency        STRING,
    transaction_payment_method  STRING,
    transaction_revenue         DECIMAL(9, 2),
    transaction_total_quantity  INT,
    transaction_credit_order    BOOLEAN,
    transaction_discount_amount DECIMAL(9, 2),
    transaction_discount_code   STRING,
    transaction_shipping        DECIMAL(9, 2),
    transaction_tax             DECIMAL(9, 2),
    ecommerce_user_email        STRING,
    ecommerce_user_is_guest     BOOLEAN,
    number_products             BIGINT
);

insert into (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new    select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions;
drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions;
create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new ;
drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new;
```

**redshift:**

```sql
create table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new
(
    event_id CHAR(36)   ENCODE lzo
    ,page_view_id VARCHAR(4096)   ENCODE lzo
    ,domain_sessionid CHAR(128)   ENCODE lzo
    ,event_in_session_index BIGINT   ENCODE az64
    ,domain_userid VARCHAR(128)   ENCODE lzo
    ,network_userid VARCHAR(128)   ENCODE lzo
    ,user_id VARCHAR(255)   ENCODE lzo
    ,ecommerce_user_id VARCHAR(128)   ENCODE lzo
    ,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
    ,derived_tstamp_date DATE   ENCODE az64
    ,cart_id VARCHAR(4096)   ENCODE lzo
    ,cart_currency CHAR(3)   ENCODE lzo
    ,cart_total_value NUMERIC(9,2)   ENCODE az64
    ,cart_created BOOLEAN   ENCODE RAW
    ,cart_emptied BOOLEAN   ENCODE RAW
    ,cart_transacted BOOLEAN   ENCODE RAW
    ,ecommerce_action_type VARCHAR(16)   ENCODE lzo
)

insert into (your_schema)_derived.snowplow_ecommerce_cart_interactions_new  select * from (your_schema)_derived.snowplow_ecommerce_cart_interactions;
drop table (your_schema)_derived.snowplow_ecommerce_cart_interactions;
alter table (your_schema)_derived.snowplow_ecommerce_cart_interactions_new
rename to snowplow_ecommerce_cart_interactions;

create table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new
(
    event_id CHAR(36)   ENCODE lzo
    ,page_view_id VARCHAR(4096)   ENCODE lzo
    ,domain_sessionid CHAR(128)   ENCODE lzo
    ,event_in_session_index BIGINT   ENCODE az64
    ,domain_userid VARCHAR(128)   ENCODE lzo
    ,network_userid VARCHAR(128)   ENCODE lzo
    ,user_id VARCHAR(255)   ENCODE lzo
    ,ecommerce_user_id VARCHAR(128)   ENCODE lzo
    ,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
    ,derived_tstamp_date DATE   ENCODE az64
    ,transaction_id VARCHAR(4096)   ENCODE lzo
    ,transaction_currency CHAR(3)   ENCODE lzo
    ,transaction_payment_method VARCHAR(128)   ENCODE lzo
    ,transaction_revenue NUMERIC(9,2)   ENCODE az64
    ,transaction_total_quantity INTEGER   ENCODE az64
    ,transaction_credit_order BOOLEAN   ENCODE RAW
    ,transaction_discount_amount NUMERIC(9,2)   ENCODE az64
    ,transaction_discount_code VARCHAR(99)   ENCODE lzo
    ,transaction_shipping NUMERIC(9,2)   ENCODE az64
    ,transaction_tax NUMERIC(9,2)   ENCODE az64
    ,ecommerce_user_email VARCHAR(256)   ENCODE lzo
    ,ecommerce_user_is_guest BOOLEAN   ENCODE RAW
    ,number_products BIGINT   ENCODE az64
)

insert into (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new    select * from (your_schema)_derived.snowplow_ecommerce_transaction_interactions;
drop table (your_schema)_derived.snowplow_ecommerce_transaction_interactions;
alter table (your_schema)_derived.snowplow_ecommerce_transaction_interactions_new rename to snowplow_ecommerce_transaction_interactions;
```

***

### Upgrading to 0.4.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

### Upgrading to 0.3.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Fractribution
> Migration guide for upgrading the legacy Snowplow Fractribution dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/fractribution/

This is a legacy package.

### Upgrading to 0.3.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)
- Other changes required by [snowplow-web version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/#upgrading-to-0140), in particular the deduplication logic for Redshift and Postgres users

### Upgrading to 0.2.0

- Variable names are now prefixed with `snowplow__`, please align the new ones found in the `dbt_project.yml` file
- `snowplow__path_transforms` variable is a dictionary instead of an array and that the path transform names have also changed (e.g: `Exposure` -> `exposure_path`). See docs for the latest values
- Version 1.3.0 of `dbt-core` now required
- The python scripts have changed and been renamed per warehouse, please ensure you run the correct version and/or pull the latest docker image

---

# Migration guides for dbt data models
> Migration guides for upgrading from previous versions of Snowplow dbt packages including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/

With each package release, new features and fixes become available, as well as changes to existing functionalities. In many cases, including some breaking changes, nothing needs to be done by the user of the packages to use these new features beyond simply installing the new version, although the logic within the package may have changed. All such changes are always listed in the package changelog and release posts on [Community](https://community.snowplow.io/).

In some cases certain core dependencies (such as the version of dbt required) or names of variables may change that will require the user to make changes to their models and/or configuration of the package. These are listed in the release post and changelog, and repeated in the following pages.

---

# Migration guide for Media Player
> Migration guide for upgrading the Snowplow Media Player dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/media-player/

### Upgrading to 0.8.0

**We recommend a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** There might be inconsistencies in the calculation of impressions and play rate in the media stats table without doing a full refresh.

**Breaking changes:**

- The calculation of impressions in the stats table changed to use distinct plays instead of page views. This allows for multiple videos on the same page to be counted as separate impressions.

### Upgrading to 0.7.0

**This version requires a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

**Breaking changes:**

- A new and more robust `media_identifier` field to replace using `media_id` as a key in the derived tables.
- The introduction of new base macro functionality means in places the session and user identifier fields have been renamed to `session_identifier` and `user_identifier`.
- The default session identifier has been updated from using the `domain_sessionid`, to now be the media session id (or the page/screen view id if the media session entity is not set).
- Adds a primary key, `media_ad_view_id`, to the ad views table.

### Upgrading to 0.6.0

**This version requires a [full refresh run](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) if you have been using any previous versions.** You will not be able to upgrade and have the package work without doing a full refresh.

Please [check the quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/media-player/) for a guide on which new configuration options to enable. If you remain tracking events using the HTML5 and YouTube tracking plugins for the JavaScript tracker, you should add the following configuration to your `dbt_project.yml`:

```yml
vars:
  snowplow_media_player:
    # enable the version 1 player property
    snowplow__enable_media_player_v1: true
    # disable the new version 2 schemas
    snowplow__enable_media_player_v2: false
    snowplow__enable_media_session: false
```

On Redshift, the context names no longer require the use of the `{{ source('...') }}` macro, the table names can be passed directly.

### Upgrading to 0.5.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)
- Other changes required by [snowplow-web version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/#upgrading-to-0140), in particular the deduplication logic for Redshift and Postgres users

### Upgrading to 0.4.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Mobile
> Migration guide for upgrading the legacy Snowplow Mobile dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/mobile/

This is a legacy package.

### Upgrading to 0.7.0

- Version 1.4.0 of `dbt-core` now required

- You must add the following to the top level of your project yaml

  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```

- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

- **Redshift/Postgres users**: The deduplication logic has changed as a result of which any downstream joins on context and custom event tables will have to be deduped. Previously duplicate events were removed right in the base module to avoid any complications but this meant that the data could have been incomplete unlike in case of other data warehouses. These events will now arrive complete and deduped but they will likely have duplicate counterparts in their respective context tables as well, hence the warning.

  In case you have any custom models, please check them and make sure that deduplication logic is applied on them when they are joined to the main events table. We have provided a macro - `get_sde_or_context()` - for you to use for this purpose in the latest v0.14.0 snowplow-utils package. Check out the [package documentation](https://snowplow.github.io/dbt-snowplow-utils/#!/overview/snowplow_utils) on how to use it.

### Upgrading to 0.6.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Normalize
> Migration guide for upgrading the Snowplow Normalize dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/normalize/

### Upgrading to 0.3.0

- Version 1.4.0 of `dbt-core` now required
- You must add the following to the top level of your project yaml
  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```
- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

### Upgrading to 0.2.0

- Version 1.3.0 of `dbt-core` now required

- Upgrading your config file

  - Change the `event_name` field to `event_names` and make the value a list
  - Change the `self_describing_event_schema` field to `self_describing_event_schemas` and make the value a list
  - If you wish to make use of the new features, see the example config or the docs for more information

- Upgrading your models (preferred method is to re-run the python script, but can be done manually following these steps)

  - For each normalized model:

    - Convert the `event_name` and `sde_cols` fields to lists, and pluralize the names in both the set and the macro call
    - Add a new field, `sde_aliases` which is an empty list, add this between `sde_types` and `context_cols` in the macro call

  - For your filtered events table:

    - Change the `unique_key` in the config section to `unique_id`
    - Add a line between the `event_table_name` and from lines for each select statement; `, event_id||'-'||'<that_event_table_name>' as unique_id`, with the event table name for that select block.

  - For your users table:
    - Add 3 new values to the start of the macro call, `'user_id','',''`, before the `user_cols` argument.

- Upgrade your filtered events table
  - If you use the master filtered events table, you will need to add a new column for the latest version to work. If you have not processed much data yet it may be easier to simply re-run the package from scratch using dbt run --full-refresh --vars 'snowplow\_\_allow\_refresh: true', alternatively run the following in your warehouse, replacing the schema/dataset/warehouse and table name for your table:
    ```sql
    ALTER TABLE {schema}.{table} ADD COLUMN unique_id STRING;
    UPDATE {schema}.{table} SET unique_id = event_id||'-'||event_table_name WHERE 1 = 1;
    ```

---

# Migration guide for Unified Digital
> Migration guide for upgrading the Snowplow Unified Digital dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- for a full upgrade walkthrough, please follow [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.4.0

If you make use of the `conversion` module, and don't wish to do a full refresh, you will need to add the new column and update the existing records with the correct values for the model to run successfully incrementally.

**Redshift:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id TEXT;

UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as TEXT), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as TEXT), '_dbt_utils_surrogate_key_null_') as TEXT))
where 1=1;
```

**Snowflake:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id TEXT;

UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as TEXT), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as TEXT), '_dbt_utils_surrogate_key_null_') as TEXT))
where 1=1;
```

**BigQuery:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id string;

UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = to_hex(md5(cast(coalesce(cast(event_id as string), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as string), '_dbt_utils_surrogate_key_null_') as string)))
where 1=1;
```

**Databricks:**

```sql
ALTER TABLE your_schema_derived.snowplow_unified_conversions add column cv_id string;

UPDATE your_schema_derived.snowplow_unified_conversions set cv_id = md5(cast(coalesce(cast(event_id as string), '_dbt_utils_surrogate_key_null_') || '-' || coalesce(cast(cv_type as string), '_dbt_utils_surrogate_key_null_') as string))
where 1=1;
```

***

### Upgrading to 0.3.0

- `snowplow__page_view_passthroughs` has been renamed to `snowplow__view_passthroughs`. Please rename this in your project yaml if you have set any.
- If you currently use `snowplow__allow_refresh` without also adding a `--full-refresh` flag in any script this is now required to ensure the manifest tables are fully refreshed.

### Upgrading to 0.2.0

The views table needs to be altered for existing Snowflake, Databricks or Redshift users as the data type of `engaged_time_in_s`, `absolute_time_in_s` have changed. Please modify the below script to fit your schemas and apply them before running the upgraded package.

The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

#### Updates to the views table

- Changed type: `engaged_time_in_s`, `absolute_time_in_s`

**SQL scripts**

**snowflake:**

```sql
create or replace TRANSIENT TABLE (your_schema)_derived.snowplow_unified_views_new cluster by (to_date(start_tstamp))(
	VIEW_ID VARCHAR(16777216),
	EVENT_NAME VARCHAR(16777216),
	EVENT_ID VARCHAR(16777216),
	SESSION_IDENTIFIER VARCHAR(16777216),
	VIEW_IN_SESSION_INDEX NUMBER(18,0),
	VIEWS_IN_SESSION NUMBER(18,0),
	SESSION__PREVIOUS_SESSION_ID VARCHAR(36),
	USER_ID VARCHAR(16777216),
	USER_IDENTIFIER VARCHAR(16777216),
	STITCHED_USER_ID VARCHAR(16777216),
	NETWORK_USERID VARCHAR(16777216),
	DVCE_CREATED_TSTAMP TIMESTAMP_NTZ(9),
	COLLECTOR_TSTAMP TIMESTAMP_NTZ(9),
	DERIVED_TSTAMP TIMESTAMP_NTZ(9),
	START_TSTAMP TIMESTAMP_NTZ(9),
	END_TSTAMP TIMESTAMP_NTZ(9),
	MODEL_TSTAMP TIMESTAMP_NTZ(9),
	APP_ID VARCHAR(16777216),
	PLATFORM VARCHAR(16777216),
	DEVICE_IDENTIFIER VARCHAR(16777216),
	DEVICE_CATEGORY VARCHAR(16777216),
	DEVICE_SESSION_INDEX NUMBER(38,0),
	OS_VERSION VARCHAR(16777216),
	OS_TYPE VARCHAR(16777216),
	MOBILE__DEVICE_MANUFACTURER VARCHAR(16777216),
	MOBILE__DEVICE_MODEL VARCHAR(16777216),
	MOBILE__OS_TYPE VARCHAR(16777216),
	MOBILE__OS_VERSION VARCHAR(16777216),
	MOBILE__ANDROID_IDFA VARCHAR(16777216),
	MOBILE__APPLE_IDFA VARCHAR(16777216),
	MOBILE__APPLE_IDFV VARCHAR(16777216),
	MOBILE__CARRIER VARCHAR(16777216),
	MOBILE__OPEN_IDFA VARCHAR(16777216),
	MOBILE__NETWORK_TECHNOLOGY VARCHAR(16777216),
	MOBILE__NETWORK_TYPE VARCHAR(255),
	MOBILE__PHYSICAL_MEMORY NUMBER(38,0),
	MOBILE__SYSTEM_AVAILABLE_MEMORY NUMBER(38,0),
	MOBILE__APP_AVAILABLE_MEMORY NUMBER(38,0),
	MOBILE__BATTERY_LEVEL NUMBER(38,0),
	MOBILE__BATTERY_STATE VARCHAR(16777216),
	MOBILE__LOW_POWER_MODE BOOLEAN,
	MOBILE__AVAILABLE_STORAGE NUMBER(38,0),
	MOBILE__TOTAL_STORAGE NUMBER(38,0),
	MOBILE__IS_PORTRAIT BOOLEAN,
	MOBILE__RESOLUTION VARCHAR(16777216),
	MOBILE__SCALE FLOAT,
	MOBILE__LANGUAGE VARCHAR(16777216),
	MOBILE__APP_SET_ID VARCHAR(16777216),
	MOBILE__APP_SET_ID_SCOPE VARCHAR(16777216),
	OS_TIMEZONE VARCHAR(16777216),
	SCREEN_RESOLUTION VARCHAR(16777216),
	YAUAA__DEVICE_CLASS VARCHAR(16777216),
	YAUAA__DEVICE_VERSION VARCHAR(16777216),
	YAUAA__OPERATING_SYSTEM_VERSION VARCHAR(16777216),
	YAUAA__OPERATING_SYSTEM_CLASS VARCHAR(16777216),
	YAUAA__OPERATING_SYSTEM_NAME VARCHAR(16777216),
	YAUAA__OPERATING_SYSTEM_NAME_VERSION VARCHAR(16777216),
	GEO_COUNTRY VARCHAR(16777216),
	GEO_REGION VARCHAR(16777216),
	GEO_REGION_NAME VARCHAR(16777216),
	GEO_CITY VARCHAR(16777216),
	GEO_ZIPCODE VARCHAR(16777216),
	GEO_LATITUDE FLOAT,
	GEO_LONGITUDE FLOAT,
	GEO_TIMEZONE VARCHAR(16777216),
	USER_IPADDRESS VARCHAR(16777216),
	ENGAGED_TIME_IN_S DOUBLE,
	ABSOLUTE_TIME_IN_S DOUBLE,
	HORIZONTAL_PIXELS_SCROLLED NUMBER(38,0),
	VERTICAL_PIXELS_SCROLLED NUMBER(38,0),
	HORIZONTAL_PERCENTAGE_SCROLLED FLOAT,
	VERTICAL_PERCENTAGE_SCROLLED FLOAT,
	MKT_MEDIUM VARCHAR(16777216),
	MKT_SOURCE VARCHAR(16777216),
	MKT_TERM VARCHAR(16777216),
	MKT_CONTENT VARCHAR(16777216),
	MKT_CAMPAIGN VARCHAR(16777216),
	MKT_CLICKID VARCHAR(16777216),
	MKT_NETWORK VARCHAR(16777216),
	DEFAULT_CHANNEL_GROUP VARCHAR(25),
	PAGE_URL VARCHAR(16777216),
	PAGE_REFERRER VARCHAR(16777216),
	REFR_MEDIUM VARCHAR(16777216),
	REFR_SOURCE VARCHAR(16777216),
	REFR_TERM VARCHAR(16777216),
	PAGE_TITLE VARCHAR(16777216),
	CONTENT_GROUP VARCHAR(39),
	PAGE_URLSCHEME VARCHAR(16777216),
	PAGE_URLHOST VARCHAR(16777216),
	PAGE_URLPATH VARCHAR(16777216),
	PAGE_URLQUERY VARCHAR(16777216),
	PAGE_URLFRAGMENT VARCHAR(16777216),
	REFR_URLSCHEME VARCHAR(16777216),
	REFR_URLHOST VARCHAR(16777216),
	REFR_URLPATH VARCHAR(16777216),
	REFR_URLQUERY VARCHAR(16777216),
	REFR_URLFRAGMENT VARCHAR(16777216),
	BR_LANG VARCHAR(16777216),
	BR_VIEWWIDTH NUMBER(38,0),
	BR_VIEWHEIGHT NUMBER(38,0),
	BR_COLORDEPTH VARCHAR(16777216),
	BR_RENDERENGINE VARCHAR(16777216),
	DOC_WIDTH NUMBER(38,0),
	DOC_HEIGHT NUMBER(38,0),
	IAB__CATEGORY VARCHAR(16777216),
	IAB__PRIMARY_IMPACT VARCHAR(16777216),
	IAB__REASON VARCHAR(16777216),
	IAB__SPIDER_OR_ROBOT BOOLEAN,
	YAUAA__DEVICE_NAME VARCHAR(16777216),
	YAUAA__AGENT_CLASS VARCHAR(16777216),
	YAUAA__AGENT_NAME VARCHAR(16777216),
	YAUAA__AGENT_NAME_VERSION VARCHAR(16777216),
	YAUAA__AGENT_NAME_VERSION_MAJOR VARCHAR(16777216),
	YAUAA__AGENT_VERSION VARCHAR(16777216),
	YAUAA__AGENT_VERSION_MAJOR VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_CLASS VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_NAME VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_NAME_VERSION VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_NAME_VERSION_MAJOR VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_VERSION VARCHAR(16777216),
	YAUAA__LAYOUT_ENGINE_VERSION_MAJOR VARCHAR(16777216),
	UA__DEVICE_FAMILY VARCHAR(16777216),
	UA__OS_VERSION VARCHAR(16777216),
	UA__OS_MAJOR VARCHAR(16777216),
	UA__OS_MINOR VARCHAR(16777216),
	UA__OS_PATCH VARCHAR(16777216),
	UA__OS_PATCH_MINOR VARCHAR(16777216),
	UA__USERAGENT_FAMILY VARCHAR(16777216),
	UA__USERAGENT_MAJOR VARCHAR(16777216),
	UA__USERAGENT_MINOR VARCHAR(16777216),
	UA__USERAGENT_PATCH VARCHAR(16777216),
	UA__USERAGENT_VERSION VARCHAR(16777216),
	SCREEN_VIEW__NAME VARCHAR(16777216),
	SCREEN_VIEW__PREVIOUS_ID VARCHAR(36),
	SCREEN_VIEW__PREVIOUS_NAME VARCHAR(16777216),
	SCREEN_VIEW__PREVIOUS_TYPE VARCHAR(16777216),
	SCREEN_VIEW__TRANSITION_TYPE VARCHAR(16777216),
	SCREEN_VIEW__TYPE VARCHAR(16777216),
	APP__BUILD VARCHAR(255),
	APP__VERSION VARCHAR(255),
	GEO__ALTITUDE FLOAT,
	GEO__ALTITUDE_ACCURACY FLOAT,
	GEO__BEARING FLOAT,
	GEO__LATITUDE FLOAT,
	GEO__LATITUDE_LONGITUDE_ACCURACY FLOAT,
	GEO__LONGITUDE FLOAT,
	GEO__SPEED FLOAT,
	SCREEN__FRAGMENT VARCHAR(16777216),
	SCREEN__TOP_VIEW_CONTROLLER VARCHAR(16777216),
	SCREEN__VIEW_CONTROLLER VARCHAR(16777216),
	USERAGENT VARCHAR(16777216),
	V_COLLECTOR VARCHAR(16777216),
	EVENT_ID2 VARCHAR(16777216),
	LAST_LIST_ITEM_INDEX NUMBER(38,0),
	LIST_ITEMS_COUNT NUMBER(38,0),
	LIST_ITEMS_PERCENTAGE_SCROLLED FLOAT
);

insert into (your_schema)_derived.snowplow_unified_views_new  select *, NULL AS LAST_LIST_ITEM_INDEX, NULL AS LIST_ITEMS_COUNT, NULL AS LIST_ITEMS_PERCENTAGE_SCROLLED from (your_schema)_derived.snowplow_unified_views;
drop table (your_schema)_derived.snowplow_unified_views;
create table (your_schema)_derived.snowplow_unified_views as select * from (your_schema)_derived.snowplow_unified_views_new;
drop table (your_schema)_derived.snowplow_unified_views_new;
```

**databricks:**

#### Updates to the views table

```sql
CREATE TABLE (your_schema)_derived.snowplow_unified_views_new (
  view_id STRING,
  event_name STRING,
  event_id STRING,
  session_identifier STRING,
  view_in_session_index INT,
  views_in_session INT,
  session__previous_session_id STRING,
  user_id STRING,
  user_identifier STRING,
  stitched_user_id STRING,
  network_userid STRING,
  dvce_created_tstamp TIMESTAMP,
  collector_tstamp TIMESTAMP,
  derived_tstamp TIMESTAMP,
  start_tstamp TIMESTAMP,
  end_tstamp TIMESTAMP,
  model_tstamp TIMESTAMP,
  app_id STRING,
  platform STRING,
  device_identifier STRING,
  device_category STRING,
  device_session_index INT,
  os_version STRING,
  os_type STRING,
  mobile__device_manufacturer STRING,
  mobile__device_model STRING,
  mobile__os_type STRING,
  mobile__os_version STRING,
  mobile__android_idfa STRING,
  mobile__apple_idfa STRING,
  mobile__apple_idfv STRING,
  mobile__carrier STRING,
  mobile__open_idfa STRING,
  mobile__network_technology STRING,
  mobile__network_type STRING,
  mobile__physical_memory INT,
  mobile__system_available_memory INT,
  mobile__app_available_memory INT,
  mobile__battery_level INT,
  mobile__battery_state STRING,
  mobile__low_power_mode BOOLEAN,
  mobile__available_storage BIGINT,
  mobile__total_storage BIGINT,
  mobile__is_portrait BOOLEAN,
  mobile__resolution STRING,
  mobile__scale FLOAT,
  mobile__language STRING,
  mobile__app_set_id STRING,
  mobile__app_set_id_scope STRING,
  screen_resolution STRING,
  geo_country STRING,
  geo_region BIGINT,
  geo_region_name BIGINT,
  geo_city BIGINT,
  geo_zipcode BIGINT,
  geo_latitude BIGINT,
  geo_longitude BIGINT,
  geo_timezone BIGINT,
  user_ipaddress STRING,
  engaged_time_in_s FLOAT,
  absolute_time_in_s FLOAT,
  horizontal_pixels_scrolled BIGINT,
  vertical_pixels_scrolled BIGINT,
  horizontal_percentage_scrolled FLOAT,
  vertical_percentage_scrolled FLOAT,
  last_list_item_index INT,
  list_items_count INT,
  list_items_percentage_scrolled FLOAT,
  mkt_medium STRING,
  mkt_source STRING,
  mkt_term STRING,
  mkt_content STRING,
  mkt_campaign STRING,
  mkt_clickid BIGINT,
  mkt_network BIGINT,
  default_channel_group STRING,
  page_url STRING,
  page_referrer STRING,
  refr_medium STRING,
  refr_source STRING,
  refr_term STRING,
  screen_view__name STRING,
  screen_view__previous_id STRING,
  screen_view__previous_name STRING,
  screen_view__previous_type STRING,
  screen_view__transition_type STRING,
  screen_view__type STRING,
  app__build STRING,
  app__version STRING,
  screen__fragment STRING,
  screen__top_view_controller STRING,
  screen__view_controller STRING,
  useragent STRING,
  v_collector STRING,
  event_id2 STRING,
  start_tstamp_date DATE)
USING delta
PARTITIONED BY (start_tstamp_date)
TBLPROPERTIES (
  'delta.autoOptimize.autoCompact' = 'true',
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.checkpoint.writeStatsAsJson' = 'false',
  'delta.checkpoint.writeStatsAsStruct' = 'true',
  'delta.minReaderVersion' = '1',
  'delta.minWriterVersion' = '2');

insert into (your_schema)_derived.snowplow_unified_views_new  select *, null as last_list_item_index, null as list_items_count, null as list_items_percentage_scrolled from (your_schema)_derived.snowplow_unified_views;
drop table (your_schema)_derived.snowplow_unified_views;
create table (your_schema)_derived.snowplow_unified_views as select * from (your_schema)_derived.snowplow_unified_views_new;
drop table (your_schema)_derived.snowplow_unified_views_new;
```

**redshift:**

#### Updates to the views table

```sql
CREATE TABLE IF NOT EXISTS (your_schema)_derived.snowplow_unified_views_new
(
	view_id VARCHAR(65535)   ENCODE lzo
	,event_name VARCHAR(22)   ENCODE lzo
	,event_id VARCHAR(36)   ENCODE lzo
	,session_identifier VARCHAR(36)   ENCODE lzo
	,view_in_session_index BIGINT   ENCODE az64
	,views_in_session BIGINT   ENCODE az64
	,session__previous_session_id VARCHAR(36)   ENCODE lzo
	,user_id VARCHAR(256)   ENCODE lzo
	,user_identifier VARCHAR(36)   ENCODE lzo
	,stitched_user_id VARCHAR(65535)   ENCODE lzo
	,network_userid VARCHAR(36)   ENCODE lzo
	,dvce_created_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
	,collector_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
	,derived_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
	,start_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE RAW
	,end_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
	,model_tstamp TIMESTAMP WITHOUT TIME ZONE   ENCODE az64
	,app_id VARCHAR(256)   ENCODE lzo
	,platform VARCHAR(3)   ENCODE lzo
	,device_identifier VARCHAR(36)   ENCODE lzo
	,device_category VARCHAR(22)   ENCODE lzo
	,device_session_index INTEGER   ENCODE az64
	,os_version VARCHAR(3)   ENCODE lzo
	,os_type VARCHAR(25)   ENCODE lzo
	,mobile__device_manufacturer VARCHAR(7)   ENCODE lzo
	,mobile__device_model VARCHAR(37)   ENCODE lzo
	,mobile__os_type VARCHAR(25)   ENCODE lzo
	,mobile__os_version VARCHAR(3)   ENCODE lzo
	,mobile__android_idfa VARCHAR(36)   ENCODE lzo
	,mobile__apple_idfa VARCHAR(2)   ENCODE lzo
	,mobile__apple_idfv VARCHAR(2)   ENCODE lzo
	,mobile__carrier VARCHAR(7)   ENCODE lzo
	,mobile__open_idfa VARCHAR(2)   ENCODE lzo
	,mobile__network_technology VARCHAR(2)   ENCODE lzo
	,mobile__network_type VARCHAR(4)   ENCODE lzo
	,mobile__physical_memory INTEGER   ENCODE az64
	,mobile__system_available_memory INTEGER   ENCODE az64
	,mobile__app_available_memory INTEGER   ENCODE az64
	,mobile__battery_level INTEGER   ENCODE az64
	,mobile__battery_state VARCHAR(8)   ENCODE lzo
	,mobile__low_power_mode BOOLEAN   ENCODE RAW
	,mobile__available_storage INTEGER   ENCODE az64
	,mobile__total_storage BIGINT   ENCODE az64
	,mobile__is_portrait BOOLEAN   ENCODE RAW
	,mobile__resolution VARCHAR(9)   ENCODE lzo
	,mobile__scale DOUBLE PRECISION   ENCODE RAW
	,mobile__language VARCHAR(2)   ENCODE lzo
	,mobile__app_set_id VARCHAR(2)   ENCODE lzo
	,mobile__app_set_id_scope VARCHAR(2)   ENCODE lzo
	,screen_resolution VARCHAR(9)   ENCODE lzo
	,geo_country VARCHAR(256)   ENCODE lzo
	,geo_region INTEGER   ENCODE az64
	,geo_region_name INTEGER   ENCODE az64
	,geo_city INTEGER   ENCODE az64
	,geo_zipcode INTEGER   ENCODE az64
	,geo_latitude INTEGER   ENCODE az64
	,geo_longitude INTEGER   ENCODE az64
	,geo_timezone INTEGER   ENCODE az64
	,user_ipaddress VARCHAR(10)   ENCODE lzo
	,engaged_time_in_s DOUBLE PRECISION   ENCODE RAW
	,absolute_time_in_s DOUBLE PRECISION   ENCODE RAW
	,horizontal_pixels_scrolled INTEGER   ENCODE az64
	,vertical_pixels_scrolled INTEGER   ENCODE az64
	,horizontal_percentage_scrolled DOUBLE PRECISION   ENCODE RAW
	,vertical_percentage_scrolled DOUBLE PRECISION   ENCODE RAW
	,last_list_item_index INTEGER   ENCODE az64
	,list_items_count INTEGER   ENCODE az64
	,list_items_percentage_scrolled DOUBLE PRECISION   ENCODE RAW
	,mkt_medium VARCHAR(256)   ENCODE lzo
	,mkt_source VARCHAR(256)   ENCODE lzo
	,mkt_term VARCHAR(256)   ENCODE lzo
	,mkt_content VARCHAR(256)   ENCODE lzo
	,mkt_campaign VARCHAR(256)   ENCODE lzo
	,mkt_clickid INTEGER   ENCODE az64
	,mkt_network INTEGER   ENCODE az64
	,default_channel_group VARCHAR(25)   ENCODE lzo
	,page_url VARCHAR(256)   ENCODE lzo
	,page_referrer VARCHAR(256)   ENCODE lzo
	,refr_medium VARCHAR(256)   ENCODE lzo
	,refr_source VARCHAR(256)   ENCODE lzo
	,refr_term VARCHAR(256)   ENCODE lzo
	,screen_view__name VARCHAR(2)   ENCODE lzo
	,screen_view__previous_id VARCHAR(2)   ENCODE lzo
	,screen_view__previous_name VARCHAR(2)   ENCODE lzo
	,screen_view__previous_type VARCHAR(2)   ENCODE lzo
	,screen_view__transition_type VARCHAR(2)   ENCODE lzo
	,screen_view__type VARCHAR(2)   ENCODE lzo
	,app__build VARCHAR(2)   ENCODE lzo
	,app__version VARCHAR(2)   ENCODE lzo
	,screen__fragment VARCHAR(2)   ENCODE lzo
	,screen__top_view_controller VARCHAR(2)   ENCODE lzo
	,screen__view_controller VARCHAR(2)   ENCODE lzo
	,useragent VARCHAR(51)   ENCODE lzo
	,v_collector VARCHAR(28)   ENCODE lzo
	,event_id2 VARCHAR(36)   ENCODE lzo
)
DISTSTYLE KEY
 DISTKEY (view_id)
 SORTKEY (
	start_tstamp
	)
;

insert into (your_schema)_derived.snowplow_unified_views_new  select *, null as last_list_item_index, null as list_items_count, null as list_items_percentage_scrolled from (your_schema)_derived.snowplow_unified_views;
drop table (your_schema)_derived.snowplow_unified_views;
alter table (your_schema)_derived.snowplow_unified_views_new
rename to snowplow_unified_views;
```

***

---

# Migration guide for Utils
> Migration guide for upgrading the Snowplow Utils dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/

### Upgrading to 1.0.0

- Version 1.10.6 of `dbt-core` now required
- for a full upgrade walkthrough, please follow [official dbt guide](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10)
- Generic test arguments must be nested under arguments: (see dbt’s guidance on the require\_generic\_test\_arguments\_property behavior change [here](https://docs.getdbt.com/reference/global-configs/behavior-changes#generic-test-arguments-property))
- Adapter dbt-redshift 1.10.0+ is required for Redshift users
- Users unable to upgrade immediately may remain on v0.x, which receives critical bug fixes only

### Upgrading to 0.14.0

- Version 1.4.0 of `dbt-core` now required
- Any custom models using the `snowplow_incremental` materialization should be changed to use the `incremental` materialization with `snowplow_optimize=true` set in the model config. `snowplow_incremental` has been deprecated
- BigQuery incremental models now require the `upsert_date_key` config
- `snowplow_utils.type_string` it has been deprecated and should be replaced with `dbt.type_string()` for non-length specific calls, and `snowplow_utils.type_max_string` for redshift strings above 256 characters.
- `snowplow_utils.get_cluster_by` and `snowplow_utils.get_partition_by` they have been deprecated and should be replaced with `snowplow_utils.get_value_by_target_type` with arg names `bigquery_val`, `databricks_val` etc.
- `snowplow_utils.snowplow_is_incremental` has been deprecated and should be replaced with standard `is_incremental()` to align with removal of the custom materialization
- You must add the following to the top level of your project yaml to take advantage of our optimized incremental materialization
  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```

### Upgrading to 0.13.0

- Version 1.3.0 of `dbt-core` now required

---

# Migration guide for Web
> Migration guide for upgrading the legacy Snowplow Web dbt package including breaking changes and configuration updates.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/

This is a legacy package.

### Upgrading to 0.16.0

Note the minimum `dbt-core` version now required is 1.5.0

Most upgrades and new features should be available automatically after the first run of the new version of the package, however there are a few breaking changes to manage before this first run. To enable any other new optional features or make use of the new configuration options, please see the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/legacy/web/) page.

1. The seed files have been prefixed with `snowplow_web_` so you will need to re-run `dbt seed` if this is not part of your standard workflow (note this will not drop the existing tables)
2. Both the quarantined sessions and the sessions lifecycle table have has column(s) renamed, please run the below statements to rename this column in your warehouse if not doing a full-refresh of the package.
   ```sql
   alter table <your_schema>_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest rename column session_id to session_identifier;
   alter table <your_schema>_snowplow_manifest.snowplow_web_base_quarantined_sessions rename column session_id to session_identifier;
   alter table <your_schema>_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest rename column domain_userid to user_identifier;
   ```
3. For Redshift/Postgres users, the context variables are now just the table name, no need for the complicated `source` function if you had overwritten these previously.

### Upgrading to 0.15.1

For existing Snowflake and Databricks Core Web Vitals optional module users the derived table `snowplow_web_vitals` needs to be altered due to column data type change. Please run the below queries to migrate. The other option is to do a [complete refresh](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.

**SQL scripts**

**snowflake:**

```sql
create table (your_schema)_derived.snowplow_web_vitals_new
(

    event_id               STRING,
    event_name             STRING,
    app_id                 STRING,
    platform               STRING,
    domain_userid          STRING,
    user_id                STRING,
    page_view_id           STRING,
    domain_sessionid       STRING,
    collector_tstamp       TIMESTAMP,
    derived_tstamp         TIMESTAMP,
    load_tstamp            TIMESTAMP,
    geo_country            STRING,
    page_url               STRING,
    url_group              STRING,
    page_title             STRING,
    useragent              STRING,
    device_class           STRING,
    device_name            STRING,
    agent_name             STRING,
    agent_version          STRING,
    operating_system_name  STRING,
    lcp                    DECIMAL(14, 4),
    fcp                    DECIMAL(14, 4),
    fid                    DECIMAL(14, 4),
    cls                    DECIMAL(14, 4),
    inp                    DECIMAL(14, 4),
    ttfb                   DECIMAL(14, 4),
    navigation_type        STRING,
    dedupe_index           INT,
    lcp_result             STRING,
    fid_result             STRING,
    cls_result             STRING,
    ttfb_result            STRING,
    inp_result             STRING
);

insert into (your_schema)_derived.snowplow_web_vitals_new  select * from (your_schema)_derived.snowplow_web_vitals;
drop table (your_schema)_derived.snowplow_web_vitals;
alter table (your_schema)_derived.snowplow_web_vitals_new rename to (your_schema)_derived.snowplow_web_vitals;
```

**databricks:**

```sql
create table (your_schema)_derived.snowplow_web_vitals_new
(
    event_id               STRING,
    event_name             STRING,
    app_id                 STRING,
    platform               STRING,
    domain_userid          STRING,
    user_id                STRING,
    page_view_id           STRING,
    domain_sessionid       STRING,
    collector_tstamp       TIMESTAMP,
    derived_tstamp         TIMESTAMP,
    load_tstamp            TIMESTAMP,
    geo_country            STRING,
    page_url               STRING,
    url_group              STRING,
    page_title             STRING,
    useragent              STRING,
    device_class           STRING,
    device_name            STRING,
    agent_name             STRING,
    agent_version          STRING,
    operating_system_name  STRING,
    lcp                    DECIMAL(14, 4),
    fcp                    DECIMAL(14, 4),
    fid                    DECIMAL(14, 4),
    cls                    DECIMAL(14, 4),
    inp                    DECIMAL(14, 4),
    ttfb                   DECIMAL(14, 4),
    navigation_type        STRING,
    dedupe_index           INT,
    lcp_result             STRING,
    fid_result             STRING,
    cls_result             STRING,
    ttfb_result            STRING,
    inp_result             STRING
);

insert into (your_schema)_derived.snowplow_web_vitals_new  select * from (your_schema)_derived.snowplow_web_vitals;
drop table (your_schema)_derived.snowplow_web_vitals;
create table (your_schema)_derived.snowplow_web_vitals select * from (your_schema)_derived.snowplow_web_vitals_new;
drop table (your_schema)_derived.snowplow_web_vitals_new;
```

***

### Upgrading to 0.15.0

Some of the new fields that are added will contain nulls for existing records. To avoid failing out of the box tests please update your profiles.yml to disable those tests:

```yml
# dbt_project.yml
...

tests:
  snowplow_web:
    sessions:
      not_null_snowplow_web_sessions_is_engaged:
        +enabled: false
      not_null_snowplow_web_sessions_total_events:
        +enabled: false
```

- Consent enablement is now done via the `snowplow__enable_consent` variable instead of via the enable block. If you are using consent please set this variable to `true`
- Some seed files are now required to run the package, please run `dbt seed --select snowplow_web --full-refresh` to load these. The schema they are loaded to can be changed in your `dbt_project.yml`
- New columns will be auto-added to users, sessions, and page views tables, please be aware for any downstream queries/BI tools.

### Upgrading to 0.14.0

- Version 1.4.0 of `dbt-core` now required

- You must add the following to the top level of your project yaml

  ```yml
  dispatch:
    - macro_namespace: dbt
      search_order: ['snowplow_utils', 'dbt']
  ```

- Other changes required by [snowplow-utils version 0.14.0](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/utils/#upgrading-to-0140)

- **Redshift/Postgres users**: The deduplication logic has changed as a result of which any downstream joins on context and custom event tables will have to be deduped. Previously duplicate events were removed right in the base module to avoid any complications but this meant that the data could have been incomplete unlike in case of other data warehouses. These events will now arrive complete and deduped but they will likely have duplicate counterparts in their respective context tables as well, hence the warning.

  In case you have any custom models, please check them and make sure that deduplication logic is applied on them when they are joined to the main events table. We have provided a macro - `get_sde_or_context()` - for you to use for this purpose in the latest v0.14.0 snowplow-utils package. Check out the [package documentation](https://snowplow.github.io/dbt-snowplow-utils/#!/overview/snowplow_utils) on how to use it.

### Upgrading to 0.13.0

- RDB Loader 4.0.0 or BigQuery Loader 1.0.0. If using postgres loader or older versions set `snowplow__enable_load_tstamp` to `false` in your project yaml and you will not be able to use the consent models.

### Upgrading to 0.12.0

- Version 1.3.0 of `dbt-core` now required

---

# Migrate from Web to Unified Digital
> Migration guide for transitioning from the legacy Snowplow Web dbt package to the Unified Digital package.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web_to_unified/

Follow this migration guide to upgrade from the legacy [Snowplow Web](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) dbt package to the [Snowplow Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) package. Unified Digital is the most supported and feature-rich Snowplow dbt package, which you can use to model web and/or mobile event data.

## Simplified upgrade overview

![Migration steps - part I](/assets/images/migrate_1-d38ec8820d5eb2d61f1ef4c7633f225a.png)

![Migration steps - part II](/assets/images/migrate_2-4382295f0ebf3e74071762d3f2ec5b77.png)

## Breaking changes between the packages

- The `page_views` table has been renamed to `views`
- The `domain_userid` field has been renamed to `user_identifier`, as you now have the ability to define your own custom logic / alternative user id field for the aggregation
- The `domain_sessionid` has been renamed to `session_identifier`, as you now have the ability to define your own custom logic / alternative session id field for the aggregation
- Apart from these, fields generated by the out-of-the-box entities and enrichments have been added a prefix to indicate where they are coming from, e.g. `device_version` has been renamed to `yauaa__device_version`
- These out-of-the-box entities and enrichments can be optionally enabled or disabled for the model (previously these values were there as NULLs) through variables e.g. `snowplow__enable_yauaa: true` which are then flattened automatically within the `base_events_this_run` table and can be referred to from there directly for custom models, if needed
- Some package variables have been renamed, as well as new ones added

## Things to bear in mind

Due to the breaking changes, ideally it's best to start from scratch and run the new package on your whole historic events dataset. This could be costly in both time and effort. In certain scenarios, e.g. in case of large data volumes, it may be useful to try and upgrade instead using the existing modeled derived dataset.

There is no 100% accurate upgrade solution, but with this guide you may be able to get 95% of the way there.

1. **Incomplete or inaccurate fields for historic data**: the unified package contains a list of new fields, some of which we can compute on a row-by-row basis, in which case we provided update scripts to compute them for your existing derived data. There are, however, a few other new fields that are a result of aggregate or other computation that we cannot calculate without a full refresh. E.g. `absolute_time_in_s` for users therefore they will stay `NULL`. Then when a new event comes to update that record it may either correct it or it will just be inaccurate.

2. **Fields that aren't available in the out-of-the-box derived tables**: there are a few fields that we've dropped as they were incorporated into a new field based on different grouping. There may also be custom fields you have added to the derived tables. The below passthrough variables come in handy in this case:

```yml
    snowplow__view_passthroughs: []
    snowplow__session_passthroughs: []
    snowplow__user_first_passthroughs: []
    snowplow__user_last_passthroughs: []
```

For more on this, have a look at the [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) section.

## Upgrade steps

### Step 1: Migrate to the latest version of the Web package

Once you decided you would like to go ahead with the upgrade process you will first need to make sure your version of the web model is version 1.0.0 or version 1.0.1. The [Web model migration guides](/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/web/) will help you decide what other changes you need to take into consideration. If the changes apply to your warehouse, you may need to execute the upgrade SQL scripts to bridge the gap.

### Step 2: Execute SQL scripts to create the new tables

The provided SQL scripts will first create the new tables based on your existing derived tables created by the web package. Secondly, they will make the changes i.e. renaming, adding, dropping, and updating columns wherever possible, including in the manifest tables. You will then have everything ready for a new run in the Unified Digital package as if nothing happened.

Update `(your_schema)` in the SQL scripts to match your derived schema name. Execute the scripts with your database IDE to create your new derived and manifest tables without having to reprocess your event data from scratch.

You might need to adjust the data `varchar` data type to `string` depending on your warehouse (most likely to be relevant for BigQuery or Databricks users) or limit it to the maximum in case there are limitations (potentially Redshift).

**SQL scripts**

```sql
create table (your_schema)_derived.snowplow_unified_views as (select * from (your_schema)_derived.snowplow_web_page_views);
create table (your_schema)_derived.snowplow_unified_sessions as (select * from (your_schema)_derived.snowplow_web_sessions);
create table (your_schema)_derived.snowplow_unified_users as (select * from (your_schema)_derived.snowplow_web_users);
create table (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest as (select * from (your_schema)_snowplow_manifest.snowplow_web_incremental_manifest);
create table (your_schema)_snowplow_manifest.snowplow_unified_base_sessions_lifecycle_manifest as (select * from (your_schema)_snowplow_manifest.snowplow_web_base_sessions_lifecycle_manifest)

alter table (your_schema)_derived.snowplow_unified_views rename column domain_sessionid to session_identifier;
alter table (your_schema)_derived.snowplow_unified_views rename column domain_userid to user_identifier;
alter table (your_schema)_derived.snowplow_unified_views rename column original_domain_userid to device_identifier;
alter table (your_schema)_derived.snowplow_unified_views drop column original_domain_sessionid;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_class to yauaa__agent_class;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_name to yauaa__agent_name;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_name_version to yauaa__agent_name_version;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_name_version_major to yauaa__agent_name_version_major;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_version to yauaa__agent_version;
alter table (your_schema)_derived.snowplow_unified_views rename column agent_version_major to yauaa__agent_version_major;
alter table (your_schema)_derived.snowplow_unified_views rename column category to iab__category;
alter table (your_schema)_derived.snowplow_unified_views drop column device_brand;
alter table (your_schema)_derived.snowplow_unified_views rename column device_class to yauaa__device_class;
alter table (your_schema)_derived.snowplow_unified_views rename column device_family to ua__device_family;
alter table (your_schema)_derived.snowplow_unified_views rename column device_name to yauaa__device_name;
alter table (your_schema)_derived.snowplow_unified_views rename column device_version to yauaa__device_version;
alter table (your_schema)_derived.snowplow_unified_views rename column domain_sessionidx to device_session_index;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_class to yauaa__layout_engine_class;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name to yauaa__layout_engine_name;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name_version to yauaa__layout_engine_name_version;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_name_version_major to yauaa__layout_engine_name_version_major;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_version to yauaa__layout_engine_version;
alter table  (your_schema)_derived.snowplow_unified_views rename column layout_engine_version_major to yauaa__layout_engine_version_major;
alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_class to yauaa__operating_system_class;
alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_name to yauaa__operating_system_name;
alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_name_version to yauaa__operating_system_name_version;
alter table  (your_schema)_derived.snowplow_unified_views rename column operating_system_version to yauaa__operating_system_version;
alter table  (your_schema)_derived.snowplow_unified_views drop column os_family;
alter table  (your_schema)_derived.snowplow_unified_views rename column os_major to ua__os_major;
alter table  (your_schema)_derived.snowplow_unified_views rename column os_minor to ua__os_minor;
alter table  (your_schema)_derived.snowplow_unified_views rename column os_patch to ua__os_patch;
alter table  (your_schema)_derived.snowplow_unified_views rename column os_patch_minor to ua__os_patch_minor;
alter table  (your_schema)_derived.snowplow_unified_views rename column os_version to ua__os_version;
alter table  (your_schema)_derived.snowplow_unified_views rename column page_views_in_session to views_in_session;
alter table  (your_schema)_derived.snowplow_unified_views rename column page_view_id to view_id;
alter table  (your_schema)_derived.snowplow_unified_views rename column page_view_in_session_index to view_in_session_index;
alter table  (your_schema)_derived.snowplow_unified_views rename column primary_impact to iab__primary_impact;
alter table  (your_schema)_derived.snowplow_unified_views rename column reason to iab__reason;
alter table  (your_schema)_derived.snowplow_unified_views rename column spider_or_robot to iab__spider_or_robot;
alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_family to ua__useragent_family;
alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_major to ua__useragent_major;
alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_minor to ua__useragent_minor;
alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_patch to ua__useragent_patch;
alter table  (your_schema)_derived.snowplow_unified_views rename column useragent_version to ua__useragent_version;
alter table  (your_schema)_derived.snowplow_unified_views add column if not exists default_channel_group varchar(25);
alter table  (your_schema)_derived.snowplow_unified_views add column event_name varchar(1000);
update (your_schema)_derived.snowplow_unified_views
set event_name = 'page_view'
where 1=1;
alter table  (your_schema)_derived.snowplow_unified_views add column os_type varchar(16777216);
alter table  (your_schema)_derived.snowplow_unified_views add column os_version varchar(16777216);
alter table  (your_schema)_derived.snowplow_unified_views add column session__previous_session_id varchar(36);
alter table  (your_schema)_derived.snowplow_unified_views add column if not exists stitched_user_id varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_sessionid to session_identifier;
alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_userid to user_identifier;
alter table (your_schema)_derived.snowplow_unified_sessions rename column original_domain_userid to device_identifier;
alter table (your_schema)_derived.snowplow_unified_sessions drop column original_domain_sessionid;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_class to yauaa__agent_class;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name to yauaa__agent_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name_version to yauaa__agent_name_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_name_version_major to yauaa__agent_name_version_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_version to yauaa__agent_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column agent_version_major to yauaa__agent_version_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column br_lang to first_br_lang;
alter table (your_schema)_derived.snowplow_unified_sessions rename column br_lang_name to first_br_lang_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column category to iab__category;
alter table (your_schema)_derived.snowplow_unified_sessions drop column device_brand;
alter table (your_schema)_derived.snowplow_unified_sessions rename column device_class to yauaa__device_class;
alter table (your_schema)_derived.snowplow_unified_sessions rename column device_family to ua__device_family;
alter table (your_schema)_derived.snowplow_unified_sessions rename column device_name to yauaa__device_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column device_version to yauaa__device_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column domain_sessionidx to device_session_index;
alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_city to first_geo_city;
alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_continent to first_geo_continent;
alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_country to first_geo_country;
alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_country_name to first_geo_country_name;
alter table (your_schema)_derived.snowplow_unified_sessions drop column geo_region;
alter table (your_schema)_derived.snowplow_unified_sessions rename column geo_region_name to first_geo_region_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_class to yauaa__layout_engine_class;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name to yauaa__layout_engine_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name_version to yauaa__layout_engine_name_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_name_version_major to yauaa__layout_engine_name_version_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_version to yauaa__layout_engine_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column layout_engine_version_major to yauaa__layout_engine_version_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_class to yauaa__operating_system_class;
alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_name to yauaa__operating_system_name;
alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_name_version to yauaa__operating_system_name_version;
alter table (your_schema)_derived.snowplow_unified_sessions rename column operating_system_version to yauaa__operating_system_version;
alter table (your_schema)_derived.snowplow_unified_sessions drop column os_family;
alter table (your_schema)_derived.snowplow_unified_sessions rename column os_major to ua__os_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column os_minor to ua__os_minor;
alter table (your_schema)_derived.snowplow_unified_sessions rename column os_patch to ua__os_patch;
alter table (your_schema)_derived.snowplow_unified_sessions rename column os_patch_minor to ua__os_patch_minor;
alter table (your_schema)_derived.snowplow_unified_sessions rename column page_views to views;
alter table (your_schema)_derived.snowplow_unified_sessions rename column primary_impact to iab__primary_impact;
alter table (your_schema)_derived.snowplow_unified_sessions rename column reason to iab__reason;
alter table (your_schema)_derived.snowplow_unified_sessions rename column referrer to page_referrer;
alter table (your_schema)_derived.snowplow_unified_sessions rename column spider_or_robot to iab__spider_or_robot;
alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_family to ua__useragent_family;
alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_major to ua__useragent_major;
alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_minor to ua__useragent_minor;
alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_patch to ua__useragent_patch;
alter table (your_schema)_derived.snowplow_unified_sessions rename column useragent_version to ua__useragent_version;
alter table (your_schema)_derived.snowplow_unified_sessions add column os_type varchar(16777216);
alter table (your_schema)_derived.snowplow_unified_sessions add column session_duration_s number(18,0);
alter table (your_schema)_derived.snowplow_unified_sessions add column session__previous_session_id varchar(36);
alter table (your_schema)_derived.snowplow_unified_sessions add column first_event_name varchar(1000);
alter table (your_schema)_derived.snowplow_unified_sessions add column last_event_name varchar(1000);
alter table  (your_schema)_derived.snowplow_unified_sessions add column if not exists stitched_user_id varchar(16777216);

alter table (your_schema)_derived.snowplow_unified_users rename column domain_userid to user_identifier;
alter table (your_schema)_derived.snowplow_unified_users drop column original_domain_userid;
alter table (your_schema)_derived.snowplow_unified_users rename column page_views to views;
alter table (your_schema)_derived.snowplow_unified_users rename column referrer to page_referrer;
alter table (your_schema)_derived.snowplow_unified_users add column on_mobile boolean;
update (your_schema)_derived.snowplow_unified_users
set on_mobile = false
where 1=1;
alter table (your_schema)_derived.snowplow_unified_users add column on_web boolean;
update (your_schema)_derived.snowplow_unified_users
set on_web = true
where 1=1;
alter table (your_schema)_derived.snowplow_unified_users add column screen_names_viewed number(30,0);
alter table (your_schema)_derived.snowplow_unified_users add column sessions_duration_s number(30,0);
alter table (your_schema)_derived.snowplow_unified_users add column active_days number(18,0);
alter table (your_schema)_derived.snowplow_unified_users add column last_platform varchar(255);
update (your_schema)_derived.snowplow_unified_users
set last_platform = 'web'
where 1=1;
alter table (your_schema)_derived.snowplow_unified_users add column last_screen_resolution varchar(16777216);
alter table (your_schema)_derived.snowplow_unified_users add column last_os_type varchar(16777216);
alter table (your_schema)_derived.snowplow_unified_users add column last_os_version varchar(16777216);
alter table (your_schema)_derived.snowplow_unified_users add column first_platform varchar(255);
update (your_schema)_derived.snowplow_unified_users
set first_platform = 'web'
where 1=1;
alter table (your_schema)_derived.snowplow_unified_users add column geo_latitude float;
alter table (your_schema)_derived.snowplow_unified_users add column geo_longitude float;
alter table (your_schema)_derived.snowplow_unified_users add column geo_timezone varchar(64);
alter table (your_schema)_derived.snowplow_unified_users add column geo_zipcode varchar(15);
alter table  (your_schema)_derived.snowplow_unified_users add column if not exists stitched_user_id varchar(16777216);

update (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest
set model = replace(model, 'snowplow_web', 'snowplow_unified')
where 1=1;

update (your_schema)_snowplow_manifest.snowplow_unified_incremental_manifest
set model = case when model = 'snowplow_unified_page_views_this_run' then 'snowplow_unified_views_this_run'
when model = 'snowplow_unified_page_views' then 'snowplow_unified_views' else model end
where 1=1;
```

### Step 3: Setting up a new dbt project for Unified Digital

Due to a package conflict resulting from the last Web and Unified Digital releases requiring a different version of the `snowplow-utils` package, it's probably best to create a new dbt project for running the package.

**3.1 Packages.yml**

```yml
packages:
  - package: snowplow/snowplow_unified
    version: 0.4.1
```

**3.2 Project.yml**

In the project.yml file, disable the mobile events:

```yml
vars:
  snowplow_unified:
    snowplow__enable_mobile: false
```

If you changed any of the default variables in your web project, add them here as well. While 90% of the variables stayed the same, there are some notable changes, deletions, additions, and default modifications. Have a look below or at the detailed [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page and adjust accordingly:

**Removed variables:**

- `snowplow__limit_page_views_to_session`

**Renamed variables:**

- `snowplow__page_view_stitching` to `snowplow__view_stitching`
- `snowplow__consent_cmp_visible` to `snowplow__cmp_visible_events` (Redshift only)
- `snowplow__consent_preferences` to `snowplow__consent_preferences_events` (Redshift only)
- `snowplow__cwv_context` to `snowplow__cwv_events` (Redshift only)

**Added variables:**

- `snowplow__conversion_stitching`
- `snowplow__conversion_passthroughs`
- `snowplow__enable_browser_context`
- `snowplow__enable_app_errors`
- `snowplow__enable_conversions`
- `snowplow__use_refr_if_mkt_null`
- `snowplow__browser_context`: com\_snowplowanalytics\_snowplow\_browser\_context\_1 (Redshift only)
- `snowplow__session_context`: com\_snowplowanalytics\_snowplow\_client\_session\_1 (Redshift only)
- `snowplow__mobile_context`: com\_snowplowanalytics\_snowplow\_mobile\_context\_1 (Redshift only)
- `snowplow__geolocation_context`: com\_snowplowanalytics\_snowplow\_geolocation\_context\_1 (Redshift only)
- `snowplow__application_context`: com\_snowplowanalytics\_mobile\_application\_1 (Redshift only)
- `snowplow__screen_context`: com\_snowplowanalytics\_mobile\_screen\_1 (Redshift only)
- `snowplow__application_error_events`: com\_snowplowanalytics\_snowplow\_application\_error\_1 (Redshift only)
- `snowplow__screen_view_events`: com\_snowplowanalytics\_mobile\_screen\_view\_1 (Redshift only)
- `snowplow__deep_link_context`: com\_snowplowanalytics\_mobile\_deep\_link\_1 (Redshift only)
- `snowplow__screen_summary_context`: com\_snowplowanalytics\_mobile\_screen\_summary\_1 (Redshift only)

**Variables with changed defaults:**

- `snowplow__session_identifiers`: `[{"schema" : "atomic", "field" : "domain_sessionid"}]`
- `snowplow__user_identifiers`: `[{"schema": "atomic", "field" : "domain_userid"}]` to none
- `snowplow__databricks_catalog`: `hive_metastore`

### Step 4: Verifying the new derived datasets

At this stage you could potentially run both packages simultaneously until you are happy to stop the old jobs that updated the Web package.

If you used the original derived tables as a source for your BI tool or for downstream data models, you can use the views below to replicate the original tables. This can help you maintain your existing reporting.

**SQL scripts to create views**

```sql
create view (your_schema)_derived.mock_snowplow_web_page_views as (
  select
    view_id as page_view_id,
    event_id,
    app_id,
    platform,
    user_id,
    user_identifier as domain_userid,
    device_identifier as original_domain_userid,
    stitched_user_id,
    network_userid,
    session_identifier as domain_sessionid,
    null as original_domain_sessionid,
    device_session_index as domain_sessionidx,
    view_in_session_index as page_view_in_session_index,
    views_in_session as page_views_in_session,
    dvce_created_tstamp,
    collector_tstamp,
    derived_tstamp,
    start_tstamp,
    end_tstamp,
    model_tstamp,
    engaged_time_in_s,
    absolute_time_in_s,
    horizontal_pixels_scrolled,
    vertical_pixels_scrolled,
    horizontal_percentage_scrolled,
    vertical_percentage_scrolled,
    doc_width,
    doc_height,
    content_group,
    page_title,
    page_url,
    page_urlscheme,
    page_urlhost,
    page_urlpath,
    page_urlquery,
    page_urlfragment,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network,
    default_channel_group,
    page_referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    geo_country,
    geo_region,
    geo_region_name,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_timezone,
    user_ipaddress,
    useragent,
    br_lang,
    br_viewwidth,
    br_viewheight,
    br_colordepth,
    br_renderengine,
    os_timezone,
    iab__category as category,
    iab__primary_impact as primary_impact,
    iab__reason as reason,
    iab__spider_or_robot as spider_or_robot,
    ua__useragent_family as useragent_family,
    ua__useragent_major as useragent_major,
    ua__useragent_minor as useragent_minor,
    ua__useragent_patch as useragent_patch,
    ua__useragent_version as useragent_version,
    ua__os_family as  os_family,
    ua__os_major as os_major,
    ua__os_minor as os_minor,
    ua__os_patch as os_patch,
    ua__os_patch_minor as os_patch_minor,
    ua__os_version as os_version,
    ua__device_family as device_family,
    yauaa__device_class as device_class,
    case
        when device_class = 'Desktop' then 'Desktop'
        when device_class = 'Phone' then 'Mobile'
        when device_class = 'Tablet' then 'Tablet'
        else 'Other'
    end as device_category,
    screen_resolution,
    yauaa__agent_class as agent_class,
    yauaa__agent_name as agent_name,
    yauaa__agent_name_version as agent_name_version,
    yauaa__agent_name_version_major as agent_name_version_major,
    yauaa__agent_version as agent_version,
    yauaa__agent_version_major as agent_version_major,
    yauaa__device_brand as device_brand,
    yauaa__device_name as device_name,
    yauaa__device_version as device_version,
    yauaa__layout_engine_class as layout_engine_class,
    yauaa__layout_engine_name as layout_engine_name,
    yauaa__layout_engine_name_version as layout_engine_name_version,
    yauaa__layout_engine_name_version_major as layout_engine_name_version_major,
    yauaa__layout_engine_version as layout_engine_version,
    yauaa__layout_engine_version_major as layout_engine_version_major,
    yauaa__operating_system_class as operating_system_class,
    yauaa__operating_system_name as operating_system_name,
    yauaa__operating_system_name_version as operating_system_name_version,
    yauaa__operating_system_version as operating_system_version,
  from (your_schema)_derived.snowplow_unified_views
);

create view (your_schema)_derived.mock_snowplow_web_sessions as (
  select
    app_id,
    platform,
    session_identifier as domain_sessionid,
    null as original_domain_sessionid,
    device_session_index as domain_sessionidx,
    start_tstamp,
    end_tstamp,
    model_tstamp,
    user_id,
    user_identifier as domain_userid,
    device_identifier as original_domain_userid,
    stitched_user_id,
    network_userid,
    views as page_views,
    engaged_time_in_s,
    total_events,
    is_engaged,
    absolute_time_in_s,
    first_page_title,
    first_page_url,
    first_page_urlscheme,
    first_page_urlhost,
    first_page_urlpath,
    first_page_urlquery,
    first_page_urlfragment,
    last_page_title,
    last_page_url,
    last_page_urlscheme,
    last_page_urlhost,
    last_page_urlpath,
    last_page_urlquery,
    last_page_urlfragment,
    page_referrer as referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network,
    mkt_source_platform,
    default_channel_group,
    first_geo_country as geo_country,
    null as geo_region,
    first_geo_region_name as geo_region_name,
    first_geo_city as geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_timezone,
    first_geo_country_name as geo_country_name,
    first_geo_continent as geo_continent,
    last_geo_country,
    last_geo_region_name,
    last_geo_city,
    last_geo_country_name,
    last_geo_continent,
    user_ipaddress,
    useragent,
    br_renderengine,
    first_br_lang as br_lang,
    first_br_lang_name as br_lang_name,
    last_br_lang,
    last_br_lang_name,
    os_timezone,
    iab__category as category,
    iab__primary_impact as primary_impact,
    iab__reason as reason,
    iab__spider_or_robot as spider_or_robot,
    ua__useragent_family as useragent_family,
    ua__useragent_major as useragent_major,
    ua__useragent_minor as useragent_minor,
    ua__useragent_patch as useragent_patch,
    ua__useragent_version as useragent_version,
    null as  os_family,
    ua__os_major as os_major,
    ua__os_minor as os_minor,
    ua__os_patch as os_patch,
    ua__os_patch as os_patch_minor,
    os_version,
    ua__device_family as device_family,
    yauaa__device_class as device_class,
    device_category,
    screen_resolution,
    yauaa__agent_class as agent_class,
    yauaa__agent_name as agent_name,
    yauaa__agent_name_version as agent_name_version,
    yauaa__agent_name_version_major as agent_name_version_major,
    yauaa__agent_version as agent_version,
    yauaa__agent_version_major as agent_version_major,
    null as device_brand,
    yauaa__device_name as device_name,
    yauaa__device_version as device_version,
    yauaa__layout_engine_class as layout_engine_class,
    yauaa__layout_engine_name as layout_engine_name,
    yauaa__layout_engine_name_version as layout_engine_name_version,
    yauaa__layout_engine_name_version_major as layout_engine_name_version_major,
    yauaa__layout_engine_version as layout_engine_version,
    yauaa__layout_engine_version_major as layout_engine_version_major,
    yauaa__operating_system_class as operating_system_class,
    yauaa__operating_system_name as operating_system_name,
    yauaa__operating_system_name_version as operating_system_name_version,
    yauaa__operating_system_version as operating_system_version
  from (your_schema)_derived.snowplow_unified_sessions
);

create view (your_schema)_derived.mock_snowplow_web_users as (
  select
    user_id,
    user_identifier as domain_userid,
    null as original_domain_userid,
    network_userid,
    start_tstamp,
    end_tstamp,
    model_tstamp,
    views as pageviews,
    sessions,
    engaged_time_in_s,
    first_page_title,
    first_page_url,
    first_page_urlscheme,
    first_page_urlhost,
    first_page_urlpath,
    first_page_urlquery,
    first_page_urlfragment,
    first_geo_country,
    first_geo_country_name,
    first_geo_continent,
    first_geo_city,
    first_geo_region_name,
    first_br_lang,
    first_br_lang_name,
    last_geo_country,
    last_geo_country_name,
    last_geo_continent,
    last_geo_city,
    last_geo_region_name,
    last_br_lang,
    last_br_lang_name,
    last_page_title,
    last_page_url,
    last_page_urlscheme,
    last_page_urlhost,
    last_page_urlpath,
    last_page_urlquery,
    last_page_urlfragment,
    page_referrer as referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network,
    mkt_source_platform,
    default_channel_group
    from (your_schema)_derived.snowplow_unified_users
);
```

---

# Configure custom aggregations for dbt models
> Details on custom aggregations in our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/

While [passthrough fields](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/) enable you to add fields from the event itself into derived tables, you may at times also wish to do aggregations on the identifier for that table, e.g. session identifier, to provide more information in your derived tables without having to make a full [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/). This is where Custom Aggregations come in; they allow you to calculate aggregations directly as part of the derived table logic, without having to build your own models.

## Availability

| Package | Minimum Required Version |
| ------- | ------------------------ |
| Unified | 0.3.0                    |

## Usage

To use a the custom aggregation, you need to set the relevant variable in your root `dbt_project.yml` file; e.g `snowplow__session_aggregations` (see your package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for a full list of aggregation variables).

These variables are lists of objects that define your aggregations; they have keys of the `type` of aggregations you wish to do, the `field` to aggregate over, and the `alias` to give your aggregation in the derived table.

The `field` can be any valid column sql, including `case when` statements or making use of other sql functions. For example, to count the number of `purchase` events over a session you would have:

```yml
vars:
  snowplow_unified:
    snowplow__view_aggregations: [{'type': 'sum', 'field': "case when event_name = 'purchase' then 1 else 0 end", 'alias': 'num_purchase_events'}]
```

The aggregation always runs on the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table so all events (with a session identifier) and columns are available for you to use.

Note that how to extract and use field from your entity or self-describing event columns will depend on your warehouse (see our [querying guide](/docs/destinations/warehouses-lakes/querying-data/#entities) for more information), and you are unable to use dbt macros in this variable.

### Supported Aggregations

Currently we support the following aggregations:

| Aggregation    | `type` value |
| -------------- | ------------ |
| Count          | `count`      |
| Sum            | `sum`        |
| Minimum        | `min`        |
| Maximum        | `max`        |
| Average        | `avg`        |
| Count Distinct | `countd`     |

## Usage Notes

> **Warning:** It is unlikely, although not impossible, that when using the SQL approach you may need to provide a table alias to avoid ambiguous references, in this case please see the model sql file for the specific alias used for the `snowplow_unified_base_events_this_run` table in each case.

---

# Configure custom sessionization
> Define custom session and user identifiers in Snowplow dbt packages using SQL expressions and custom logic.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/

> **Danger:** As this changes the core logic of the package, you should make sure you have a good understanding of how the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) works (including things such as quarantining sessions), and a good certainty around the tracking of any custom fields you plan to use.

Our packages come with a very robust [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) that makes sure when a session has a new event in the latest run, that _all_ events within that session are processed as part of that run to ensure all aggregations are correct and valid. There are some exceptions, such as quarantined sessions to avoid large table scans, but in general this approach ensures that you can be confident that in each run you have the full scope of a given session to process in our models, as well as any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you build to take advantage of this feature.

Traditionally a session in this sense has been a _true_ session i.e. it has been the `domain_sessionid` and `device_sessionid` value that has identified which events belong to the same session, however in our packages it is possible to overwrite the default field as part of the package logic and use your own session identifier, to ensure all events within a "session" are processed in the same run, whatever a "session" in this case means to you.

> **Info:** Throughout this page we refer to entity columns only by their major version (e.g. `com_mycompany_session_identifier_1`). If you use BigQuery, you will need to adjust these for the full version number (or use the `_*` wildcard to denote all minor versions), and may need to manage the combination of different versions yourself, details of how can be found below.

> **Tip:** Remember that any events with a null "session" identifier will be excluded from the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table and any processing of the package, make sure your identifier has a value for all events you want to process!

## Customizing session identifiers

To customize your session identifiers, you can make use of the `snowplow__session_identifiers` variable. This variable allows you to provide a list of identifiers that dbt will then try to use to create an identifying field for each session (by coalescing them in order), which will always be saved under the `session_identifier` column in your tables.

### Using additional atomic fields

For web data by default, your identifier will be the `domain_sessionid` field which is found in the atomic events table (this will vary by package). If you wanted to instead use a different field, say the `domain_userid` field that can also be found in the flat fields of the atomic events table, you could define your `snowplow__session_identifiers` as follows:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'atomic', 'field': 'domain_userid'}]
    ...
...
```

> **Tip:** `schema` here refers to the schemata the data was created from i.e. an atomic field, a self describing event, or an entity value. This has no relation to the database schema the table is in.

If you wanted to include multiple identifiers, in case the first one is sometimes null, then you could define the `snowplow__session_identifiers` as follows:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'atomic', 'field': 'domain_userid'},
                                    {'schema': 'atomic', 'field': 'domain_sessionid'}]
    ...
...
```

This would then compile into the following SQL code:

```sql
SELECT
    ...
    COALESCE(e.domain_userid, e.domain_sessionid, NULL) as session_identifier,
...
FROM events e
```

> **Info:** The order in which you provide your identifiers is the order of precedence they will take in the `COALESCE` statement. In other words, in the example above the value of `domain_sessionid` will **only** be used when `domain_userid` is `NULL`. If both are `NULL`, the `session_identifier` will be `NULL` for that event, and that event will not be processed with this package.

### Using custom entities

If you wanted to instead use session identifiers that come from a custom entity, you can do that as well. Let's assume you've created a session identifier entity that you attach to your events, and it's called `com_mycompany_session_identifier_1`. Let's also say you're interested in the `session_id` field of this entity as the identifier for each session. To make sure that your dbt models extract and use this field as the value for `session_identifier`, you need to define the `snowplow__session_identifiers` as follows:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si'}]
    ...
...
```

> **Warning:** Make sure you include a `prefix` value if you are running on **Postgres or Redshift**, as this ensures that you don't have duplicate column names somewhere in your SQL select statement. It is not required for the other warehouses, but is recommended.

Similar to before, if you want to combine multiple identifiers in different (or the same) entities, you can do so by defining your `snowplow__session_identifiers` as shown below. First, however, let's assume there's another entity called `com_mycompany_logged_session_id_1` which has both a `logged_in_id` and `session_identifier` field in it. To include all of these:

```yml
vars:
    ...
    snowplow__session_identifiers: [
            {'schema': 'com_mycompany_logged_session_id_1', 'field': 'logged_in_id', 'prefix': 'lsi'},
            {'schema': 'com_mycompany_logged_session_id_1', 'field': 'session_identifier', 'prefix': 'lsi'},
            {'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si'}]
    ...
...
```

> **Warning:** Make sure if you include multiple fields from the same entity that you give them the same prefix.

This will then render into the following SQL:

**Databricks & Snowflake:**

```sql
SELECT
    ...
    COALESCE(
        com_mycompany_logged_session_id_1[0].logged_in_id,
        com_mycompany_logged_session_id_1[0].session_identifier,
        com_mycompany_session_identifier_1[0].session_id,
        NULL
        ) as session_identifier,
...
```

**Bigquery:**

```sql
SELECT
    ...
    COALESCE(
        com_mycompany_logged_session_id_1_0_0[safe_offset(0)].logged_in_id,
        com_mycompany_logged_session_id_1_0_0[safe_offset(0)].session_identifier,
        com_mycompany_session_identifier_1_0_0[safe_offset(0)].session_id,
        NULL
        ) as session_identifier,
...
```

**Redshift & Postgres:**

```sql
SELECT
    ...
    COALESCE(
        lsi_logged_in_id,
        lsi_session_identifier,
        si_session_id,
        NULL
        ) as session_identifier,
...
```

Note that we will manage joining any self describing event or entity tables for you as part of the package logic.

***

Again the order of precedence being decided by the order of your list of identifiers.

### Schema evolution of custom entities

> **Tip:** You can use this approach for schema evolution in BigQuery for any changes, in other warehouses you would only need to do this for major version changes.

This can be extended into entities where schema evolution takes place. Suppose your session identifying entity has previously always been `com_mycompany_session_identifier_1`, where you extract the `session_id` field as the identifier. Then suppose you introduce some breaking changes that cause you to now use the `com_mycompany_session_identifier_2` entity, where you extract `new_session_id` as the identifying field. In order to ensure you can track both as your session identifier, you would define your `snowplow__session_identifiers` as follows:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_2', 'field': 'new_session_id', 'prefix': 'si_t'},
                                    {'schema': 'com_mycompany_session_identifier_1', 'field': 'session_id', 'prefix': 'si_o'}]
    ...
...
```

This setup implies that the `new_session_id` from `com_mycompany_session_identifier_2` should have precedence over the `session_id` field from `com_mycompany_session_identifier_1`, meaning that if both are filled for a particular event, the `new_session_id` value would be the one present in the `session_identifier` field in your dbt tables. If you'd prefer to have the precedence swapped, you can swap the ordering in the variable in your `dbt_project.yml`.

**BigQuery Version Changes**

Handling minor version bumps of your schemas where this isn't automatically handled by the loader, such as in BigQuery, works in the exact same way. A small example of bumping `com_mycompany_session_identifier_1_0_0` to `com_mycompany_session_identifier_1_1_0` but leaving the `session_id` field as the key identifier would look as follows:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1_1_0', 'field': 'session_id', 'prefix': 'si_t'},
                                    {'schema': 'com_mycompany_session_identifier_1_0_0', 'field': 'session_id', 'prefix': 'si_o'}]
    ...
...
```

Alternatively you can use a `_*` in the schema name to get the field from all columns of the same major version:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'com_mycompany_session_identifier_1_*1_0*', 'field': 'session_id', 'prefix': 'si'}]
    ...
...
```

### Combining atomic fields and custom entities

Combining atomic fields and custom entities is straightforward if you're comfortable with what we described above. Let's say you want to combine the `logged_in_id` field from the `com_mycompany_logged_session_id_1` entity together with the standard `domain_sessionid` field in the `events` table. To achieve that, you would include the following in your `dbt_project.yml`:

```yml
vars:
    ...
    snowplow__session_identifiers: [{'schema': 'com_mycompany_logged_session_id_1', 'field': 'logged_in_id', 'prefix': 'lsi'},
                                    {'schema': 'atomic', 'field': 'domain_sessionid', 'prefix': 'e'}]
    ...
...
```

### Adding your own custom session logic

If there are session identifiers that are more complicated to utilize, then you can also provide your own session logic that will be used instead of the logic explained in the preceding sections. As an example, if you would want to concat two fields to create a session identifier, or instead apply a SQL function to a field to then use as a session identifier, that is completely possible using the `snowplow__session_sql` variable.

> **Info:** Defining the `snowplow__session_sql` variable will ensure that the package takes it's value as the `session_identifier` **over** anything you may have defined with the `snowplow__session_identifiers` variable.

> **Warning:** For Redshift/Postgres, if you want to leverage custom entities for your custom session logic, you will need to include them in the `snowplow__session_identifiers` variable in the same way as in previous sections to ensure they are correctly joined to the table.

#### Concatenating multiple fields to create a session identifier

To start, suppose you want to combine the atomic `domain_sessionid` and `domain_userid` fields to create a session identifier. It's simple to do that by defining the following variable in your `dbt_project.yml`:

```yml
vars:
   ...
   snowplow__session_sql: "e.domain_userid || '_' || e.domain_sessionid"
   ...
...
```

This would be parsed into the following SQL:

```sql
SELECT
    ...
    e.domain_userid || '_' || e.domain_sessionid as session_identifier,
...
```

#### Applying a SQL function to a field to use as a session identifier

Instead, suppose you want to take the `DATE` value of your `derived_tstamp` as your session identifier. It's also simple to do that by defining the following variable in your `dbt_project.yml`:

```yml
vars:
   ...
   snowplow__session_sql: "DATE(e.derived_tstamp)"
   ...
```

This would be parsed into the following SQL:

```sql
SELECT
    ...
    DATE(e.derived_tstamp) as session_identifier,
...
```

---

# Users and identity stitching
> Details on mapping between `domain_userid` and `user_id` in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/identity-stitching/

**Identity stitching is the process of taking various user identifiers and combining them into a single user identifier, to better identify and track users throughout their journey on your site/app.**

Stitching users together is not an easy task: depending on the typical user journey the complexity can range from having individually identified (logged in) users, thus not having to do any extra modeling to never identified users mainly using the same common public device (e.g. school or library) where it is technically impossible to do any user stitching. As stitching is a reiterative process as it constantly needs to be updated after each incremental run for a desirably large range of data, compute power and extra expenses as well as time constraints may limit and dictate the best course of action.

## Session stitching

For the out-of-the-box user stitching we opt for the sweet spot method: applying a logic that the majority of our users will benefit from while not introducing compute-heavy calculations. We do this in our core dbt package, the `Snowplow Unified Digital` package that produces a `users` table.

We provide user stitching on one layer: between the primary `user_identifier` field and a custom defined (`var('snowplow__user_stitching_id')`) id field. The package considers the `user_identifier` field as the primary key for the users table and this would typically be the most "reliable" cookie based field, hence it is defaulted to `domain_userid` when processing web events and the session\_context entity based user\_id for mobile events. The custom defined id to be used for stitching is recommended to be the `atomic.user_id` field that is used for tracking logged in users. This reliable user identifier field is the common denominator for applying the so called session-stitching in our packages.

This works by having an `User Mapping` table that collects and incrementally updates the latest official logged in `user_id` field for all user\_identifiers processed in the current run (taken from the event\_this\_run table). This mapping is then applied to the derived tables (e.g views table if `snowplow__view_stitching` is enabled and both the sessions and users table if `snowplow__session_stitching` is enabled). The update is carried out by a post-hook (defined in the config of each derived incremental table), which updates the `stitched_user_id` column with the latest mapping. If no mapping is present, the default value for `stitched_user_id` is the user identifier itself. This process is known as session stitching, and effectively allows you to attribute logged-in and non-logged-in sessions back to a single user.

## Cross platform stitching

The `snowplow_unified` package means that all the user data, from both web and mobile, is modeled in one place. This makes it easy to effectively perform cross-platform stitching, which means that as soon as a users identify themselves by logging in as the same user on separate platforms, all the user data will be found within one package making it convenient for performing further analysis. We encourage everyone to take the base stitching logic provided by the package further by applying a custom aggregation downstream layer that takes the first/last fields per stitched\_user\_id from the users table as well a applying additional stitching based on custom user\_mapping table(s) depending on need.

> **Info:** In order for the user\_mapping table to successfully update the stitched\_user\_id field in the users table, it needs to stay unique on session\_identifier which means that if for some reason there are multiple user\_ids per user\_identifier, we take the latest one. For a standard use case when the user\_identifier is a cookie based domain\_userid and the user\_stitching\_id is the logged in atomic.user\_id field, we encourage everyone to [remove cookies](/docs/sources/web-trackers/anonymous-tracking/#clear-user-data) to force creating a new session upon the user log out action to avoid a scenario when multiple users log in and out quickly after one another on the same device (e.g. school, library). For tracking advice, have a look at [this](/docs/events/identifiers/#reset-generated-identifiers-after-the-user-logs-out) documentation.

![Session stitching in the Unified Digital Model](/assets/images/session_stitching_light_unified.drawio-e4c7cb45d21c40e619445fcb19a58f04.png)

If required, this update operation can be disabled by setting in your `dbt_project.yml` file (selecting one of web/mobile, or both, as appropriate):

```yml
vars:
  snowplow_<package>:
    snowplow__session_stitching: false
```

> **Tip:** Consider processing costs before enabling `snowplow__view_stitching` to `true`. It may be enough to apply this with less frequency than on sessions to keep costs down, by only enabling this at runtime on only some of the runs.

### Custom solutions

#### **Customizing user identifiers**

Customizing user identifiers works in the exact same way as [customizing session identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/#customizing-session-identifiers), please refer to that link to understand the breakdown of how to set this up, although you need to make use of the `snowplow__user_identifiers` variable instead of the `snowplow__session_identifiers`, and `snowplow__user_sql` in place of `snowplow__session_sql`.

```yml
vars:
  snowplow_unified:
    # This is an example of user identifiers for BigQuery
    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2_*", "field" : "user_id"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1_*", "field" : "user_id"}]
    # For Databricks
    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "user_id"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "user_id"}]
    # For Redshift/Postgres
    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "user_id", "prefix" : "ui_t", "alias": "uidt"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "user_id", "prefix": "ui_o", "alias": "uido"}]
    # For Snowflake
    snowplow__user_identifiers: [{"schema": "contexts_com_snowplowanalytics_user_identifier_2", "field" : "userId"}, {"schema": "contexts_com_snowplowanalytics_user_identifier_1", "field" : "userId"}]
```

If you need a specific way to refer to a custom user you can also use the `snowplow_user_sql` variable, which will override any default or overwrites on `snowplow__user_identifiers`.

Please note, however, that the `snowplow__user_identifiers` variable is not designed to handle user stitching the way you might expect it. Currently, there is a limitation that if any of the identifiers in the list do not persist throughout the session, there is no guarantee that the highest-prestige identifier (first in the list) will take precedence.

This is particularly relevant for those adding user\_id as the first element in their list, as it may be NULL for some events within a session. For that it is best to rely on the package based user stitching logic, leave user\_id as the `snowplow__user_stitching_id` and rely on the `stitched_user_id` field produced by the package in each derived tables as the main identifier field. This way even if the user logged in for some period of the time within a session the user\_id field will be found and prioritized over the rest.

Alternatively, to ensure a deterministic user selection, you could rely on the `snowplow_user_sql` variable instead to something like this:

```sql
COALESCE(
    MAX(user_id),
    MAX(domain_userid),
    MAX(network_userid)
) OVER (PARTITION BY session_identifier)
```

This ensures that `MAX(user_id)` is computed separately before applying COALESCE() at the session level, preventing another identifier from accidentally taking precedence.

#### **Handling Anonymized Users**

In case of applying Client-side anonymization with session tracking, the `userId` property of the `contexts_com_snowplowanalytics_snowplow_client_session_1` equates to a null UUID which will appear as `00000000-0000-0000-0000-000000000000` in the database. It may be convenient to make this field an actual NULL field to make it easier to exclude them from modeling (e.g the user mapping table of the Unified Package excludes null values). This can be made possible with the use of the `snowplow__user_sql` variable, however, this means that the extraction from the relevant context/sde field needs to be handled manually.

Example implementation (Snowflake):

```yml
vars:
  snowplow_unified:
    snowplow__user_sql: 'coalesce(case when contexts_com_snowplowanalytics_snowplow_client_session_1[0]:userId::varchar(36) == "00000000-0000-0000-0000-000000000000" then null else contexts_com_snowplowanalytics_snowplow_client_session_1[0]:userId::varchar(36) end, domain_userid)'
```

> **Info:** Defining the `snowplow__user_sql` variable will ensure that the package takes its value as the `user_identifier` **over** anything you may have defined with the `snowplow__user_identifiers` variable.

User mapping is typically not a one-size-fits-all exercise. Depending on your tracking implementation, business needs and desired level of sophistication you may want to write bespoke logic. Please refer to this [blog post](https://snowplow.io/blog/developing-a-single-customer-view-with-snowplow/) for ideas. The unified package offer the ability to change what field is used as your stitched user id, so instead of `user_id` you can use any field you wish (note that it will still be called `user_id` in your mapping table), and by taking advantage of the [custom sessionization and users](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) you can also change the field used as the user\_identifier (unified model).

### Overview

The below diagram shows a potential flow your user may take across multiple devices. It does not matter if they are web or mobile events as Unified will correctly process and stitch both. As the user progresses through the (simplified) sessions table tracks their sessions, user identifier, user ID, and stitched user id. Once a user ID is identifier for specific user identifier it is backdated in the stitched column for all sessions with that identifier. Note that this is not possible _until_ the user logs in during a session.

![Overview of stitching scenarios](/assets/images/stitching_scenarios.drawio-4e90f4a1e53df103697695a6ca3f0d07.png)

---

# Snowplow dbt package features
> Top level page for features contained in our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/

Our packages contain a wide range of functionality and features you can use to configure the package to suit your needs. Details of these features can be found in the following pages.

---

# Model entities and self-describing events
> Model entities and self-describing events in Snowplow dbt packages with out-of-the-box support and custom modeling.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/

Our dbt packages provide out-of-the-box modeling for many [entities](/docs/fundamentals/entities/) that are relevant to each package. Fields from these contexts are added to the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table for use within downstream models. For example our [unified package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) will add all fields from the [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) entity by setting the `snowplow__enable_yauaa` to true.

Relevant [self-describing events](/docs/fundamentals/events/#self-describing-events) fields are extracted in the same way as needed per-package, and are often dependent on enabling optional modules.

> **Tip:** In some cases, these fields may exist but be null when the entity or self-describing event is not enabled in the project yaml to enable more consistent structure to the events this run table.

## Custom Entities & Events

For warehouses other than Redshift, all columns in your atomic events table are available in the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table, so you can use them as needed in any other models.

For Redshift we provide the `snowplow__entities_or_sdes` in our packages that have an events this run table. This variable takes the name of your entity or self-describing event table, a prefix, an alias, and if it is a single entity.

> **Warning:** Only entities that are "single valued" should be added directly to the event this run table i.e. the list should only contain one instance of the entity. A multi-valued entity would cause duplicates in the events this run table and lead to incorrect data in all downstream models. See below for how to use multi-valued entities.

This variable is a list of objects which have the following structure:

| Key             | Description                                                                                                                                                                                                  | Example                          |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
| `schema`        | The schema denotes the name of the entity or SDE that you would like to join, which should also be the name of the table that is in your warehouse.                                                          | `contexts_com_mycompany_click_1` |
| `prefix`        | The prefix that each field in the context will receive. E.g with a prefix of `my_click` and a field name of `id`, this field will be accessible in the events this run table under the `my_click_id` column. | `my_click`                       |
| `alias`         | The alias that is used for the context table join, for reference in your custom SQL queries.                                                                                                                 | `mc`                             |
| `single_entity` | A boolean to say whether this is a single entity or whether there can be multiple for each event. Should always be set to `true`.                                                                            | `true`                           |

An example with two custom contexts would be:

```yaml
...
vars:
  snowplow_<package_name>:
    snowplow__entities_or_sdes: [
        {'schema': 'contexts_com_mycompany_click_1', 'prefix': 'click', 'alias': 'mc', 'single_entity': true},
        {'schema': 'contexts_com_mycompany_submit_1', 'prefix': 'submit', 'alias': 'sub', 'single_entity': true}
        ]
...
```

### Multi-valued Entities

In the case of a mutli-valued entity, you must make use of custom models and the `snowplow_utils.get_sde_or_context` to manage de-duplication. See the [using multi-valued entities](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) custom model example page for more information.

---

# Passthrough fields for derived tables
> Details for how to pass additional fields through to the derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/passthrough-fields/

Passthrough fields are the term used to define any field from your [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) type table that you want available _as is_ in a derived table, i.e. these fields are _passed through_ the processing to be available in your derived tables. There are many use cases for this, but the most common is the need to add a field from a custom context to the e.g. the views table to use as an additional dimension in analysis. Previously you would have to write a custom model to join this field on, or join the original view event at the time of the analysis, but with passthrough fields this is now far easier (and cheaper) to achieve.

## Availability

| Package      | Minimum Required Version |
| ------------ | ------------------------ |
| Unified      | 0.3.0                    |
| Media Player | 0.7.0                    |
| Ecommerce    | 0.6.0                    |

## Usage

To enable the passthrough fields, you need to set the relevant variable in your root `dbt_project.yml` file. E.g. `snowplow__view_passthroughs` (see your package [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for a full list of passthrough variables).

> **Info:** Note that in some cases you may be able to specify multiple variables for the same table for the first and last of a given record (e.g. first and last session for a user).

To set these variables, you need to define field references in the format of a list of `field names` (in case it is one of the event table columns that can be used as is without the need to alias it) or `dictionaries` specifying the `sql` which defines the sql creating a new field, as well as the `alias` to give a name to the custom field name. The list of such field references to be used as passthrough fields will get compiled and used in a select clause, therefore any valid sql within that context is applicable.

### 1. Passing through columns as is

In case of out-of-the-box event table columns, you can simply reference the name of the field as it appears as a column within your events table. E.g.:

```yml

snowplow__view_passthroughs: ['page_url', 'page_title']
```

For those warehouses where the self-describing events or custom entities are also available in the events table you can opt for unnesting the column in a downstream table only, passing through the unnested field as is. E.g.:

```yml
snowplow__view_passthroughs: ['contexts_my_entity_1']
```

### 2. SQL-defined Columns

You can also use any valid sql supported by your warehouse that can be used in a select clause. This time you need to use the dictionary format, specifying the relevant sql and the field alias. E.g.:

```yml
snowplow__view_passthroughs: [
  \{'sql': 'COALESCE(page_url, refr_url)', 'alias': 'final_url'}
]
```

### 3. Extracting fields from a self-describing event or custom context entity

A more useful case for the SQL block is to extract a specific field from an entity or sde. However, please note that how to extract a field from your self describing event / entity column will depend on your warehouse:

**Redshift, Postgres:**

**Step 1. Making fields available in the events table**

For Redshift and Postgres users, entities and self describing events are not part of the standard events table. Instead, each type of entity/sde is in its own table. The table name and the fields in the table will be determined by the entity’s schema. See [how schemas translate to the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/) for more details.

In order for you to use fields from there through passthrough fields, you would need to first make sure that those fields are part of the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table. Any custom entities or self-describing events can be added to this table (which get de-duped by taking the earliest `collector_tstamp` record) by using the `snowplow__entities_or_sdes` variable in our package. See [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for more information and examples.

> **Warning:** It is unlikely, although not impossible, that when using the SQL approach you may need to provide a table alias to avoid ambiguous references, in this case please see the model sql file for the specific alias used for the `snowplow_unified_base_events_this_run` table in each case.

```yml
snowplow__entities_or_sdes: [
      \{'schema': 'custom_table_name', 'prefix': 'my_entity_1', 'alias': 'entity_1', 'single_entity': true},
]
```

**Step 2. Referencing the fields as passthrough fields**

Once you add those fields into your events\_this\_run table through the `snowplow__entities_or_sdes` variable, you can simply refer to them as passthrough fields, coalesce different versions etc.:

```yml
snowplow__view_passthroughs: [
  'my_entity_1_field_x',
  'my_entity_1_field_y',
  'my_entity_1_field_z',
  \{'sql': 'COALESCE(my_entity_1_field_x, my_entity_1_field_y)', 'alias': 'entity_field_combined'}
]
```

**BigQuery:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from self-describing events
  {'sql': 'unstruct_event_my_event_1_0_0.field_name', 'alias': 'my_event_field'},
  # Custom sql example (joining two extracted fields together)
  {'sql': 'unstruct_event_my_event_1_0_0.field1 || unstruct_event_my_event_1_0_0.field2', 'alias': 'my_event_fields'}
]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from a custom context field
  {'sql': 'contexts_my_entity_1_0_0[SAFE_OFFSET(0)].field_name', 'alias': 'entity_field'},
  # Custom sql example (coalescing different major versions of the same field)
  \{'sql': 'COALESCE(contexts_entity_1[SAFE_OFFSET(0)].field, contexts_entity_2[SAFE_OFFSET(0)].field)', 'alias': 'entity_field_combined'}
]
```

**Snowflake:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from self-describing events
  {'sql': 'unstruct_event_my_event_1:fieldName::varchar', 'alias': 'event_field'},
  # Custom sql example (joining two extracted fields together)
  {'sql': 'unstruct_event_my_event_1:field1::varchar || unstruct_event_my_event_1:field2::varchar', 'alias': 'combined_fields'}
]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from a custom context field
  {'sql': 'contexts_my_entity_1[0]:fieldName::varchar', 'alias': 'entity_field'},
  # Custom sql example (coalescing different major versions of the same field)
  \{'sql': 'COALESCE(contexts_my_entity_1[0]:field::varchar, contexts_my_entity_2[0]:field::varchar)', 'alias': 'entity_field_combined'}
]
```

**Databricks, Spark SQL:**

**Self-describing Events**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from self-describing events
  {'sql': 'unstruct_event_my_event_1.field_name', 'alias': 'event_field'},
  # Custom sql example (joining two extracted fields together)
  {'sql': 'unstruct_event_my_event_1.field1 || unstruct_event_my_event_1.field2', 'alias': 'combined_fields'}
]
```

**Context Entities**

```yml
snowplow__view_passthroughs: [
  # Extract specific fields from a custom context field
  {'sql': 'contexts_my_entity_1[0].field_name', 'alias': 'entity_field'},
  # Custom sql example (coalescing different major versions of the same field)
  \{'sql': 'COALESCE(contexts_my_entity_1[0].field, contexts_my_entity_2[0].field)', 'alias': 'entity_field_combined'}
]
```

***

> **Tip:** When using passthrough fields with entities or self-describing events, always consider:
>
> 1. Field type casting may be required (especially in Snowflake)
> 2. Array indexes start at 0 for the first occurrence
> 3. Use COALESCE when dealing with multiple versions of the same field
> 4. Please note you are unable to use dbt macros in this variable, and also not able to use lateral flatten / explode type extractions.

## Usage Notes

The event that the passthrough fields are taken from depends on the derived table you are adding them to. For example, in the `snowplow_unified_views` table, the `page_view` event is used, not other events, such as `page_ping` or any other custom events. Therefore, if you are adding a passthrough field to the `snowplow_unified_views` table, you must access a property or custom entity that is attached to a `page_view` event.

A general rule of thumb is that the field comes from the _first_ event of that type at that table level. If you are unsure, you can always check the [configuration page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) for the variable description or the model sql.

As mentioned, the fields are passed through _as is_, which means it is not currently possible aggregate the value of a field across a page view or session as we do some other fields in the table. In the case where this is required you should use the [custom aggregations](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/) if that is supported by the package, or build a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/).

> **Warning:** In certain cases, such as the users table in Unified, it may be required to first set passthrough fields on an upstream table (the sessions table in that case) to make sure it is available for selection. The best way to identify this is to look at the DAG in the dbt docs for the package you are using.

---

# Table grants for dbt packages
> Details for granted access to our tables to other users/roles
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/table-grants/

> **Tip:** This functionality requires dispatching to our macros over dbt core, see how to do this on the [dispatch setup](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/) page.

## Availability

> **Tip:** Any package can make use of the table grants feature provided by the `snowplow__grant_select_to` variable if you are using at least version 0.16.2 of snowplow-utils, even if the variable is not listed in the configuration. Granting usage on schemas however requires specific package versions.

| Package         | Minimum Required Version |
| --------------- | ------------------------ |
| Unified Digital | 0.3.0                    |
| E-commerce      | 0.8.1                    |
| Media Player    | 0.7.2                    |
| Normalize       | 0.3.5                    |
| Attribution     | 0.2.0                    |

Note that this feature is not supported for BigQuery due to the different approach to permissions they take via IAM roles.

## Usage

### Granting select on tables

To grant `select` on all tables created within our package, you can provide a list of the users/roles in the `snowplow__grant_select_to` variable e.g.

```yaml
vars:
  snowplow_<package_name>:
    snowplow__grant_select_to: ['myuser1', 'myuser2']
```

Note that these user/role names are case sensitive. Databricks Principals are also supported. If the user does not exist an error will occur when the grant tries to run. This feature is compatible with the built-in [dbt grants](https://docs.getdbt.com/reference/resource-configs/grants) functionality and we will grant to a combination of the two.

> **Warning:** It is important to scope this to the relevant package, if you set this variable at the top level of your project then all models will have this grant applied.

> **Warning:** Note this will overwrite any existing grants applied to the table manually in the warehouse.

### Granting usage on schemas

In the case of some warehouses, users also need `usage` permissions on the schema a table is in to be able to access the data. We provide this functionality via a post-hook that grants usage on **any** schema interacted with during the `dbt run` to the users listed in `snowplow__grant_select_to`.

> **Danger:** Due to limitations with dbt, we are not able to scope this only to schemas interacted with by our package. This means **ALL** schemas in the run will have `usage` granted to these users.

This functionality will only trigger if `snowplow__grant_select_to` is not empty, and you can disable this by setting `snowplow__grant_schema_usage` to false.

---

# Snowplow dbt packages deduplicate events
> How we deal with duplicate events in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/

Duplicates in your data can happen, because of Snowplow's _At Least Once_ delivery approach. However, it's important for analysis and reporting that these duplicates are removed when doing any kind of data modeling. As part of our packages, we make sure to remove duplicate events at the very start to ensure any downstream data is correct.

All our packages perform de-duplication on both `event_id`'s as part of the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run), and then ensure distinct counts and records are used for any higher level tables in the packages e.g. page views. The de-duplication method for Redshift & Postgres is different to BigQuery, Snowflake, & Databricks due to their shredded table design.

## BigQuery, Snowflake, & Databricks

Any duplicate `event_id`s are removed by taking the earliest `collector_tstamp` record. In the case of multiple rows with this timestamp, we take one at random (as they will all be the same). The same methodology is applied to `page/screen_view_id`s, however we order by `derived_tstamp`.

There should be no need to further de-duplicate the base [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table or any contexts for each row.

## Redshift & Postgres

### Single-valued entities

When events are expected to only have at most a single entity in the attached context, any duplicate `event_id`s are removed from the atomic events table, and any context/self-describing event tables, by taking the earliest `collector_tstamp` record. In the case of multiple rows with this timestamp, we take one at random (as they will all be the same).

Any custom entities or self-describing events are able to be added to the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table, and are de-duped on by taking the earliest `collector_tstamp` record, by using the `snowplow__entities_or_sdes` variable in our package. See [modeling entities](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) for more information and examples.

### Multi-valued entities

In the case where it may be possible for a entity to contain multiple instances (e.g. products returned in a search result) a more complex approach is taken to ensure that all of the instances in the attached entity are still in the table before the join, even in the case where some of these entities may be identical.

When we de-duplicate the events this run table we keep the number of duplicates there were. In the de-duplication of the entity table we generate a row number per unique combination of **all** fields in the record. A join is then made on `root_id` and `root_tstamp` as before, but with an **additional** clause that the row number is a multiple of the number of duplicates to support the 1-to-many join. This ensures all duplicates are removed while retaining all original instances of the entity. Because this leads to multiple rows per event, this is only ever done in downstream models in our packages.

If you need to make use of a multi-valued entity in a custom model, see the [using multi-valued entities](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/examples/using-mulit-valued-entities/) custom model example page.

---

# Dispatch setup for dbt packages
> Details on how to setup the dispatch order for macros
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/

Some of the functionality of our packages overwrites the default functionality of dbt core. To enable the use of our version over dbt, you must add the following must be added to your `dbt_project.yml` file once.

```yml
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
```

This will ensure that, when it exists, our version of a macro is used over the dbt-core version.

---

# Full calculation for run timestamps
> A detailed deep dive into how we calculate which events to process in a run.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/full-run-calculation/

> **Tip:** The following information is meant for educational purposes only, there should never be a need to alter this logic as the package manages it all for you.

## Timestamps used in our packages

When calculating the timestamps required for our [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) by default we use `collector_tstamp`; historically this was the field the events table was most likely to be partitioned on. Nowadays `load_tstamp` is increasingly common, and users may have changed the partition field of their table. To accommodate this and make use of more efficient filtering you can alter the field we use to identify new events using the `snowplow__session_timestamp` variable. Note that where possible this should match your partition key, and the choice will impact the calculation when looking for which events to process.

## Calculation for run timestamps

The details provided above cover how we calculate the first step of the date range to process on a run, based on the state of models in the package and any new data. This is not the full story as in a given run the date range of specific events that are available in the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table will differ from this. What follows is the full details of how this is calculated and processed.

### Step 1: Calculate Base New Event Limits

The first step is to identify the model limits for the run. This is accomplished by the [`get_incremental_manifest_status` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/get_incremental_manifest_status.sql) to find the min and max last success in the manifest, and then uses the [`get_run_limits` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/get_run_limits.sql) to identify which state the package is in and calculate the timestamp range for this run, as detailed above.

This range is then printed to the console and stored in the `snowplow_<package_name>_base_new_event_limits` table for later use.

### Step 2: Update sessions lifecycle manifest

Next, we need to now process the sessions with new events into the [sessions lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#sessions-lifecycle-manifest) table. Here we apply many filters to the events table before calculating the [identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) and minimum and maximum `snowplow__session_timestamp` for each session.

The filters that are applied are:

#### Filter to events within the limits

To minimize the table scan and ensure a consistent processing of data, we filter to events with a timestamp bound by the limits defined in step 1.

```jinja2
and {{ session_timestamp }} >= {{ lower_limit }}
and {{ session_timestamp }} <= {{ upper_limit }}
```

If `snowplow__derived_tstamp_partitioned` is set to true, and it is running in BigQuery, this will also apply the filter to the `derived_tstamp` field as well.

#### Filter late arriving data

To avoid scanning too far back in the events table for a session, and to avoid recalculating sessions far in the past, we filter out late arriving data based on the device created and send timestamp. How late these events can be sent is defined by the `snowplow__days_late_allowed` variable, which has a default of 3.

```jinja2
and dvce_sent_tstamp <= {{ snowplow_utils.timestamp_add('day', days_late_allowed, 'dvce_created_tstamp') }}
```

#### Filter app IDs

Based on the input provided in the `snowplow__app_id` variable, we filter to only those app ids provided.

```jinja2
and {{ snowplow_utils.app_id_filter(app_ids) }}
```

#### Filter out quarantined sessions

We also exclude any session identifiers listed in the [quarantine manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) table to avoid processing long running sessions. For a session to be quarantined it must have events spanning longer than the value in your `snowplow__max_session_days` variable.

```jinja2
where session_identifier is not null
    and not exists (select 1 from {{ ref(quarantined_sessions) }} as a where a.session_identifier = e.session_identifier)
```

Once this is all calculated, we merge these with the existing manifest and take the least of the start timestamp, and the greatest of the end timestamp. In the case of a session running over the `snowplow__max_session_days` the end timestamp is re-calculated based on this instead.

### Step 3: Building sessions this run

We next identify which sessions need to be included in the current run, based on our event run limits and the _start_ timestamp of that session. Using the [`return_base_new_event_limits` macro](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/incremental_hooks/return_base_new_event_limits.sql) we get the upper and lower limits as calculated in step 1, but also the lower limit minus the `snowplow__max_session_days` to get the earliest a session could start and still have events included in this run (the session start limit).

We then include in the run any session that:

- Starts after the session start limit
- Starts before or at the upper limit for the run
- Ends after the lower limit for the run

The [base sessions this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#base-sessions-this-run) table then contains any sessions that satisfy these conditions from the lifecycle manifest, but with an `end_tstamp` capped at the upper limit.

### Step 4: Build events this run

Finally we query the [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table and filter to events that satisfy the following conditions:

- Session identifier is within the base sessions this run table created in Step 3
- Timestamp is between the min start and max end of sessions in the base sessions this run table created in Step 3
- Timestamp is greater than or equal to the start timestamp for that specific session identifier in base sessions this run table created in Step 3
- (Optional) App ID is in the list provided

This is the [deduplicated](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/) and other transformations are applied such as adding any entities, SDEs, or custom sql.

> **Tip:** The result of all of this is that the range of timestamps in the events this run table will be larger than the range printed in the log, and stored in the base new events limits table, but is required to ensure we correctly process complete sessions and are able to handle late arriving data.

---

# Incremental sessionization logic
> A detailed overview of our incremental run logic using manifest tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/

The general principle behind an incremental model is to identify new events/rows since the previous run of the model, and only process these new events. This minimizes cost and reduces run times. This is great for basic event streams, however you begin to encounter issues for any aggregations you wish to do. To resolve this we treat a `session` as the atomic unit of the majority of our packages, ensuring that when we have a new event for a previously processed session, we have to reprocess all historic events for that session as well as the new events - otherwise our metrics would be incorrect. Note that because we allow [custom session identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) this does not have to be a true web-type `session`.

The logic we have implemented to achieve this is:

1. Identify new events since the previous run of the package
2. Identify all `session_identifier`s associated with the new events
3. Look back over the events table to find all events associated with these `session_identifier`
4. Use only this set of events in all downstream processing

Given the large nature of event tables, Step 3 can be an expensive operation, we solve this issue in 3 ways:

| Problem                                                                                                           | Solution                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| We have to scan the whole events table to find events for a subset of sessions                                    | **Record when any given session started.** We use a [lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#sessions-lifecycle-manifest) table to record the start and end timestamp of all sessions.                                                                                                                                                                                                       |
| Sessions generated by bots can persist for years which would mean scanning years of data every run of the package | **Limit the maximum allowed session length.** We use a [quarantine](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#quarantine-table) table to store the `session_identifier` of any sessions that have exceeded the max allowed session length (`snowplow__max_session_days`). For such sessions, all events are processed up until the max allowed length. Moving forward, no more data is processed for that session. |
| We don't know how up-to-date each model is, so we have to scan each derived table                                 | **Record the max processed timestamp for each model.** We use an [incremental manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/#incremental-manifest) table to record the max processed timestamp of all derived tables.                                                                                                                                                                                         |

## Package State

Your Snowplow dbt package can be considered to be in one of 4 states at any given time:

1. First run of the package
2. New model introduced
3. Models out of sync
4. Standard run

This state determines the range of events to start with to process in the next run of the package (note that events earlier than this range may be processed as part of a complete session, as detailed above). We use the `min_last_success` and `max_last_success` for models enabled in the current run, as well as the number of models in the manifest, as the base for our calculation.

**Technical Details**

The identification of which state the package is in is computed by the `get_run_limits` macro which is called in the `snowplow_<package>_base_new_event_limits` model. This macro uses the metadata recorded in `snowplow_<package>_incremental_manifest` to determine the `lower_limit` and `upper_limit` for the run of the package. This limit is based on the calculations shown below, and is based on the field defined in your `snowplow__session_timestamp` variable.

To get the min and max last success, we run the following query:

```sql
select
    min(last_success) as min_last_success,
    max(last_success) as max_last_success,
    coalesce(count(*), 0) as models
from snowplow_<package>_incremental_manifest
where model in (array_of_snowplow_tagged_enabled_models)
```

Based on these the model is in one of the four states.

> **Tip:** In all states the `upper_limit` is limited by the `snowplow__backfill_limit_days` variable. This protects against back-fills with many rows causing very long run times.

### State 1: First run of the package

If there are no enabled models already in the manifest then we process from the start date up to the backfill limit or now, whichever is older:

**`lower_limit`**: `snowplow__start_date` **`upper_limit`**: `least(current_tstamp, snowplow__start_date + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    snowplow__start_date :crit, m1, 2020-01-01, 2m

    section Unprocessed
    Future Runs :crit, c, after a1, 2d

    section Processing
    snowplow__backfill_limit_days :active, a1, 2020-01-01, 1d

    section Processed
```

### State 2: New model introduced

If there are enabled models that aren't in the manifest table then a new model tagged with `snowplow_<package>_incremental` has been added since the last run; this can happen with a new custom model, or you have enabled some previously disabled custom modules. In this case the package will replay all previously processed events in order to back-fill the new model.

**`lower_limit`**: `snowplow__start_date` **`upper_limit`**: `least(max_last_success, snowplow__start_date + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    snowplow__start_date :crit, m1, 2020-01-01, 2m

    section Unprocessed
    Future Runs (new model) :crit, c, after a1 , 2d

    section Processing
    snowplow__backfill_limit_days :active, a1, 2020-01-01, 1d

    section Processed
    Other models :c, 2020-01-01, 3d
```

### State 3: Models out of sync

If the `min_last_success` is less than the `max_last_success` it means the tagged models are out of sync, for example due to a particular model failing to execute successfully during the previous run or as part of catching up on a new model. The package will attempt to sync all models as far as your backfill limit will allow.

**`lower_limit`**: `min_last_success - snowplow__lookback_window_hours` **`upper_limit`**: `least(max_last_success, min_last_success + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    min_last_success :crit, m1, 2020-01-02, 5m

    section Unprocessed
    Future Runs (out of sync model) :crit, c, after a2, 2d

    section Processing
    snowplow__lookback_window_hours :active, a1, 2020-01-01 18:00:00, 6h
    snowplow__backfill_limit_days :active, a2, after b1, 1d

    section Processed
    Out of sync model :b1, 2020-01-01, 1d
    Other models :b2, 2020-01-01, 4d
```

### State 4: Standard run

If none of the above criteria are met, then we consider it a 'standard run' where all models are in sync and we carry on from the last processed event.

**`lower_limit`**: `max_last_success - snowplow__lookback_window_hours` **`upper_limit`**: `least(current_tstamp, max_last_success + snowplow__backfill_limit_days)`

```mermaid
%%{init: { 'gantt': {'barHeight': 100, 'fontSize': 30, 'sectionFontSize': 30, 'leftPadding': 200, 'numberSectionStyles':3, 'displayMode': 'compact'}}}%%
gantt
    dateFormat  YYYY-MM-DD
    axisFormat
    tickInterval 10day
    section  
    max_last_success :crit, m1, 2020-01-04, 8m

    section Unprocessed
    Future Runs :crit, c, after a2 , 2d

    section Processing
    snowplow__lookback_window_hours :active, a1, 2020-01-03 18:00:00, 6h
    snowplow__backfill_limit_days :active, a2, after b1, 1d

    section Processed
    Previous Runs :b1, 2020-01-01, 3d
```

> **Note:** Remember, this upper and lower limit is only used to identify which events are considered to get the session identifiers for the run, additional events outside this date range may be included to ensure that full sessions are processed. See [late arriving data](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/) for more information.

### How to identify the current state

If you want to check the current state of a model, run the `snowplow_<package>_base_new_event_limits` model. This will log the current state to the CLI while causing no disruption to the incremental processing of events. You will also see this in the logs of any incremental run.

```bash
dbt run --select snowplow_<package>_base_new_event_limits
...
00:26:29 + Snowplow: Standard incremental run
00:26:29 + Snowplow: Processing data between 2021-01-05 17:59:32 and 2021-01-07 23:59:32
```

- State 1: `Snowplow: No data in manifest. Processing data from start_date`
- State 2: `Snowplow: New Snowplow incremental model. Backfilling`
- State 3: `Snowplow: Snowplow incremental models out of sync. Syncing`
- State 4: `Snowplow: Standard incremental run`

> **Warning:** In all states, although much more likely in state 1, it is possible that a run finds no incremental data to process. In this case you will see a warning in the logs of the form `Snowplow Warning: No data in <table> for date range from variables, please modify your run variables to include data if this is not expected.`. In this case you should increase your start date if no runs have completed, or increase your backfill limit days if there is a gap in your data before the next records.

In reality, the events in your [events this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#events-this-run) table will have a wider range of timestamps than those calculated here. For more details see the [full run calculation](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/full-run-calculation/) page.

---

# Snowplow dbt package mechanics
> Key mechanics of our packages
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/

The following pages contain information relating to the core logic and processing of our packages, it is not required to understand these concepts to make use of our packages but they are provided for more advanced users who may wish to produce more complex [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) or better understand what's going on under the hood of the packages. We refer to these elements throughout the docs.

> **Tip:** Throughout the packages and these docs, we will refer to a `session`, or `session_identifier`. This is the atomic unit of re-processing, as described in the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) and by default usually refers to the actual session identifier within your tracker; however this language is in part only used for historic reasons. Thanks to the [custom identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) in our packages, these _sessions_ can refer to any identifier you define and our packages will ensure that all events (within the limits) are processed in a single run. A similar case applies to user identifiers as well.

---

# Late arriving data
> A detailed overview of how the data flows and how to manage variables for handling edge cases.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/late-arriving-data/

```mermaid
graph LR
    style A color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style B color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style C color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;
    style D color:#724512, fill:#FEEEBD, stroke:#724512, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#FEEEBD


    A[event created] -->B[event sent to Snowplow]
    B[event sent to Snowplow] --> C[event received by collector]
    C[event received by collector] --> D[event loaded to events table]
```

Normally, there is very little delay between each step of the dataflow, dependent on your pipeline setup and other environmental conditions.

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|millisecond| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |millisecond|D[load_tstamp]
```

However there can be times when data arrives "late", which can mean different things depending on where the delay in the flow of data occurs. Here we cover the 2 most common.

### Late Sent Events

The most common scenario is that the device the event is generated on goes offline, leading to a larger than usual gap between the `dvce_created_tstamp` and the `dvce_sent_tstamp`:

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|HOURS| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |millisecond|D[load_tstamp]
```

In our packages, processing this type of data is managed by the `snowplow__days_late_allowed` variable. Any event with a delay greater than this value between being created and being sent will not be processed. However, assuming the data lands within this limit the whole session will be reprocessed to include this event.

### Late Loaded Events

The other most common case is that of late loaded events, where the gap is between the collector and load timestamps. This may be due to batch loading, or issues with your pipeline.

```mermaid
graph LR
    style A color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style B color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style C color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;
    style D color:#3F2874, fill:#F0EBF8, stroke:#3F2874, stroke-width:1px;

    linkStyle default stroke-width:px,fill:#633EB5,stroke:#633EB5,color:#633EB5


    A[dvce_created_tstamp] -->|millisecond| B[dvce_sent_tstamp]
    B[dvce_sent_tstamp] --> |millisecond|C[collector_tstamp]
    C[collector_tstamp] --> |HOURS|D[load_tstamp]
```

Because by default our packages uses `collector_tstamp` to identify new data, this can mean that these events are not picked up (and therefore the sessions not reprocessed to include them). If you are using `load_tstamp` as your `snowplow__session_timestamp` then this is not an issue.

To manage this we use the `snowplow__lookback_window_hours` variable to add a _buffer_ to the previous maximum processed timestamp to account for some events being delayed. In the case of batch loading, you should ensure this window is at least a few hours greater than your batch period.

Note that this isn't an issue if _all_ events are delayed, because if all events are delayed the previous maximum processed timestamp would still be before any of these events. In the event of the model running part way through the loading of a set of out-of-order events (e.g. an event recovery), you can increase the `snowplow__lookback_window_hours` to forcibly reprocess a larger window of data.

For more details on how all these variables combine, please see the docs on the [incremental logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) of our packages.

---

# Manifest tables
> Details around the manifest tables we use in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/

Each of our packages has a set of manifest tables that manage the [Incremental Sessionization Logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/) logic of our package, as well as quarantining long running sessions.

> **Danger:** These manifest tables are critical to the package **and as such are protected from full refreshes, i.e. being dropped, when running in production by default**. In development refreshes are enabled.

The `allow_refresh()` macro defines the protection behavior. As [dbt recommends](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles#understanding-targets-in-profiles), target names are used here to differentiate between your prod and dev environment. By default, this macro assumes your dev target is named `dev`. This can be changed by setting the `snowplow__dev_target_name` var in your `dbt_project.yml` file.

To full refresh any of the manifest models in production as part of a `--full-refresh`, set the `snowplow__allow_refresh` to `true` at run time.

Alternatively, you can amend the behavior of this macro entirely by overwriting it. See the [Overwriting Macros](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/) section for more details.

## Incremental manifest

The majority of our packages have an incremental manifest table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_incremental_manifest`. This table exists to track the [state](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/#how-to-identify-the-current-state) of each of the models in the package, including any [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) that have been tagged.

This table has 2 columns:

- `model`: The name of the model
- `last_success`: The timestamp of the max last event processed by the model, based on the field defined in the `snowplow__session_timestamp` variable

As this is the source of truth for processing for the package, it is highly recommended to never alter or edit this table directly as this can lead to unexpected outcomes. The manifest table is updated as part of an `on-run-end` hook, which calls the `snowplow_incremental_post_hook()` macro.

## Sessions lifecycle manifest

The majority of our packages have a session lifecycle manifest table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_base_sessions_lifecycle_manifest`. This table exists to track the start and end timestamps of any given session, allowing us to avoid a full table scan each run.

This table has 4 columns:

- `session_identifier`: The unique identifier for the session
- `user_identifier`: The unique user identifier for the session. Note that if multiple user identifiers exist for a given session we only take one.
- `start_tstamp`: The timestamp of the first event to process for the session (note due to late arriving data, this may not actually be the true first event in rare cases)
- `end_tstamp`: The timestamp of the last event to process for the session (note this may be capped with the `snowplow__max_session_days` variable)

Timestamps based on the field defined in the `snowplow__session_timestamp` variable. This table is used to calculate which sessions are to be processed in a run, and also helps inform the run limits.

## Quarantine table

Many of our packages have a quarantine table; by default this is in your `_snowplow_manifest` suffixed schema, and will have the name `snowplow_<package_name>_base_quarantined_sessions`. This table exists to track sessions that have gone beyond the `snowplow__max_session_days` limit and avoid re-processing them again.

This table has 1 column:

- `session_identifier`: The unique identifier for the session to not process in any further run.

This table is updated on the post-hook of the [base sessions this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/#base-sessions-this-run) table. If there are any additional sessions you identify you want to remove from processing, you can manually add them to this table but ensure they do not already exist.

---

# Snowplow optimized materialization
> Details on our optimized version of the dbt incremental materialization and how to enable it.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/optimized-upserts/

> **Tip:** This functionality requires dispatching to our macros over dbt core, see how to do this on the [dispatch setup](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/dispatch/) page.

Our packages make use of the standard dbt `incremental` [materialization](https://docs.getdbt.com/docs/build/materializations#incremental) with an optimization applied for incremental models. The advantage is that it limits table scan on the target table when updating/inserting based on the new data. This improves performance and reduces cost. We do this by overriding the macro that generates the sql for the `merge` and `insert_delete` incremental methods.

All other features of the `incremental` materialization are supported including `incremental_predicates` and `on_schema_change`. The code for the overridden macro can be found [on GitHub](https://github.com/snowplow/dbt-snowplow-utils/blob/main/macros/materializations/base_incremental/common/get_merge_sql.sql).

## Usage

### Controlling the buffer size

We calculate the upper and lower limit from the source table (usually a [this run](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/) table) before adding a buffer to the lower limit to cover late arriving data or other issues with specific timestamps. This buffer is controlled by the `snowplow__upsert_lookback_days` variable and usually has a default of 30.

To disable this buffer entirely, you can either set `snowplow__upsert_lookback_days` to 0 or you can `disable_upsert_lookback` to `true` in your \*\* model config\*\* if you want to do this for a specific model. **Note this is a [model config](https://docs.getdbt.com/reference/model-configs), not a variable.**

### Adding to your own models

All our incremental models have this functionality enabled by default, but if you wish to use it in a [custom model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) you you need to ensure a `unique_key` and `upsert_date_key` are provided in the model config, and that `snowplow_optimize=true` in the config as well, e.g.

```jinja2
{{
  config(
    materialized='incremental',
    unique_key='page_view_id',
    upsert_date_key='start_tstamp',
    snowplow_optimize = true,
    ...
  )
}}
```

## Disabling the optimization

To disable the optimized upsert entirely and use the default incremental materialization, set the `snowplow_optimize` to `false` in your model config. **Note this is a [model config](https://docs.getdbt.com/reference/model-configs), not a variable.**

---

# Overridable macros in Snowplow dbt packages
> Override and customize default macros in Snowplow dbt packages to extend package functionality.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/overridable-macros/

Many of our packages are built using [macros](https://docs.getdbt.com/docs/build/jinja-macros) to allow easier support of multiple warehouses. Some of these macros are designed to be overridable to give an easy route to customization for the user. You can find a list of the overridable macros for each package in the [package details](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) pages.

The easiest way to override any given macro is to create a `default__` version within your dbt project, for example, to change the `filter_bots` macro in the `unified` package you would have:

```jinja2
{% macro default__filter_bots(table_alias = none) %}
    and {% if table_alias %}{{table_alias~'.'}}{% endif %}useragent not similar to '%(YOUR_CUSTOM_PATTERN|bot|crawl|slurp|spider|archiv|spinn|sniff|seo|audit|survey|pingdom|worm|capture|(browser|screen)shots|analyz|index|thumb|check|facebook|PingdomBot|PhantomJS|YandexBot|Twitterbot|a_archiver|facebookexternalhit|Bingbot|BingPreview|Googlebot|Baiduspider|360(Spider|User-agent)|semalt)%'
{% endmacro %}
```

> **Tip:** Make sure to check the source code for the macro to understand what arguments are required and make sure you start from the version applicable to your warehouse.

---

# This run tables
> Details about the this run table approach use in our packages.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/this-run-tables/

Our packages make use of a series of `this run` tables, named such because they are dropped and recreated each `dbt run` to only contain relevant information that that run. These tables allow for efficient re-use of information throughout a single run, and also allow you to debug any issues should a run fail.

## Events this run

As described in the [incremental sessionization logic](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/), the first thing we do in any given run is identify what events need to be processed as part of that run. These events are [deduplicated](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/deduplication/), have relevant [entities and SDEs](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/modeling-entities/) extracted and attached, and the appropriate [identifiers](/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-identifiers/) are calculated.

This table of events is then used for all further processing in the package, we refer to this table as `events this run`. This is the table to use in the case of building most [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) that require the event level data.

In all cases where this table exists, it is in the `models/base/scratch/` directory within the package.

| Package     | Events This Run Name                              |
| ----------- | ------------------------------------------------- |
| unified     | `snowplow_unified_events_this_run`                |
| ecommerce   | `snowplow_ecommerce_base_events_this_run`         |
| media       | `snowplow_media_player_base_events_this_run`      |
| normalize   | `snowplow_normalize_base_events_this_run`         |
| attribution | None, this package does not build from raw events |

> **Tip:** Unified also contains a `snowplow_unified_base_events_this_run`, however this builds into `snowplow_unified_events_this_run` so we recommend you use the more complete table.

## Base sessions this run

The sessions this run table is built before the events this run table, and contains a list of all all the sessions that are being processed in the current run. It is a modified sub-set of the [sessions lifecycle manifest](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/manifest-tables/) with only the sessions within the run limits, and with the end timestamp capped to the upper limit of the run (in case of historic re-processing).

In all packages that contain this table, it named of the form `snowplow_<package_name>_base_sessions_this_run` and is located in the `models/base/scratch` directory within the package.

There should be no need to directly interact with this table but may be useful debugging any errors.

## Other this run tables

Our packages are built in a way that makes it easy to make small alterations and additions to the derived models. Most derived models, such as `sessions`, or `views` will have a corresponding this run table. The actual incremental derived model itself is just a `select *` from this table (with a date field added for Databricks). This becomes useful in [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) and is also helpful to run tests on just the incremental data or to identify any issues. We recommend using the output of these where possible, but not directly overwriting them as this may put you out of sync with any updates we release. See the [custom models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) docs for more information about how best to use these tables.

The usual folder structure, although this can vary a bit by package is that these this run tables tend to sit at either `models/<module>/scratch/snowplow_<package_name>_<module>_this_run` or `models/<module>/scratch/<warehouse>/snowplow_<package_name>_<module>_this_run` when there are different versions of the model per warehouse.

---

# Model your data with SQL Runner (legacy)
> Guides for setting up the legacy SQL Runner and running our web and mobile models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/

> **Tip:** For any new developments we highly recommend using our [dbt packages](/docs/modeling-your-data/modeling-your-data-with-dbt/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

[SQL Runner](https://github.com/snowplow/sql-runner) enables you to execute SQL scripts against the Snowplow data in your data warehouse. Specifically, it allows you to organize your SQL scripts in templatable playbooks, and execute them in series or in parallel on Snowflake, Amazon Redshift, GCP BigQuery and PostgreSQL.

A SQL Runner data model consists of:

- SQL files (containing one or more SQL statements)
- Playbooks (YAML files organizing the SQL into steps)

### Available models

| Model                                                                                               | Redshift | BigQuery | Snowflake |
| --------------------------------------------------------------------------------------------------- | -------- | -------- | --------- |
| [Web](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/)       | 1.3.1    | 1.0.4    | 1.0.2     |
| [Mobile](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/) | 1.1.0    | 1.1.0    | 1.1.0     |

### Playbooks

A playbook consists of one of more _steps_, each of which consists of one or more _queries_. Steps are run in series, queries are run in parallel within the step.

Each query contains the path to a _query file_. 

All steps are applied against all _targets_. All targets are processed in parallel.

In the following example, a.sql, b.sql and c.sql are run in parallel.

```yaml
:steps:
  - :name: "Run a,b and c in parallel"
    :queries:
      - :name: a
        :file: a.sql
      - :name: b
        :file: b.sql
      - :name: c
        :file: c.sql
```

By contrast, in the example below, the three SQL files are executed in sequence.

```yaml
:steps:
  - :name: "Run a..."
    :queries:
      - :name: a
        :file: a.sql
  - :name: "...then run b..."
    :queries:
      - :name: b
        :file: b.sql
  - :name: "...then run c..."
    :queries:
      - :name: c
        :file: c.sql
```

Playbooks can be templated, and corresponding variables can be passed in with the var flag like this:

```bash
sql-runner -var host=value,username=value2,password=value3
```

Here is the corresponding playbook template:

```yaml
:targets:
- :name: "My Postgres database 1"
  :type: postgres
  :host: {{.host}}
  :database: sql_runner_tests_1
  :port: 5432
  :username: {{.username}}
  :password: {{.password}}
  :ssl: false # SSL disabled by default
:variables:
  :test_schema: sql_runner_tests
  :timeFormat: "2006_01_02"
:steps:
- :name: Create schema and table
  :queries:
    - :name: Create schema and table
      :file: postgres-sql/good/1.sql
      :template: true
```

### SQL files

A query file contains one or more SQL statements. These are executed "raw" (i.e. not in a transaction) in series by SQL Runner. If the query file is flagged as a _template_ in the playbook, then the file is pre-processed as a template before being executed.

**Note**: If your query is a template that requires pre-processing, you must add `template: true` to the query definition in the playbook yml file, for example:

```yaml
:name: "Run a.."
    :queries:
      - :name: a
        :file: a.sql
        :template: true
```

### Templates

Templates are run through Golang's [text template processor](https://pkg.go.dev/text/template). The template processor can access all _variables_ defined in the playbook.

The following custom functions are also supported:

- `nowWithFormat [timeFormat]`: where `timeFormat` is a valid Golang [time format](http://golang.org/pkg/time/#Time.Format)
- `systemEnv "ENV_VAR"`: where `ENV_VAR` is a key for a valid environment variable
- `awsEnvCredentials`: supports passing credentials through environment variables, such as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
- `awsProfileCredentials`: supports getting credentials from a credentials file, also used by boto/awscli
- `awsEC2RoleCredentials`: supports getting role-based credentials, i.e. getting the automatically generated credentials in EC2 instances
- `awsChainCredentials`: tries to get credentials from each of the three methods above in order, using the first one returned
- `randomInt`: will return a random integer

**Note**: All AWS functions output strings in the Redshift credentials format (`CREDENTIALS 'aws_access_key_id=%s;aws_secret_access_key=%s'`).

For an example query file using templating see: [integration/resources/postgres-sql/good/3.sql](https://github.com/snowplow/sql-runner/blob/master/integration/resources/postgres-sql/good/3.sql)

### Failure modes

If a statement fails in a query file, the query will terminate and report failure.

If a query fails, its sibling queries will continue running, but no further steps will run.

Failures in one target do not affect other targets in any way.

### Return codes

```text
- 0 for no errors
- 5 for target initialization errors
- 6 for query errors
- 7 for both types of error
- 8 for no queries run
```

### Target configuration

#### Redshift

If your storage target is Amazon Redshift, then the target configuration in the playbook is:

```yaml
targets:
  - name: "My Redshift database"
    type: redshift
    host: ADD HERE # The endpoint as shown in the Redshift console
    database: ADD HERE # Name of database
    port: 5439 # Default Redshift port
    username: ADD HERE
    password: ADD HERE
    ssl: false # SSL disabled by default
variables:
  ...
```

#### BigQuery

To access a BigQuery project, sql-runner will need some Google credentials. These can be set up by creating a new service account in the GCP console, then providing its private key to the application via a GOOGLE\_APPLICATION\_CREDENTIALS environment variable - a detailed walkthrough of this process is available on the [GCP documentation website](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually).

After the credentials are set up, simply create a playbook with the following BigQuery-specific target configuration:

```yaml
targets:
  - name: "My BigQuery database"
    type: bigquery
    project: ADD HERE # Project ID as shown in the GCP console's front page
variables:
    ...
```

#### Snowflake

If your data warehouse is Snowflake, then the SQL Runner playbooks will have a target configuration as:

```yaml
targets:
  - name: "My Snowflake database"
    type: snowflake
    account: ADD HERE # Your Snowflake account name
    database: ADD HERE # Name of database
    warehouse: ADD HERE # Name of warehouse to run the queries
    username: ADD HERE
    password: ADD HERE
    host: # Leave blank
    port: # Leave blank
    ssl: true # Snowflake connection is always secured by TLS
    query_tag: ADD HERE # optional, available since v0.10.0
variables:
    ...
```

The `query_tag` parameter sets the `QUERY_TAG` [session parameter](https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag) in Snowflake. When set, it will be applied to all queries included in the playbook.

#### PostgreSQL

Finally, if your storage target is PostgreSQL, then can be configured as:

```yaml
targets:
  - name: "My Postgres database"
    type: postgres
    host: ADD HERE
    database: ADD HERE # Name of database
    port: 5432 # Default Postgres port
    username: ADD HERE
    password: ADD HERE
    ssl: false # SSL disabled by default
variables:
```

That's it - you're now ready to start running SQL against your data warehouse!

## User guide

SQL Runner is a zero-dependency binary and can be found as a [release asset](https://github.com/snowplow/sql-runner/releases/latest) for:

- [macOS](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_darwin_amd64.zip)
- [Linux](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_linux_amd64.zip)
- [Windows](https://github.com/snowplow/sql-runner/releases/download/0.10.1/sql_runner_0.10.1_windows_amd64.zip)

### CLI Arguments

```text
./sql-runner --help
sql-runner version: 0.10.1
Run playbooks of SQL scripts in series and parallel on Redshift, Postgres, BigQuery and Snowflake
Usage:
-checkLock string
  	Checks whether the lockfile already exists
-consul string
  	The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
  	Will read playbooks locally, but use Consul for locking.
-deleteLock string
  	Will attempt to delete a lockfile if it exists
-dryRun
  	Runs through a playbook without executing any of the SQL
-fillTemplates
  	Will print all queries after templates are filled
-fromStep string
  	Starts from a given step defined in your playbook
-help
  	Shows this message
-lock string
  	Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
  	Playbook of SQL scripts to execute
-runQuery string
  	Will run a single query in the playbook
-showQueryOutput
  	Will print all output from queries
-softLock string
  	Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
  	Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
  	Variables to be passed to the playbook, in the key=value format
-version
  	Shows the program version
```

#### More on Consul

Using the `-consul` argument results in the following changes:

- The `-playbook` argument becomes the key that is used to look for the playbook in Consul.
- The `-sqlroot` argument also becomes a key argument for Consul.
- The `-lock` argument creates a lock as a Consul key value pair
- The `-softLock` argument creates a lock as a Consul key value pair
- The `-checkLock` argument searches in Consul for a lock
- The `-deleteLock` argument searches in Consul for a lock

If you pass in the default:

- `./sql-runner -consul "localhost:8500" -playbook "sql-runner/playbook/1"`

This results in:

- Looking for your playbook file at this key `sql-runner/playbook/1`
- Expecting all your SQL file keys to begin with `sql-runner/playbook/<SQL path from playbook>`

However as the key here can be used as a both a data and folder node we have added a new sqlroot option:

- `./sql-runner -consul "localhost:8500" -playbook "sql-runner/playbook/1" -sqlroot PLAYBOOK_CHILD`

This results in:

- Looking for your playbook file at this key `sql-runner/playbook/1`
- Expecting all your SQL file keys to begin with `sql-runner/playbook/1/<SQL path from playbook>`
  - The data node is used as a folder node as well.

---

# Migrating from SQL Runner to dbt
> SQL Runner is no longer actively developed and users should try to migrate to dbt where possible, this guide helps you do that.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/migrating-to-dbt/

> **Info:** This guide assumes you are running the standard web and/or mobile SQL Runner models, if you have built any custom models or customized the standard models in any way, you will need to make these same changes to the dbt models. See our section on [custom dbt models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/) for more information on how to do this.

## Why Migrate?

SQL Runner is currently in maintenance mode, while we will continue to fix bugs when they are identified, we are not actively developing the tool or the models anymore and at some point in the future may deprecate it entirely. Our dbt models on the other hand are under active development, with new features and optimizations being made regularly. It is also a more widely used tool, meaning installation and management is far easier (or you can use tools like dbt Cloud, or our CDI customers can run dbt models the same way you can SQL Runner). We also have a far wider range of packages available in dbt including e-commerce, marketing attribution, and a package to normalize your Snowplow data.

In dbt we also support Databricks and Postgres warehouses in addition to Snowflake, BigQuery, and Redshift. Newer tracking plugins are only being modeled within our dbt packages.

In general, if you are happy with SQL Runner and don't foresee a need to add more models in the future then there is no need to migrate; however if you are starting from scratch, or would like to make use of our wider range of models, then you should consider migrating to dbt.

## Differences between the tools

The core of the web and mobile models (e.g. page/screen views, sessions, and users) are the same across SQL Runner and dbt, however the dbt versions contain some additional fields which may be useful for analysis (such as conversions, event counts, and human-readable language information in the web package), as well as additional optional modules for web such as consent and core web vitals. Below this the logic used to process the data is roughly the same, although we have made some optimizations and added more flexibility in how this processing is done in dbt.

We recommend you take a look at the [docs](/docs/modeling-your-data/modeling-your-data-with-dbt/) for our dbt packages to get a better understanding of how they work and how you can use them going forward.

## Pre-requisites

We assume that you have dbt [installed](https://docs.getdbt.com/docs/core/installation), a working connection, and some basic understanding of using dbt including installing packages and running models.

## Mapping the variables

While the variables for your SQL Runner models are spread throughout the files, in dbt all variables are in your `dbt_project.yml` file. We have mostly been consistent between the two tools, with the dbt variables being prefixed by `snowplow__`, but some have new names. The table below maps each SQL Runner variable to the equivalent dbt variable, but there are many more you can set to customize how the models run - you can read about these in the relevant [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/) page.

> **Warning:** When using multiple dbt packages you must be careful to specify which scope a variable or configuration is defined within. In general, always specify each value in your `dbt_project.yml` nested under the specific package e.g.
>
> ```yml
> vars:
>   snowplow_web:
>     snowplow__atomic_schema: schema_with_snowplow_web_events
>   snowplow_mobile:
>     snowplow__atomic_schema: schema_with_snowplow_mobile_events
> ```
>
> You can read more about variable scoping in dbt's docs around [variable precedence](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#variable-precedence).

> Values in bold have a different name instead of just the prefix

| SQL Runner Variable                    | dbt variable                                                                                            |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **`app_errors`**                       | **`snowplow__enable_app_errors_module`**                                                                |
| **`app_id_filters`**                   | **`snowplow__app_id`**                                                                                  |
| **`application_context`**              | **`snowplow__enable_application_context`**                                                              |
| `cleanup_mode`                         | No equivalent variable _(closest is `snowplow__allow_refresh` combined with dbt `--full-refresh` flag)_ |
| `cluster_by`                           | No equivalent variable _(Clustering defined in model)_                                                  |
| `days_late_allowed`                    | `snowplow__days_late_allowed`                                                                           |
| `derived_tstamp_partitioned`           | `snowplow__derived_tstamp_partitioned`                                                                  |
| **`enabled`** (Mobile app errors only) | **`snowplow__enable_app_errors_module`**                                                                |
| `ends_run`                             | No equivalent variable                                                                                  |
| `entropy`                              | No equivalent variable                                                                                  |
| **`geolocation_context`**              | **`snowplow__enable_geolocation_context`**                                                              |
| `heartbeat`                            | `snowplow__heartbeat`                                                                                   |
| **`iab`**                              | **`snowplow__enable_iab`**                                                                              |
| **`input_schema`**                     | **`snowplow__atomic_schema`**                                                                           |
| `lookback_window_hours`                | `snowplow__lookback_window_hours`                                                                       |
| **`minimumVisitLength`**               | **`snowplow_min_visit_length`**                                                                         |
| **`mobile_context`**                   | **`snowplow__enable_mobile_context`**                                                                   |
| **`model_version`**                    | No equivalent variable                                                                                  |
| `output_schema`                        | Set in `models` part of project file, see relevant configuration page for more info.                    |
| **`platform_filters`**                 | **`snowplow__platform`**                                                                                |
| `scratch_schema`                       | Set in `models` part of project file, see relevant configuration page for more info.                    |
| **`screen_context`**                   | **`snowplow__enable_screen_contextt`**                                                                  |
| `session_lookback_days`                | `snowplow__session_lookback_days` _(default increased to 730)_                                          |
| `skip_derived`                         | No equivalent variable _(use dbt `--select` flag)_                                                      |
| `stage_next`                           | No equivalent variable                                                                                  |
| `start_date`                           | `snowplow__start_date`                                                                                  |
| `ua_bot_filter`                        | `snowplow__ua_bot_filter`                                                                               |
| **`ua_parser`**                        | **`snowplow__enable_ua`**                                                                               |
| **`update_cadence_days`**              | **`snowplow__backfill_limit_days`** _(default increased to 30)_                                         |
| `upsert_lookback_days`                 | `snowplow__upsert_lookback_days`                                                                        |
| **`yauaa`**                            | **`snowplow__enable_yauaa`**                                                                            |

## Setting up and running our dbt packages

The latest information for our packages can be found in the [quickstart](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-quickstart/) section of our docs. It is highly recommended you just start running your dbt project from scratch and process all data from the start date to ensure the package works as intended and all your data is processed in the same way.

## Migrate existing derived data

> **Danger:** The dbt package uses slightly different logic for processing, including the quarantining of sessions and different format manifest tables. It is highly recommended that you just run the dbt project from your start date. The following is a best-effort suggestion and we make no guarantee that all data will be correctly processed or that issues may not happen later in the lifetime of the project.
>
> This method will also not correctly populate the user stitching table or process user stitching for historic data.

There may be cases where running the dbt models from scratch is not a viable option for you, in this case it is possible to migrate your existing derived SQL Runner data into the derived tables produced by dbt, however this will result in your data being generated from two slightly differing logics.

It is advisable to produce your dbt tables into new schemas where possible, even though the derived tables should have different names; this will help keep your data separate and ensure that as we go through the following steps that dbt does not overwrite your SQL Runner tables.

> **Note:** Postgres only supports the `MERGE` statement in version 15 and up, if you are using an older version you will need to alter the commands to be a `DELETE`+`INSERT` instead.

### Create dbt tables by doing a recent-dated run

Because of the difference in manifest tables and incremental logic between SQL Runner and dbt models it makes sense to first create the dbt tables and then insert your existing data into them, rather than try and create the dbt tables directly from your SQL Runner data.

Once you have your dbt project and variables set up, change your `snowplow__start_date` to a recent date, say 7 days before the end of your last SQL Runner processed date, and run the project once. This will produce all the dbt tables including the manifest tables needed to manage the incremental logic, and ensure a good overlap between the end of your SQL Runner processing and the start of dbt processing.

### Merge your existing data into the new tables (web)

The following SQL will merge the existing web records in your SQL Runner derived tables into the new dbt derived tables, please run each one as required. This approach will leave any columns that only exist in dbt `null`. Please ensure you replace the schema and table names with your ones where appropriate.

**If you are using Redshift, be sure to `commit` your changes.**

> **Warning:** It is possible, particularly for columns which may have been null, that the types of columns across the two tables don't entirely match. Your warehouse may manage this for you, or you may have to use a `cast(col_name as new_type)` in place of just selecting the column based on any error message you receive.

#### Page Views

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_page_views t
USING <SQL_RUNNER_DERIVED_SCHEMA>.page_views s
ON T.page_view_id = s.page_view_id
WHEN NOT MATCHED THEN
INSERT
(
    page_view_id,
    event_id,
    app_id,
    user_id,
    domain_userid,
    network_userid,
    domain_sessionid,
    domain_sessionidx,
    page_view_in_session_index,
    page_views_in_session,
    dvce_created_tstamp,
    collector_tstamp,
    derived_tstamp,
    start_tstamp,
    end_tstamp,
    engaged_time_in_s,
    absolute_time_in_s,
    horizontal_pixels_scrolled,
    vertical_pixels_scrolled,
    horizontal_percentage_scrolled,
    vertical_percentage_scrolled,
    doc_width,
    doc_height,
    page_title,
    page_url,
    page_urlscheme,
    page_urlhost,
    page_urlpath,
    page_urlquery,
    page_urlfragment,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network,
    page_referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    geo_country,
    geo_region,
    geo_region_name,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_timezone,
    user_ipaddress,
    useragent,
    br_lang,
    br_viewwidth,
    br_viewheight,
    br_colordepth,
    br_renderengine,
    os_timezone,
    category,
    primary_impact,
    reason,
    spider_or_robot,
    useragent_family,
    useragent_major,
    useragent_minor,
    useragent_patch,
    useragent_version,
    os_family,
    os_major,
    os_minor,
    os_patch,
    os_patch_minor,
    os_version,
    device_family,
    device_class,
    agent_class,
    agent_name,
    agent_name_version,
    agent_name_version_major,
    agent_version,
    agent_version_major,
    device_brand,
    device_name,
    device_version,
    layout_engine_class,
    layout_engine_name,
    layout_engine_name_version,
    layout_engine_name_version_major,
    layout_engine_version,
    layout_engine_version_major,
    operating_system_class,
    operating_system_name,
    operating_system_name_version,
    operating_system_version
)
VALUES
(
    s.page_view_id,
    s.event_id,
    s.app_id,
    s.user_id,
    s.domain_userid,
    s.network_userid,
    s.domain_sessionid,
    s.domain_sessionidx,
    s.page_view_in_session_index,
    s.page_views_in_session,
    s.dvce_created_tstamp,
    s.collector_tstamp,
    s.derived_tstamp,
    s.start_tstamp,
    s.end_tstamp,
    s.engaged_time_in_s,
    s.absolute_time_in_s,
    s.horizontal_pixels_scrolled,
    s.vertical_pixels_scrolled,
    s.horizontal_percentage_scrolled,
    s.vertical_percentage_scrolled,
    s.doc_width,
    s.doc_height,
    s.page_title,
    s.page_url,
    s.page_urlscheme,
    s.page_urlhost,
    s.page_urlpath,
    s.page_urlquery,
    s.page_urlfragment,
    s.mkt_medium,
    s.mkt_source,
    s.mkt_term,
    s.mkt_content,
    s.mkt_campaign,
    s.mkt_clickid,
    s.mkt_network,
    s.page_referrer,
    s.refr_urlscheme,
    s.refr_urlhost,
    s.refr_urlpath,
    s.refr_urlquery,
    s.refr_urlfragment,
    s.refr_medium,
    s.refr_source,
    s.refr_term,
    s.geo_country,
    s.geo_region,
    s.geo_region_name,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_timezone,
    s.user_ipaddress,
    s.useragent,
    s.br_lang,
    s.br_viewwidth,
    s.br_viewheight,
    s.br_colordepth,
    s.br_renderengine,
    s.os_timezone,
    s.category,
    s.primary_impact,
    s.reason,
    s.spider_or_robot,
    s.useragent_family,
    s.useragent_major,
    s.useragent_minor,
    s.useragent_patch,
    s.useragent_version,
    s.os_family,
    s.os_major,
    s.os_minor,
    s.os_patch,
    s.os_patch_minor,
    s.os_version,
    s.device_family,
    s.device_class,
    s.agent_class,
    s.agent_name,
    s.agent_name_version,
    s.agent_name_version_major,
    s.agent_version,
    s.agent_version_major,
    s.device_brand,
    s.device_name,
    s.device_version,
    s.layout_engine_class,
    s.layout_engine_name,
    s.layout_engine_name_version,
    s.layout_engine_name_version_major,
    s.layout_engine_version,
    s.layout_engine_version_major,
    s.operating_system_class,
    s.operating_system_name,
    s.operating_system_name_version,
    s.operating_system_version
);
```

#### Sessions

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_sessions t
USING <SQL_RUNNER_DERIVED_SCHEMA>.sessions s
ON T.domain_sessionid = s.domain_sessionid
WHEN NOT MATCHED THEN
INSERT
(
    app_id,
    domain_sessionid,
    domain_sessionidx,
    start_tstamp,
    end_tstamp,
    user_id,
    domain_userid,
    network_userid,
    page_views,
    engaged_time_in_s,
    absolute_time_in_s,
    first_page_title,
    first_page_url,
    first_page_urlscheme,
    first_page_urlhost,
    first_page_urlpath,
    first_page_urlquery,
    first_page_urlfragment,
    last_page_title,
    last_page_url,
    last_page_urlscheme,
    last_page_urlhost,
    last_page_urlpath,
    last_page_urlquery,
    last_page_urlfragment,
    referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network,
    geo_country,
    geo_region,
    geo_region_name,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_timezone,
    user_ipaddress,
    useragent,
    br_renderengine,
    br_lang,
    os_timezone,
    category,
    primary_impact,
    reason,
    spider_or_robot,
    useragent_family,
    useragent_major,
    useragent_minor,
    useragent_patch,
    useragent_version,
    os_family,
    os_major,
    os_minor,
    os_patch,
    os_patch_minor,
    os_version,
    device_family,
    device_class,
    agent_class,
    agent_name,
    agent_name_version,
    agent_name_version_major,
    agent_version,
    agent_version_major,
    device_brand,
    device_name,
    device_version,
    layout_engine_class,
    layout_engine_name,
    layout_engine_name_version,
    layout_engine_name_version_major,
    layout_engine_version,
    layout_engine_version_major,
    operating_system_class,
    operating_system_name,
    operating_system_name_version,
    operating_system_version
)
VALUES
(
    s.app_id,
    s.domain_sessionid,
    s.domain_sessionidx,
    s.start_tstamp,
    s.end_tstamp,
    s.user_id,
    s.domain_userid,
    s.network_userid,
    s.page_views,
    s.engaged_time_in_s,
    s.absolute_time_in_s,
    s.first_page_title,
    s.first_page_url,
    s.first_page_urlscheme,
    s.first_page_urlhost,
    s.first_page_urlpath,
    s.first_page_urlquery,
    s.first_page_urlfragment,
    s.last_page_title,
    s.last_page_url,
    s.last_page_urlscheme,
    s.last_page_urlhost,
    s.last_page_urlpath,
    s.last_page_urlquery,
    s.last_page_urlfragment,
    s.referrer,
    s.refr_urlscheme,
    s.refr_urlhost,
    s.refr_urlpath,
    s.refr_urlquery,
    s.refr_urlfragment,
    s.refr_medium,
    s.refr_source,
    s.refr_term,
    s.mkt_medium,
    s.mkt_source,
    s.mkt_term,
    s.mkt_content,
    s.mkt_campaign,
    s.mkt_clickid,
    s.mkt_network,
    s.geo_country,
    s.geo_region,
    s.geo_region_name,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_timezone,
    s.user_ipaddress,
    s.useragent,
    s.br_renderengine,
    s.br_lang,
    s.os_timezone,
    s.category,
    s.primary_impact,
    s.reason,
    s.spider_or_robot,
    s.useragent_family,
    s.useragent_major,
    s.useragent_minor,
    s.useragent_patch,
    s.useragent_version,
    s.os_family,
    s.os_major,
    s.os_minor,
    s.os_patch,
    s.os_patch_minor,
    s.os_version,
    s.device_family,
    s.device_class,
    s.agent_class,
    s.agent_name,
    s.agent_name_version,
    s.agent_name_version_major,
    s.agent_version,
    s.agent_version_major,
    s.device_brand,
    s.device_name,
    s.device_version,
    s.layout_engine_class,
    s.layout_engine_name,
    s.layout_engine_name_version,
    s.layout_engine_name_version_major,
    s.layout_engine_version,
    s.layout_engine_version_major,
    s.operating_system_class,
    s.operating_system_name,
    s.operating_system_name_version,
    s.operating_system_version
);
```

#### Users

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_web_users t
USING <SQL_RUNNER_DERIVED_SCHEMA>.users s
ON T.domain_userid = s.domain_userid
WHEN NOT MATCHED THEN
INSERT
(
    user_id,
    domain_userid,
    network_userid,
    start_tstamp,
    end_tstamp,
    page_views,
    sessions,
    engaged_time_in_s,
    first_page_title,
    first_page_url,
    first_page_urlscheme,
    first_page_urlhost,
    first_page_urlpath,
    first_page_urlquery,
    first_page_urlfragment,
    last_page_title,
    last_page_url,
    last_page_urlscheme,
    last_page_urlhost,
    last_page_urlpath,
    last_page_urlquery,
    last_page_urlfragment,
    referrer,
    refr_urlscheme,
    refr_urlhost,
    refr_urlpath,
    refr_urlquery,
    refr_urlfragment,
    refr_medium,
    refr_source,
    refr_term,
    mkt_medium,
    mkt_source,
    mkt_term,
    mkt_content,
    mkt_campaign,
    mkt_clickid,
    mkt_network
)
VALUES
(
    s.user_id,
    s.domain_userid,
    s.network_userid,
    s.start_tstamp,
    s.end_tstamp,
    s.page_views,
    s.sessions,
    s.engaged_time_in_s,
    s.first_page_title,
    s.first_page_url,
    s.first_page_urlscheme,
    s.first_page_urlhost,
    s.first_page_urlpath,
    s.first_page_urlquery,
    s.first_page_urlfragment,
    s.last_page_title,
    s.last_page_url,
    s.last_page_urlscheme,
    s.last_page_urlhost,
    s.last_page_urlpath,
    s.last_page_urlquery,
    s.last_page_urlfragment,
    s.referrer,
    s.refr_urlscheme,
    s.refr_urlhost,
    s.refr_urlpath,
    s.refr_urlquery,
    s.refr_urlfragment,
    s.refr_medium,
    s.refr_source,
    s.refr_term,
    s.mkt_medium,
    s.mkt_source,
    s.mkt_term,
    s.mkt_content,
    s.mkt_campaign,
    s.mkt_clickid,
    s.mkt_network,
);
```

### Merge your existing data into the new tables (mobile)

The following SQL will merge the existing mobile records in your SQL Runner derived tables into the new dbt derived tables, please run each one as required. This approach will leave any columns that only exist in dbt `null`. Please ensure you replace the schema and table names with your ones where appropriate.

**If you are using Redshift, be sure to `commit` your changes.**

> **Warning:** It is possible, particularly for columns which may have been null, that the types of columns across the two tables don't entirely match. Your warehouse may manage this for you, or you may have to use a `cast(col_name as new_type)` in place of just selecting the column based on any error message you receive.

#### Screen Views

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_screen_views t
USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_screen_views s
ON T.screen_view_id = s.screen_view_id
WHEN NOT MATCHED THEN
INSERT
(
    screen_view_id,
    event_id,
    app_id,
    user_id,
    device_user_id,
    network_userid,
    session_id,
    session_index,
    previous_session_id,
    session_first_event_id,
    screen_view_in_session_index,
    screen_views_in_session,
    dvce_created_tstamp,
    collector_tstamp,
    derived_tstamp,
    model_tstamp,
    screen_view_name,
    screen_view_transition_type,
    screen_view_type,
    screen_fragment,
    screen_top_view_controller,
    screen_view_controller,
    screen_view_previous_id,
    screen_view_previous_name,
    screen_view_previous_type,
    platform,
    dvce_screenwidth,
    dvce_screenheight,
    device_manufacturer,
    device_model,
    os_type,
    os_version,
    android_idfa,
    apple_idfa,
    apple_idfv,
    open_idfa,
    device_latitude,
    device_longitude,
    device_latitude_longitude_accuracy,
    device_altitude,
    device_altitude_accuracy,
    device_bearing,
    device_speed,
    geo_country,
    geo_region,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_region_name,
    geo_timezone,
    user_ipaddress,
    useragent,
    carrier,
    network_technology,
    network_type,
    build,
    version
)
VALUES
(
    s.screen_view_id,
    s.event_id,
    s.app_id,
    s.user_id,
    s.device_user_id,
    s.network_userid,
    s.session_id,
    s.session_index,
    s.previous_session_id,
    s.session_first_event_id,
    s.screen_view_in_session_index,
    s.screen_views_in_session,
    s.dvce_created_tstamp,
    s.collector_tstamp,
    s.derived_tstamp,
    s.model_tstamp,
    s.screen_view_name,
    s.screen_view_transition_type,
    s.screen_view_type,
    s.screen_fragment,
    s.screen_top_view_controller,
    s.screen_view_controller,
    s.screen_view_previous_id,
    s.screen_view_previous_name,
    s.screen_view_previous_type,
    s.platform,
    s.dvce_screenwidth,
    s.dvce_screenheight,
    s.device_manufacturer,
    s.device_model,
    s.os_type,
    s.os_version,
    s.android_idfa,
    s.apple_idfa,
    s.apple_idfv,
    s.open_idfa,
    s.device_latitude,
    s.device_longitude,
    s.device_latitude_longitude_accuracy,
    s.device_altitude,
    s.device_altitude_accuracy,
    s.device_bearing,
    s.device_speed,
    s.geo_country,
    s.geo_region,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_region_name,
    s.geo_timezone,
    s.user_ipaddress,
    s.useragent,
    s.carrier,
    s.network_technology,
    s.network_type,
    s.build,
    s.version,
);
```

#### Sessions

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_sessions t
USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_session s
ON T.session_id = s.session_id
WHEN NOT MATCHED THEN
INSERT
(
    app_id,
    session_id,
    session_index,
    previous_session_id,
    session_first_event_id,
    session_last_event_id,
    start_tstamp,
    end_tstamp,
    model_tstamp,
    user_id,
    device_user_id,
    network_userid,
    session_duration_s,
    has_install,
    screen_views,
    screen_names_viewed,
    app_errors,
    fatal_app_errors,
    first_event_name,
    last_event_name,
    first_screen_view_name,
    first_screen_view_transition_type,
    first_screen_view_type,
    last_screen_view_name,
    last_screen_view_transition_type,
    last_screen_view_type,
    platform,
    dvce_screenwidth,
    dvce_screenheight,
    device_manufacturer,
    device_model,
    os_type,
    os_version,
    android_idfa,
    apple_idfa,
    apple_idfv,
    open_idfa,
    device_latitude,
    device_longitude,
    device_latitude_longitude_accuracy,
    device_altitude,
    device_altitude_accuracy,
    device_bearing,
    device_speed,
    geo_country,
    geo_region,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_region_name,
    geo_timezone,
    user_ipaddress,
    useragent,
    name_tracker,
    v_tracker,
    carrier,
    network_technology,
    network_type,
    first_build,
    last_build,
    first_version,
    last_version
)
VALUES
(
    s.app_id,
    s.session_id,
    s.session_index,
    s.previous_session_id,
    s.session_first_event_id,
    s.session_last_event_id,
    s.start_tstamp,
    s.end_tstamp,
    s.model_tstamp,
    s.user_id,
    s.device_user_id,
    s.network_userid,
    s.session_duration_s,
    s.has_install,
    s.screen_views,
    s.screen_names_viewed,
    s.app_errors,
    s.fatal_app_errors,
    s.first_event_name,
    s.last_event_name,
    s.first_screen_view_name,
    s.first_screen_view_transition_type,
    s.first_screen_view_type,
    s.last_screen_view_name,
    s.last_screen_view_transition_type,
    s.last_screen_view_type,
    s.platform,
    s.dvce_screenwidth,
    s.dvce_screenheight,
    s.device_manufacturer,
    s.device_model,
    s.os_type,
    s.os_version,
    s.android_idfa,
    s.apple_idfa,
    s.apple_idfv,
    s.open_idfa,
    s.device_latitude,
    s.device_longitude,
    s.device_latitude_longitude_accuracy,
    s.device_altitude,
    s.device_altitude_accuracy,
    s.device_bearing,
    s.device_speed,
    s.geo_country,
    s.geo_region,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_region_name,
    s.geo_timezone,
    s.user_ipaddress,
    s.useragent,
    s.name_tracker,
    s.v_tracker,
    s.carrier,
    s.network_technology,
    s.network_type,
    s.first_build,
    s.last_build,
    s.first_version,
    s.last_version
);
```

#### Users

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_users t
USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_users s
ON t.device_user_id = s.device_user_id
WHEN NOT MATCHED THEN
INSERT
(
    user_id,
    device_user_id,
    network_userid,
    start_tstamp,
    end_tstamp,
    model_tstamp,
    screen_views,
    screen_names_viewed,
    sessions,
    sessions_duration_s,
    active_days,
    app_errors,
    fatal_app_errors,
    first_screen_view_name,
    first_screen_view_transition_type,
    first_screen_view_type,
    last_screen_view_name,
    last_screen_view_transition_type,
    last_screen_view_type,
    platform,
    dvce_screenwidth,
    dvce_screenheight,
    device_manufacturer,
    device_model,
    os_type,
    first_os_version,
    last_os_version,
    android_idfa,
    apple_idfa,
    apple_idfv,
    open_idfa,
    geo_country,
    geo_region,
    geo_city,
    geo_zipcode,
    geo_latitude,
    geo_longitude,
    geo_region_name,
    geo_timezone,
    first_carrier,
    last_carrier
)
VALUES
(
    s.user_id,
    s.device_user_id,
    s.network_userid,
    s.start_tstamp,
    s.end_tstamp,
    s.model_tstamp,
    s.screen_views,
    s.screen_names_viewed,
    s.sessions,
    s.sessions_duration_s,
    s.active_days,
    s.app_errors,
    s.fatal_app_errors,
    s.first_screen_view_name,
    s.first_screen_view_transition_type,
    s.first_screen_view_type,
    s.last_screen_view_name,
    s.last_screen_view_transition_type,
    s.last_screen_view_type,
    s.platform,
    s.dvce_screenwidth,
    s.dvce_screenheight,
    s.device_manufacturer,
    s.device_model,
    s.os_type,
    s.first_os_version,
    s.last_os_version,
    s.android_idfa,
    s.apple_idfa,
    s.apple_idfv,
    s.open_idfa,
    s.geo_country,
    s.geo_region,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_region_name,
    s.geo_timezone,
    s.first_carrier,
    s.last_carrier
);
```

#### App Errors

```sql
MERGE INTO <DBT_DERIVED_SCHEMA>.snowplow_mobile_app_errors t
USING <SQL_RUNNER_DERIVED_SCHEMA>.mobile_app_errors s
ON t.event_id = s.event_id
WHEN NOT MATCHED THEN
INSERT
(
    event_id,
        app_id,
        user_id,
        device_user_id,
        network_userid,
        session_id,
        session_index,
        previous_session_id,
        session_first_event_id,
        dvce_created_tstamp,
        collector_tstamp,
        derived_tstamp,
        model_tstamp,
        platform,
        dvce_screenwidth,
        dvce_screenheight,
        device_manufacturer,
        device_model,
        os_type,
        os_version,
        android_idfa,
        apple_idfa,
        apple_idfv,
        open_idfa,
        screen_id,
        screen_name,
        screen_activity,
        screen_fragment,
        screen_top_view_controller,
        screen_type,
        screen_view_controller,
        device_latitude,
        device_longitude,
        device_latitude_longitude_accuracy,
        device_altitude,
        device_altitude_accuracy,
        device_bearing,
        device_speed,
        geo_country,
        geo_region,
        geo_city,
        geo_zipcode,
        geo_latitude,
        geo_longitude,
        geo_region_name,
        geo_timezone,
        user_ipaddress,
        useragent,
        carrier,
        network_technology,
        network_type,
        build,
        version,
        event_index_in_session,
        message,
        programming_language,
        class_name,
        exception_name,
        is_fatal,
        line_number,
        stack_trace,
        thread_id,
        thread_name
)
VALUES
(
    s.event_id,
    s.app_id,
    s.user_id,
    s.device_user_id,
    s.network_userid,
    s.session_id,
    s.session_index,
    s.previous_session_id,
    s.session_first_event_id,
    s.dvce_created_tstamp,
    s.collector_tstamp,
    s.derived_tstamp,
    s.model_tstamp,
    s.platform,
    s.dvce_screenwidth,
    s.dvce_screenheight,
    s.device_manufacturer,
    s.device_model,
    s.os_type,
    s.os_version,
    s.android_idfa,
    s.apple_idfa,
    s.apple_idfv,
    s.open_idfa,
    s.screen_id,
    s.screen_name,
    s.screen_activity,
    s.screen_fragment,
    s.screen_top_view_controller,
    s.screen_type,
    s.screen_view_controller,
    s.device_latitude,
    s.device_longitude,
    s.device_latitude_longitude_accuracy,
    s.device_altitude,
    s.device_altitude_accuracy,
    s.device_bearing,
    s.device_speed,
    s.geo_country,
    s.geo_region,
    s.geo_city,
    s.geo_zipcode,
    s.geo_latitude,
    s.geo_longitude,
    s.geo_region_name,
    s.geo_timezone,
    s.user_ipaddress,
    s.useragent,
    s.carrier,
    s.network_technology,
    s.network_type,
    s.build,
    s.version,
    s.event_index_in_session,
    s.message,
    s.programming_language,
    s.class_name,
    s.exception_name,
    s.is_fatal,
    s.line_number,
    s.stack_trace,
    s.thread_id,
    s.thread_name
);
```

---

# Mobile data model for SQL Runner
> Legacy SQL Runner mobile data model for transforming mobile event data into derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-mobile-data-model/

> **Tip:** For any new developments we highly recommend using our [unified digital package for dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

## Overview

The Snowplow mobile data model aggregates Snowplow's out-of-the-box mobile events to create a set of derived tables - screen views, sessions, and users. These contain many useful dimensions, as well as calculated measures such as screen views per session.

![](/assets/images/Screenshot-2022-03-25-at-10.50.33-5d404dca4e545db52a48114f0750bd14.png)

The Snowplow mobile data model processes and aggregates data collected by the Snowplow [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) and [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) mobile trackers into a set of derived tables - screen views, sessions and users - with each table generated by its respective module.

In addition to these 'standard' modules there are optional modules that can be enabled depending on your tracker implementation. Currently there is only one such optional module, Application Errors, which provides application crash reporting. These modules produce their own derived output table, as well as being aggregated and joined into 'standard' modules such as sessions for high level KPIs. As the tracker's functionality continues to expand, so will this library of optional modules.

Unlike the [web model](/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/) the level of aggregation is not linear, as shown in the diagram below. This allows the sessions module to consume events other than just screen views, such as app installs or crashes.

![Web vs Mobile](/assets/images/web_vs_mobile-c178efd12abd487001ed45f836868293.jpg)

The model runs incrementally, only processing new events (and events that have previously been modeled but are part of screen views, sessions or users for which/whom there is new information) with every run. The incremental logic is separate from the logic that calculates the tables so as to make customization of the model easier. More information on the model structure and customization options can be found below.

**The latest Snowplow mobile data model can be found in the [snowplow/data-models GitHub repository](https://github.com/snowplow/data-models/tree/master).**

#### Supported Warehouses

- Redshift
- BigQuery
- Snowflake

#### Requirements

- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker version 1.1.0 or later implemented.
- Mobile session context enabled.
- Screen view events enabled.

## Quick start

#### Prerequisites

- [SQL-runner](https://github.com/snowplow/sql-runner) is installed.
- Snowplow [Android](/docs/sources/mobile-trackers/previous-versions/android-tracker/) or [iOS](/docs/sources/mobile-trackers/previous-versions/objective-c-tracker/) mobile tracker implemented with mobile events dataset present in a database.

#### Playbook Configuration

##### Setting Schemas

For all playbooks within `mobile/v1/{warehouse}/sql-runner/playbooks/standard`, configure the following:

- `input_schema` - Set to the schema containing your Snowplow events data.
- `scratch_schema` - Set to the scratch/staging schema to contain the intermediate tables generated by this model. Create if required.
- `output_schema` - Set to a schema to contain the model's final output tables (`mobile_screen_views`, `mobile_sessions` etc.)

##### Base Module

Within the base module's main playbook, `01-base-main.yml.tmpl`:

1. Set the `start_date` as desired.

2. Enable the following contexts as desired by setting to `True`:

   - Mobile - Device type, OS etc.
   - Geolocation - Device latitude, longitude, bearing etc.
   - Application - App version and build.
   - Screen - Screen details associated with a mobile event.

3. Adjust filtering of events if required:

   - `platform_filters`: Default - `platform = 'mob'`. Override if required.
   - `app_id`: Default - No filter on `app_id`. Add if required.

```yaml
# 01-base-main.yml.tmpl
:variables:
   ...
   :start_date:         2020-01-01 #Set as required
   ...
   #Enable contexts if desired
   :mobile_context: false
   :geolocation_context: false
   :application_context: false
   :screen_context: false
   :platform_filters: [] #Override default if required e.g. ['mob','tv']
   :app_id_filters: [] #Add app_id filter if required e.g ['my_app']
```

The remaining variables are set to the recommended defaults. For more information please see the base module's README within the playbooks directory.

##### Optional Modules

Optional modules are disabled by default. Please enable where appropriate by setting `enabled: true` in the main playbook of each module.

#### Running the model

##### Authentication

1. Set database password as an environmental variable.

   - Redshift: Set environmental variable, `REDSHIFT_PASSWORD`, to your database password.
   - BigQuery: Set environmental variable, `GOOGLE_APPLICATION_CREDENTIALS`, to the path of your GBQ json credential file
   - Snowflake: Set environmental variable, `SNOWFLAKE_PASSWORD`, to your database password.

2. Fill in the relevant template in `.scripts/templates` with your database details.
   - Redshift & Snowflake: Leave `PASSWORD_PLACEHOLDER` as is. This placeholder will be replaced at run time with the credentials from env var `{WAREHOUSE}_PASSWORD`, set in step 1.

##### Execution

For a full run of the model:

```bash
bash .scripts/run_config.sh -b ~/pathTo/sql-runner -c mobile/v1/{warehouse}/sql-runner/configs/datamodeling.json -t .scripts/templates/{warehouse}.yml.tmpl;
```

For more details please see the README within the `.scripts` directory.

##### Authentication

For each playbook within `mobile/v1/{warehouse}/sql-runner/playbooks/standard` fill in your database connection details. For more details please refer to the [SQL Runner Docs](/docs/modeling-your-data/modeling-your-data-with-sql-runner/#target-configuration).

##### Execution

There are many tools available to schedule and run SQL jobs such as the mobile model. While these tools differ in operation the basic principles to running the mobile model should be similar.

A series of tasks will need to be set to execute SQL Runner against each individual playbook of the model in turn. For example, to run the main playbook of the base module:

```bash
bash {sql-runner-path} -playbook sql-runner/playbooks/standard/01-base/01-base-main.yml.tmpl -sqlroot sql-runner/sql
```

The correct order of playbook execution and the dependancies between them can be determined from the datamodeling config file within the repo.

#### Backfilling

By default when first running the model, events will be processed in 7 day increments beginning at the `start_date` set in the base module's playbook, `01-base-main.yml.tmpl`, as described above. To backfill data faster, set the `update_cadence_days` in the `01-base-main.yml.tmpl` to required number of days. Once backfilled, it is recommended to set `update_cadence_days` back to 7 days for performance.

#### Testing

The mobile model comes with a suite of data validation checks to ensure the model is running as intended. These checks are performed by [Great Expectations](https://greatexpectations.io/). The tests were used during the model's development and are not required to run the mobile model. If desired, they can however be used during the deployment of the mobile model on your own Snowplow dataset to validate the output. For further details please refer to the README in the `.scripts` directory.

#### Advanced Configurations

This quick start guide covers the implementation process for the majority of use cases. Further configurations can be made such as:

- Running specific modules more frequently than others.
- Running specific modules only.
- Adjusting table scan limits.

For further details please refer to the READMEs within each module's playbook directory.

## Technical architecture

#### The entire model

This model consists of a series of modules, each is idempotent and can be run on independent schedules, and each produces a table which serves as the input to the next module.

![](/assets/images/mobile_full_model_structure-ee33577d65c009462e5a236bf9a6e92f.jpg)

#### The individual modules

The ‘standard’ modules can be thought of as source code for the core logic of the model, which Snowplow maintains. These modules carry out the incremental logic in such a way as custom modules can be written to plug into the model’s structure, without needing to write a parallel incremental logic. We recommend that all customisations are written in this way, which allows us to safely maintain and roll out updates to the model, without impact on dependent custom SQL. For more information on this, jump to the section on customizing the model below.

Each module produces a table which acts as the input to the subsequent module (the `_staged` tables), and updates a production table – with the exception of the base module, which takes atomic data as its input, and does not update a production table.

Each module comes with a `99-{module}-complete playbook`, which marks that module complete by updating any relevant manifests, and truncating the input, and cleaning up temporary tables. The complete steps may be run at the end of each module, or at the end of the run.

More detail on each module can be found in the relevant READMEs in the [GitHub repository](https://github.com/snowplow/data-models/tree/master).

![](/assets/images/web_model_module-1-630f2fc2a9282fd3089910576d1c4c0b.jpg)

Architecture of an individual mobile module

## Customizing the model

Custom modules can fit into the incremental structure by consuming the same inputs, and running before the `99-{module}-complete` playbook runs. Custom modules may also consume and intermediary tables of the standard module, which will not be dropped until the `99-{module}-complete` playbook runs.

Any custom SQL that depends on a `_staged` table as its input should run before the complete step of the module which handles that same input. For example, custom logic which takes mobile\_events\_staged as an input should run before the `99-sessions-complete` playbook which truncates mobile\_events\_staged.

An example custom module has been included in `mobile/v1/redshift/sql-runner/sql/custom`. In this module we:

1. Read screen views from `scratch.mobile_screen_views_staged`.
2. Aggregate screen views to one row per `session_id`.
3. Delete and insert into the output table `derived.session_goals`. This table can then be joined onto `derived.mobile_sessions` on `session_id`.

The playbooks should then be run in the following order:

```text
standard/01-base/01-base-main
standard/01-base/99-base-complete
standard/02-screen-views/01-screen-views-main
standard/02-screen-views/99-screen-views-complete
standard/03-optional-modules/01-app-errors/01-app-errors-main
standard/03-optional-modules/01-app-errors/99-app-errors-complete
standard/04-sessions/01-sessions-main
custom/04-session-goals/01-session-goals-main
custom/04-session-goals/01-session-goals-complete
standard/04-sessions/01-sessions-complete
```

An example of this can be seen in the `datamodeling_custom_module.json` file within the config directory.

Custom modules can also be created with their own independent manifesting logic, in which case they are more complex, but don’t rely on the standard modules.

More information on customizing the Snowplow mobile data model can be found in the custom module [README](https://github.com/snowplow/data-models/blob/master/mobile/v1/redshift/sql-runner/sql/custom/README.md).

## Metadata

Metadata is logged in the `{{.output_schema}}.datamodel_metadata{{.entropy}}` table, per-module and per-run. A persistent ID is created, so that separate modules within the same run may be identified.

---

# Web data model for SQL Runner
> Legacy SQL Runner web data model for transforming web event data into derived tables.
> Source: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-sql-runner/sql-runner-web-data-model/

> **Tip:** For any new developments we highly recommend using our [unified digital package for dbt](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) instead. SQL Runner is no longer under active development and will only receive bug fixes in the future.

The Snowplow web data model aggregates Snowplow's out of the box page view and page ping events to create a set of derived tables - page views, sessions and users - that contain many useful dimensions as well as calculated measures such as time engaged and scroll depth.

![](/assets/images/image-3-2f58ac3c0dcb0a7256a0dbd728280cef.png)

**The latest Snowplow web data model can be found in the [snowplow/data-models GitHub repository](https://github.com/snowplow/data-models/tree/master).**

## Quickstart

#### Requirements

- [Snowplow Javascript tracker](/docs/sources/web-trackers/) version 2 or later implemented.
- Web Page context [enabled](/docs/sources/web-trackers/tracker-setup/initialization-options/) (enabled by default in [v3+](/docs/sources/web-trackers/tracker-setup/initialization-options/)).
- [Page view events](/docs/sources/web-trackers/tracking-events/page-views/) implemented.

#### Prerequisites

- [SQL-runner](https://github.com/snowplow/sql-runner) must be installed ([Setup guide](/docs/modeling-your-data/modeling-your-data-with-sql-runner/)).
- A dataset of web events from the [Snowplow Javascript tracker](/docs/sources/web-trackers/) must be available in the database.

#### Configuring and running the model

First, fill in the connection details for the target database in the relevant template in `.scripts/templates/redshift.yml.tmpl`.

Password can be left as a `PASSWORD_PLACEHOLDER`, and set as an environment variable or passed as an argument to the run\_playbooks script. See the README in `.scripts` for more detail.

Variables in each module's playbook can also optionally be configured also. See each playbook directory's README for more detail on configuration of each module.

You can then run the model, either by running playbooks individually by running SQL-runner locally, or via your Snowplow GitHub repository. Of course, as a Snowplow customer you can also reach out to Support to get the model deployed for you.

## Technical architecture

#### The entire model

This model consists of a series of modules, each is idempotent and can be run on independent schedules, and each produces a table which serves as the input to the next module.

![](/assets/images/web_full_model_structure-317e3b071cb592772020d3073ae4cb04.jpg)

#### The individual modules

The 'standard' modules can be thought of as source code for the core logic of the model, which Snowplow maintains. These modules carry out the incremental logic in such a way as custom modules can be written to plug into the model's structure, without needing to write a parallel incremental logic. We recommend that all customisations are written in this way, which allows us to safely maintain and roll out updates to the model, without impact on dependent custom SQL. For more information on this, jump to the section on customizing the model below.

Each module produces a table which acts as the input to the subsequent module (the `_staged` tables), and updates a production table - with the exception of the base module, which takes atomic data as its input, and does not update a production table.

Each module comes with a `99-{module}-complete playbook`, which marks that module complete by updating any relevant manifests, and truncating the input, and cleaning up temporary tables. The complete steps may be run at the end of each module, or at the end of the run.

![](/assets/images/web_model_module-630f2fc2a9282fd3089910576d1c4c0b.jpg)

More detail on each module can be found in the relevant READMEs in the [GitHub repository](https://github.com/snowplow/data-models/tree/master).

## Customizing the model

Custom modules can fit into the incremental structure by consuming the same inputs, and running before the `99-{module}-complete` playbook runs. Custom modules may also consume and intermediary tables of the standard module, which will not be dropped until the `99-{module}-complete` playbook runs.

Any custom SQL that depends on a `_staged` table as its input should run before the complete step of the module which handles that same input. For example, custom logic which takes events\_staged as an input should run before the `99-page-views-complete` playbook.

As an introductory example, if there is a requirement to include data from custom events and entities for page views, for example, we would write a custom module which:

- Reads events (i.e. event\_ids) from the `scratch.events_staged` table
- Aggregates to one row per `page_view_id`
- Deletes and inserts this to a custom table which can join to the `derived.page_views table` on `page_view_id`

If the playbook for this custom module is called `AA-my-custom-page-views-level-module.yml.tmpl`, then we would run the playbooks as follows:

- `standard/01-base/01-base-main.yml.tmpl`
- `standard/01-base/99-base-complete.yml.tmpl`
- `standard/02-page-views/01-page-views-main.yml.tmpl`
- `custom/AA-my-custom-page-views-level-module.yml.tmpl`
- `standard/02-page-views/99-page-views-complete.yml.tmpl`

Custom modules can also be created with their own independent manifesting logic, in which case they are more complex, but don't rely on the standard modules.

More information on customizing the Snowplow web data model can be found in the [`web/v1/redshift/sql-runner/sql/custom` folder on GitHub](https://github.com/snowplow/data-models/).

## Metadata

Metadata is logged in the `{{.output_schema}}.datamodel_metadata{{.entropy}}` table, per-module and per-run. A persistent ID is created, so that separate modules within the same run may be identified. This ID is automatically handled, as long as the `99-{module}-complete` step of the last module to run has the `:ends_run:` variable set to `false`.

Rather than setting this variable, this can be maintained manually be running the `00-setup/00-setup-metadata.yml.tmpl` before all other steps, and the `00-setup/99-complete-metadata.yml.tmpl` playbook after all other steps.

---

# Run data models from Console
> The guide to running data models in Snowplow.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/

You can use [Snowplow Console](https://console.snowplowanalytics.com) to schedule and run [dbt](https://www.getdbt.com/) data models, including our [out-of-the-box models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/), [automatically generated models](/docs/modeling-your-data/automatically-generated-data-models/), and any custom or third party models.

## Overview

```mermaid
flowchart LR
  subgraph console [Snowplow Console]
    schedule1(Schedule 1)
    schedule2(Schedule 2)
    schedule3(...)
  end
  subgraph "Your cloud account"
    job(Modeling job)
  end
  subgraph "Your Git repository"
    project(dbt project)
  end
  warehouse[(Warehouse)]
  console(Snowplow Console) -->|Trigger| job -->|Execute models| warehouse
  job -.->|Read configuration| project
```

To use this feature, you will need to store your data models in a **Git repository** organized into one or more [**dbt projects**](https://docs.getdbt.com/docs/build/projects). This provides you full control over the model logic and helps with versioning and collaboration.

In Console, you can define multiple **run configurations** and **schedules** to control what dbt commands get executed and when. For example, you could set the Unified Digital data model to run every day at midnight.

Based on what you defined, Console will trigger data modeling **jobs** in your cloud account associated with Snowplow (managed by you, in the case of Private Managed Cloud, or by Snowplow otherwise). The jobs will execute actual dbt commands against your data warehouse.

> **Tip:** This architecture allows you to take advantage of a secure connection to the warehouse (e.g., VPC Peering or AWS Private Link) that you might have already set up with Snowplow.

## Prerequisites

To get started, you will need a Git repository (and sufficient permissions to grant read access to Snowplow).

We also highly recommend installing dbt locally so that you can quickly test your model configuration. Follow the [dbt documentation](https://docs.getdbt.com/docs/core/installation-overview) to install dbt Core as well as the relevant adapter for your destination (e.g., `dbt-snowflake`).

## Initial setup

First, create a Git connection to your repository:

- In Console, go to **Destinations > Connections > Set up connection > Git connection**
- The setup flow will guide you to create and test the connection

Next, create the model project:

- Navigate to **Modeling > Model projects > Create model project**
- You will need to provide the Git connection you created in the previous step, as well as some metadata for your project
- Console will show you the dbt commands to initialize your project if you haven't done so (you can also bring a pre-existing project)

Once the project is created, you can start adding data models.

## Add data models

> **Tip:** If you already have a dbt project with a set of data models, you can skip this section. Console will work with your current dbt configuration in `dbt_project.yml` and `packages.yml`.

Because the dbt configuration is stored in your Git repository, rather than in Console, you will add and configure models by making commits to this repository.

For example, to install our Unified Digital data model, you can follow the [tutorial](/tutorials/unified-digital/intro/) and add `snowplow/snowplow_unified` to your `packages.yml`.

Likewise, to configure variables for the models, edit the `dbt_project.yml` file.

With multiple models, you might run into a situation where one depends on the output of another. As long as you use [the `ref` function](https://docs.getdbt.com/faqs/Models/create-dependencies), dbt will automatically infer and honor dependencies. In particular, see how to set up our Unified Digital and Attribution models to [work together](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/keeping-cohesion-with-unified/).

> **Note:** When running the models, Console will override any warehouse connection settings with the ones you will configure in the next section.

Once you are happy with the dbt project setup, run dbt commands locally to verify it.

## Connect to the warehouse

In this step, you will create a warehouse connection for the models to use. Note that you can define more than one connection, for example, if you would like to have separate model run configurations for production and QA data.

> **Tip:** You might have already set up a warehouse connection for loading the data. Data modeling, however, requires a new connection, as you will often want data models to run under a different user/role and with different permissions.

Navigate to **Destinations > Connections > Set up connection > Data modeling connection**. Select your warehouse and follow the steps in Console to create and test your connection.

## Add run configurations

For Console to run your data models, you will need to create run configurations.

Each run configuration controls what dbt commands will be executed and when. More concretely:

- You can add one or more run configurations to a project
- Each run configuration can contain one or more schedules
- For each schedule, you can define specific dbt commands

> **Tip:** If you want to run all your models every day at midnight, you would create:
>
> - One run configuration
> - One schedule (daily, 12 AM)
> - Add standard dbt commands, e.g. `dbt deps` and `dbt run`
>
> If you want to run different models within the project at different times, you can use dbt’s [selector features](https://docs.getdbt.com/reference/commands/run#running-specific-models). In this case, create:
>
> - One run configuration
> - Two schedules, e.g. daily at 12 AM and every Monday at 3 AM
> - Pick different commands for each schedule, e.g. `dbt run --select <...>`
>
> In both cases, note that `dbt run` will automatically run all the relevant models in the correct order, [accounting for dependencies](https://docs.getdbt.com/faqs/Models/create-dependencies).

To create a run configuration:

- Navigate to your model project in Console
- Click **Add run configuration**
- Pick the warehouse connection created earlier
- Click **Add schedule**
- Select the desired time and dbt commands
- Repeat as necessary

## Run the models

Once you’ve defined the run configurations and schedules, Console will automatically run your data models. You can see when the next run is scheduled by looking at each run configuration.

![Run configuration example](/assets/images/run-configurations-85b1a6e33da4d61b7835cbbda702f0aa.png)

You can also use the **Run now** button to trigger a one-off model run for a specific schedule. The resulting job will use the same commands and settings.

## Troubleshooting

See [Resolving data model failures](/docs/modeling-your-data/running-data-models-via-console/resolving-data-model-failures/).

---

# Resolve data model failures
> Troubleshoot and resolve data model failures in Snowplow Console including debugging steps and common issues.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/resolving-data-model-failures/

## Receive alerts

When you create a model project in Console, you have to specify one or more owner email addresses. (You can edit the project later to add or remove owners.)

Console uses these email addresses to send notifications when a data modeling job fails for any reason. The email will contain a link to the **Jobs** section in Console with the relevant failure.

## Diagnose failures

You will be able to see the details of your data model failure in the jobs interface in Console. Under **Job DAG** (Directed Acyclic Graph) for the failed job, click on the failed step to see details of what went wrong.

![Example job DAG](/assets/images/dbt-dag-390b9fb149c227f96a6367d15175d55b.png)

The **Error Output** will show you the error logs from the dbt commands that were executed. These logs will contain the failure information that dbt and the warehouse relayed back to Console.

![Example error output](/assets/images/dbt-step-error-output-2533cadbba55a302223042fe12de784e.png)

## Understand and resolve issues

An error message such as 'connection refused' or `EOF` (end of file) is typically caused by a network problem. These errors often do not need any interaction to be resolved.

If the logs show a dbt call with a model error, you will need to identify the issue in your model files and push the changes to your Git repository. For help debugging dbt errors, please see the [dbt docs](https://docs.getdbt.com/guides/debug-errors) and ensure your models run correctly locally before pushing changes.

> **Note:** Even if a job fails, it will run again next time as per the schedule (e.g. next midnight). If you identified an issue, you can temporarily disable the schedule to prevent spurious failures while you work on a fix.

---

# Retrieve data model execution data
> Retrieve data model execution data via API for monitoring and integration with alerting infrastructure.
> Source: https://docs.snowplow.io/docs/modeling-your-data/running-data-models-via-console/retrieving-job-execution-data-via-the-api/

The API that powers the warehouse jobs monitoring view in Snowplow Console (Jobs) is also available for consumption by other authenticated clients.

The exact same data about past and current jobs executions can be retrieved and processed programmatically. Hence, it possible to integrate with your monitoring infrastructure and enable additional alerting or insights.

## Getting started

You can have a look at and interact with all available endpoints in the [API documentation](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/Jobs).

### Authorization

To be able to post sample requests in the documentation you need to click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

The endpoints focus on the main operations in the workflow around:

1. Getting a list of job runs within a specific time window. This window **may span 96 hours at most and be contained within the last week**.
2. Retrieving information about a particular job run, including its individual steps.
3. Retrieving only the steps of a particular job run without any additional information.
4. Getting the full data available about the execution of a particular step of a particular job run, including `stdout`/`stderr`.

Each request will need to include your Organization ID. You can find it [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

Follow the instructions in the [Account management](/docs/account-management/) section to obtain an access token for API authentication.

## Retrieve job run information

### Getting a list of job runs for a specific time window

The invocation is as such:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs?from=X&to=Y`

`X` and `Y` are required and can be passed either as ISO8601 timestamps or as ISO8601 durations, and they must both be of the same type (i.e. it is not possible to mix a timestamp with a duration). For example, the query parameters can be:

- `from=2020-12-02T03:40:00Z&to=2020-12-02T15:40:00Z`
- `from=PT-12H&to=PT0S`

As mentioned earlier, the total time window requested cannot exceed 96 hours (4 days) or start earlier than 1 week ago.

Limitations to be aware of

When it comes to job executions, our platform tracks state change events. A side-effect of this is that in extreme cases of jobs with very significant numbers or steps or far too frequent executions the events for the requested time window may have to be automatically trimmed to the latest 10,000. This guarantees an up-to-date view but may be missing job runs that started near the beginning of the window. In this edge case the problem can be side-stepped by splitting the window to more requests (e.g. 2 + 2 days).

### Retrieving information about a specific job run

Can be invoked as:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}`

`runId` is a hashed value that can be found within results of job run listing queries. The extra information that this endpoint returns (compared to job listing results) is the list of steps in that job run and their state, alongside their inter-dependencies.

### Retrieving steps of a specific job run

If it is preferred to get only the steps for a job run, without the information already contained within the job runs list, one can use the respective endpoint:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}/steps`

### Retrieving detailed information about a specific step execution

Finally, one may wish to get the complete information about the execution of a single step. This may be useful, for instance, if the job failed on that step and the standard output or standard error of the step is of interest. To get that, one can invoke the respective endpoint:

`GET /api/msc/v1/organizations/{organizationId}/jobs/v1/runs/{runId}/steps/{stepName}`

---

# Marketing Attribution visualization
> Marketing attribution visualization for multi-channel journeys with first-touch, last-touch, linear, and position-based models.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/attribution-modeling/

In today's increasingly complex digital world, users often take multi-channel journeys before converting. Assigning credit across multiple touchpoints is vital to getting an accurate picture of the efficacy of your marketing channels, yet requires merging disparate datasets and running complex calculations.

The Snowplow Marketing Attribution visualization (together with the [Snowplow Attribution dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)) lowers the barrier to entry for your marketing team through the following features:

- Incremental SQL model in your warehouse for cost-effective computation
- Choice of first-touch, last-touch, linear and positional methods, with additional filters and transforms available
- Reports for conversions, revenue, spend and Return On Advertising Spend (ROAS) per channel and campaign
- Option to specify your own touchpoint and advertising spend tables
- Intermediate tables that you can build your own attribution models on top of

![the top half of the overview page](/assets/images/overview-f26055aba0a5f6c9837854a24c45a0c5.png)

## Requirements

- [Campaign Attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) enabled
- [Referrer Parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) enabled
- Running the [Snowplow Unified Digital dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) with `conversion event(s)` defined and the optional conversion module enabled
- Running the [Snowplow Attribution dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization

## Configuration

Configuration is best performed by a Data Analyst or Engineer.

All these settings are global for all users, meaning if you change them they will be changed for everyone. The first user of the visualization will have to define at least one **view** which is the dataset needed to generate the charts. Creating views can be done on the **Settings** page.

### Update method

Use the toggle **Last N days View (Dynamic)** to choose whether to define a dynamic view that auto-updates or a static view:

- **Dynamic (Last N Days)**: the so-called dynamic views are for generating datasets that have a rolling conversion window of last nth day and will be refreshed automatically (e.g. last 30 days).

  The visualization will save the last-refreshed date with the view configurations. Any subsequent day a user logs back into the visualization, a query will run in the background to look for any newly processed conversion events in the conversion source. If there are any, the dynamic datasets are refreshed by running all the queries that are needed to generate data for the charts to populate.

  If you choose this option, set the **auto-update days**: the number of days since the last conversion event defined here will define the conversion window. The latest conversion window in use can be checked on the **Settings** page where a table with information on all the created views is displayed, including the conversion window that is currently in use.

- **Static (Custom Date Range)**: non-dynamic views need a name, and will typically be used to generate a fixed dataset (e.g. Jan, Q1, 2023) to avoid having to recalculate the analysis for subsequent users.

  Define a fixed conversion window by selecting the appropriate date range with the date picker tool. It is activated by clicking on the default date range.

### Set a currency symbol

The currency symbol defaults to "$".

### Connect to data sources

1. Select your schema that contains the derived unified and attribution tables. This will trigger an update which checks for any tables with the names closest to what the visualization expects.

2. After waiting for the update to take place, you can check if the auto-detected source tables are what you expect. Change them to any other appropriate existing tables if they are not correct.

3. For most users, the **Paths To Non-Conversion** table should be set to **Do not use paths\_to\_non\_conversion table**.

   This drop and recompute table calculates the paths your customers have followed that have not led to a conversion, for use in the **Path Summary** page. This table is not recalculated by the visualization. Therefore it should only be used for a static view with the intention of consuming the same period as in the latest data model.

4. Overwrite the **Attribution Manifest** table. Most likely the schema name will have to be modified. Keep the `schema_name.table_name` notation here, and press enter once modified.

5. (Optional but recommended) Specify the **Spend Source** to get the ROAS calculation in your overview. This will most likely be a view you created on top of your table that holds your marketing spend data. The view will make sure you align the expected field names. It should have `campaign`, `channel`, `spend` and `spend_tstamp` for the analysis to work. Press enter once modified.

6. Once you are happy with all the inputs, press the **Create View** button. The visualization will first run a validation against the data sources making sure it has all the fields it needs. If all is well, it will save the view.

7. The dashboards are ready to be explored. The first time a dashboard page is visited, the relevant query will run once and the data will be cached to speed up subsequent dashboard explorations for other users.

## Using the Dashboard

Once at least one view is configured, all users can go ahead and use the dashboards directly.

### Filters

There are various filters at the top of each dashboard page that make the data exploration interactive. Because the queries are cached, users can make any of these interactive changes without involving the warehouse, avoiding expensive queries or laggy information retrieval.

To use a filter, first select which view to use, and then make changes with **View Settings**:

- select which **Attribution Type** to use (`First Touch`, `Last Touch`, `Linear` or `Position Based`)
- choose between using **Campaign** or **Channel** for paths

Some pages have additional filters, such as the **Number of items** filter for the **Path Summary** page, which reduces the number of items shown in specific charts.

## Editing and deleting views

Edit or delete existing views from the **Settings** page, using the **Edit/Delete Views** dropdown.

- **Delete**: click X next to the view.
- **Edit**: click on the name of the view. The visualization will take you to the view configuration page where you can make amendments. Saving will overwrite the existing view configurations.

![the edit and delete views dropdown](/assets/images/edit_delete_views-b9c2000c24b4add79f8cccd6b992a73b.png)

---

# Ecommerce Analytics visualization
> Ecommerce analytics visualization with dashboards for revenue, purchases, checkout journey, and transaction patterns.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/ecommerce-analytics/

The Ecommerce Analytics visualization offers an in-depth breakdown of ecommerce activities across your websites and mobile apps, presenting an enhanced rendition of Google Analytics v4 reports. Use the dashboards to:

- Access real-time data directly within your data warehouse in a governed manner.
- Navigate through a series of interactive dashboards designed to surface monetization data, purchasing behaviors, checkout experiences, and transaction patterns.
- Customize your analytics workflow with filters for campaigns, channels, and products.

Leveraging Snowplow's Ecommerce event tracking packages and our Ecommerce dbt package, the visualization transforms raw event data into structured tables covering carts, checkouts, products, transactions, and user interactions, providing valuable insights for informed decision-making.

![the overview page](/assets/images/overview-3594c6db8bee3690796595f15179f51b.png)

## Requirements

- Running the [Snowplow Ecommerce dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization
- Ecommerce events tracked using the ecommerce APIs for the [JavaScript (Snowplow Ecommerce plugin)](/docs/sources/web-trackers/tracking-events/ecommerce/) or [mobile trackers](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/)

## Usage

The following steps will guide you through using the visualization.

### Explore the dashboards using sample data (optional)

Before you configure your data, you are free to explore all the dashboards using the sample data included. The use of sample data is enabled by default and can be enabled/disabled using the **Use Sample Data** toggle on the **Settings** page.

You can browse the following pages:

1. **Overview**: key metrics including the revenue, number of customers and the average order value. It also shows the revenue for top performing products and product lists.
2. **Purchases**: dive deeper into the purchased products and compare product views, add to carts, units purchased, and more.
3. **Sessions**: a funnel of how users navigated from product views all the way to transactions by their number of sessions, conversion rate and abandonments.
4. **Checkout**: individual checkout steps within the funnel, showing conversion rate and abandonment for each step.
5. **Transactions**: key metrics for the transactions including types of payments.

### Configure using the Ecommerce dbt package tables

The **Settings** page lets you choose the schema and tables in your data warehouse to use. These refer to the output of the Ecommerce dbt package.

First, choose the warehouse schema where your dbt Ecommerce package produced the derived tables (e.g. `dbt_ecom_derived`). The visualization will suggest the most likely tables below. The following derived tables will need to be configured: `cart interactions`, `checkout interactions`, `product interactions`, `sessions`, and `transaction interactions`.

Verify that these match the desired tables and click **Save Settings**. If the tables have the required columns, you can navigate to the dashboard pages in the sidebar and explore your data.

### Choose the date range and filters

At the top of each page, there is an option to choose the date range (including a secondary date range to compare the data against) and filters. These options let you narrow down the data shown in the dashboard to what you are interested in.

![the top of the Checkout page, showing the filter buttons](/assets/images/filters-6234c465859d437817a38593a0cb7aa0.png)

When you click on the **Filters** button, you will see a modal with an option to add conditions on fields in the data. For example, you can add conditions on the products or product lists that you are interested in.

![setting up a filter for a completed order](/assets/images/filters_modal-3da85a650b812050f1b451d9c7701cca.png)

---

# Funnel Builder visualization
> Build custom funnel analyses to understand user journeys, conversion rates, and drop-off points across any sequence of events.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/funnel-builder/

Funnels are an essential tool for understanding user journeys on your app or website. They help to visualize how many users complete each event along a journey such as signing up or making a purchase, so you can understand which stages are leading to the most drop-off and make changes to improve conversion rates.

This visualization provides an intuitive UI for building a funnel analysis and visualizing the results. You can specify any number of conditions and steps, and will receive the following outputs when you run the analysis (see screenshots below):

- User Counts by Funnel Step Chart
- Conversion Rates Chart
- Abandonment Rates Chart
- Summary Statistics Table

It works on any table that Snowplow’s Data Modeling User has access to, including the atomic events table and derived tables. It comes with some pre-built funnels based on out-of-the-box Snowplow events such as page views and link clicks, and you can save your own custom funnels to share with teammates.



## Requirements

- Access to the table(s) you wish to run the tool on, granted to the role used when setting up the visualization

## Building a funnel

The following steps will guide you through building a funnel.

### Start from template or create a new funnel

The main page shows a list of example funnels for common use cases (e.g., ecommerce) to get you started.

You can choose to start using one of the templates, or create a new funnel configuration from scratch.

### Choose your funnel name, database schema and table

A page with the funnel configuration will be shown.

First, you are asked to provide a name for the funnel, which will be used to identify it.

Next, you can select which warehouse schema and table you want to run the analysis on. If you select the atomic events table, we have included some additional functionality allowing you to reference properties inside nested columns. Otherwise, you can only reference flat columns.

![Form to choose the funnel name, warehouse schema and table](/assets/images/table-and-schema-ff2f4be72fa5152009bf67fd3308a5fb.png)

### Configure filter conditions

To minimize query costs, we **highly** recommend including a filter on the partition key of your table. For the atomic events table, this is usually `collector_tstamp` or `derived_tstamp` for BigQuery, but may more recently be `load_tstamp`.

You can add filter conditions to select a subset of events to be used by the funnel using the partition or other columns of the table.

![Filter rules to select a subset of events to consider in the funnel.](/assets/images/filter-d29701e0f167471467c96813623df43c.png)

### Define funnel steps

Next, you are asked to define rules for the funnel steps. Each step requires a name and one or more rules, which can be combined together using conditional logic. You can define an unlimited number of steps.

![An example funnel step configuration.](/assets/images/step-f6915953b310a86243e2e0463693f598.png)

> **Note:** The **Update Steps** buttonWhenever you make a change to the funnel steps or the filter rules, it is necessary to click the **Update Steps** button.

### Check additional settings

There are a range of other options you can use to configure the funnel. Use the **Settings** tab to update the following configuration if you prefer (or keep the defaults):

- **Order events by**: this must be a timestamp column. If you have added a date range pre-filter, make sure that it is the same field as this one.
- **Group funnels by**: choose a field to group the funnels by - we recommend user or session identifiers.
- **Additional columns to group by**: use this to visualize additional dimensions in your funnel analysis e.g. experiment groups.
- **Max days since funnel start**: maximum days since the funnel's start for an event to still be considered part of that funnel. Set to 0 for unlimited.
- **Intra step time (hours)**: maximum hours since an event for the next to be included in the same funnel. Set to 0 for unlimited.

![Additional settings configuration.](/assets/images/settings-523e3b54a60ec3c88a280df2e062d057.png)

### Build the funnel charts

Click the **Build Funnel Charts** button to generate interactive funnel charts and tables. You can also download the generated SQL to edit or rerun in your own environment.

| User counts by funnel step                                                                                           | Funnel conversion rates                                                                                                |
| -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| ![Chart showing user counts by funnel step.](/assets/images/output-user-counts-ac7f31ea5f1fcebaae831992500f0d25.png) | ![Chart showing funnel conversion rates.](/assets/images/output-conversion-rates-3588ab1c1a6291f9a006ee8ba287f5b7.png) |

| Abandonment at each funnel step                                                                                           | Summary table                                                                                               |
| ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| ![Chart showing abandonment at each funnel step.](/assets/images/output-abandonment-b53bfa0feabbaeb163846dbc6128a778.png) | ![Summary table with the funnel steps.](/assets/images/output-summary-eaf18e7f7f49138a6a05440b11014a06.png) |

### Save your funnel configuration

Once you are happy with the settings of your funnel you can save it for anyone to open and use using the **Save Funnel** button. The funnel configuration will be saved under the name used in the **Funnel Name** input at the top of the page.

### (Optional) Output to a table

You can also save the results of a given run to a table in your warehouse in the **Settings** tab. Use the **Output results to table** checkbox under the **Settings** tab and choose the configuration for the output table.

### Export visualizations to a BI tool

If you would like to visualize these funnels in a different tool, the **Export** page contains instructions on how to run the generated SQL and recreate the analysis in the following tools:

- Looker
- PowerBI
- Tableau
- Preset
- Streamlit

## Editing or deleting a funnel

Once your funnel is saved you will find it on the **Welcome** page. You are free to make any modifications to them as you need if you click on them. Deletion is also simple, just click on the ❌ symbol.

> **Note:** If your underlying data sources (views/tables) change (e.g. fields removed) the pre-saved funnels might break, and you may need to edit or recreate them to avoid SQL compilation issues.

---

# Data model visualizations
> Overview of Snowplow visualization templates including ecommerce, attribution, media, and marketing analytics.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/

Along with data models, Snowplow provides dashboards and visualizations to help you extract value from your data quickly. These self-service analytics tools give you templated use-cases for data collection, modeling, and activation.

They are deployed in your cloud and aim to reduce the technical barrier, making data analysis more accessible beyond just SQL users.

![Pipeline showing data flowing from tracked events into visualizations](/assets/images/data-apps-pipeline-dcd6e0fa792a1663ec1ed612f217c477.png)

## Available visualizations

Purchase a Data Model Pack to get access to visualizations.

### Digital Analytics Data Model Pack

This includes the following dashboard visualizations, as well as the required dbt data models:

- **[User and Marketing Analytics](/docs/modeling-your-data/visualization/marketing-dashboards/)**: understand your customer engagement with digital channels.
- **[Marketing Attribution](/docs/modeling-your-data/visualization/attribution-modeling/)**: understand the impact of different marketing channels on conversions and traffic levels.
- **[Funnel Analytics](/docs/modeling-your-data/visualization/funnel-builder/)**: understand the sequential steps users take toward a specific goal, identifying drop-off points and optimizing the user journey for higher conversions.
- **[Video and Media Analytics](/docs/modeling-your-data/visualization/video-media/)**: understand engagement with video, audio, and streaming content, including clicks through to conversions and advertisements.

### Ecommerce Data Model Pack

This includes the following dashboard visualizations, as well as the required dbt data models:

- **All the visualizations** from the Digital Analytics Data Model Pack.
- **[Ecommerce Analytics](/docs/modeling-your-data/visualization/ecommerce-analytics/)**: understand your customer engagement with digital channels.

## Access

You can find the visualizations in the **Visualizations** section in the left sidebar of your Snowplow Console. A visualization can be in one of three states:

- **By request**: if you have not yet purchased a Data Model Pack that includes the visualization, click on **Learn more** and register your interest. A Snowplow Customer Success Manager will then get in contact with you to discuss getting access.
- **Available**: the visualization is ready to be set up. See the installation instructions below.
- **Live**: the visualization is ready to use.

Once the visualization is installed, clicking on the tile will launch it in a separate browser tab. By default, anyone in your Console organization will be able to access the visualizations.

If you wish to invite others to use the visualizations but without giving them access to the rest of the Console, you can [create a new user](/docs/account-management/managing-users/) with the `visualizations user` role. That user will then only see the **Visualizations** tab within the Console. The permissions can be managed in the [usual way](/docs/account-management/managing-permissions/).

## Installation

After purchase, click the tile in the Console **Visualizations** section to begin installing it and follow the steps.

An example tile:

![Visualization tile showing start install process](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAhcAAAD+CAMAAAC+09cZAAAC91BMVEX////y9Pfw6/m55/3w8vXm6OsQGCj9/f5nN7dnObf+/f/6+vrQ1d7z8/P39/hBJHf4+PklLTsTGyueoqhDSldhZnIaIjHT1Nf19fYVHSx/hI27vcEXHy8fJjba3N/q6+0/RVKipavt7vApMD/7+/zCxMhbYGwjKjnd3uGSlp0tNELh4uSZh7jP0dXv8PHZ2ty2uL6NkJhBR1Tz9PW8v8N8gIlOVGDVyO3Hyc2Hi5M2PUsxOEZpPLjm5+hKUV3LzdBRV2KKjpWCh480Okjf4OKPk5qbnqR6fogcJDOdoKZGTVkBL0QXHi78/P3o6uy+wMWXmqFWXWiym9zEuthUWmZma3ZTWGQrMkH9/P6kp61kanTExspobXiytbqtsLU6QU7x8fKoq7Bqb3leY27y8vPMztKws7iUmJ94fYbS1Nbl5ujj5OarrbOQlJuFiZHq5PbW19qmqa9wdH64ur89Q1Hp6euZnKPs7e7Ky89yd4EfTGHg1/Do6OrT1ttLeY52e4RITlsvNkS8vsK1t7yWmqA0YnfRyOLX2NxrdIhLL3+ih9RYXmmo1uyegdJ2TL9ncIX29Pvs5/fk5eZqPblvnbOFjJ7a292NlKRsVph1eoNJT1wHNky04/nj3PHb0+ydjLtsQLpYPYjaz+/WzuaSwNZmTpJnbHf59/2jk8BpcoZscXsMO1Dy7vmv3PKs2vCfzeO9stSTcs2wocmqm8Z6UcFyR72JdqyJjZRCJnjn4POMu9G4vMd/WcSSgLNgSI7g4+fGtOXS196/wsZwRLxFKXsqWG2mjNaNasqFYcd9qsCqr7ufpbJ8hJZ4gJN0fJB+gotud4tQNYI9aoAEMkcnLz35+fnZ3OFaiJ7RwurOxeCvtL+lqretr7RjkKWAa6V4Y59zXZxwWprx+v7P7v36+vzt7/K9qeGTmanBt9Zpl61TgZZEcoc5Z3yFs8l3pLoYRlsSQFbMvOiypctUOobKwd3Ivtuxmdu2qc5kk6lRfpOznd2rkdi6rdKmI8bjAAAjTklEQVR42uzUMQqDMBhA4RSULMU5W8AD5criYbxHkThY/OliN7/vDI+XXnClC351McDuu4thq23k6VrdhnMXw1xLTjxdLnXuYaTjFqpgl/sxji5qSbAr9dRFswu63E5djAm6URfoAl2gC3TBlS6I6IKILojogoguiOiCiC6I6IKILojogoguiOiCiC6I6IIb/t3FtKzvm9ZlSnzYu7uYNqs4juO/ePF76FMl7bNSWm1HKaWwkiJj0myFVQaMMcLLZhTK2AYIOKZjm1CcTTTsLcHUiDiymS3RLF64bNnNEo0xLnEvyWbm68U0i/Fy2b23euM5pwU65hC2Gi3hC01o+Ofc9NPznD5ckDUtykXuVa+Gx0zzXl2BkT0tysXXXmQg79d43AocWEKlE62Yn3miDKL1e3bjsRqbGEB6GycccvU39pzAsmhRLq5qyEDaVcxvFV0WiPIYwj+3qpNG5SnM9AztyScFJSzEg+WS7Zjfb7RD9Ck7sXD7DHrw8GysQnr1XAOggezCsmhRLp5ERnpwmTXkOoheZCX+sd00tntY4ph7zn7IxsnjeDAHeRTz66MLokJ/FRbuDMk38dAqOYT0qjkMYKDfP4hlURa58HMcWj2fSXPBIIAy5xJdLKYy8gjjS3OxnPo3XOiP6GJbpCmwYzVE3Ts6q0fkuzUe6/ut35MP2U7hYl0RB9Nd1AJ4mUkX3bVWt61QAwpjQ4MvVxYqF754LKLBsuGwu3azBa95AmQstlmMxIGRWF9Xszv0CkQNfmt1fMJjG0CytQx3scQMYK/n8MaXOgMjXgBaoa3pdu36lIuh2DaIhmN1+2NOFsUqveZmz14AjR3uI/0HJc2hmD38dDmyr0y70OVD1x/FxdM0Yla6TwHdLgaKaBccPOwkOQpZO52FHh5uTXNhZTeeNdxO6aKCLAqQG4AtLHGRVdKFL4/OXfC9S2fM4EvYS9VOMeKRixdRZBxVRxTRbfI1JAtzU28LjyG5LkURM1BFht1kcdLFMK0+wOJm4btUNQyQo3JK5rwGzUZXpZ0xC7KujLvQ5UPTl+7C3MStsMQ9fdCsrAJGeATwkOH9mwsApDaGkBdpLvawGf38wi5dbDfiwBDtrdhCWoc2Bx3kqGKBRrrLcKqJr2t7N7ElGCyYccHxU10ltAF15PETu12zLoJkKfrZn3RR/Up+nkAGrallH8yV7Ei6KHCyAuij2zEWLOJ4MB/KRZAceXVvEasxSD6LjR2eCWRdGXWhWOjnnrgY1XV9qS58du4sh+wgeQJ4Vr5IHpb4kKrCTdrLgHigeMaFt5Mb6O6VLlStfWSZdLE3eb6op/MYgH5eAJDHl2fPFykXVg04TVcrmhkC8OOsiyFWArvY0qtcNAIFBtdClTvCgHIhV/QDEW6ZPV8oF2toHALaI7WWN8lGM7KyzLqQHH6frLn+px6NCiFLvY7QaN7WBqwiazs6QuQxeJiHVEEn/R7WjzlcPDXjYp0Y5QYkXZz4osdJ+Q7dQiuUC1EjRFZWd3R0BBib7yJPLiMxqB0Kr8y4aLXyePIKoVysBhCWqhCMB0haUy66abStJkvvd5FHD5KZm0m7f3c23s/LuIuPJk2iG7qm6//sYjcZhOgI/QD2hQzSHcQeMqCqgIc7kWoHz2iHwuxZSxtmXZg76S5Puhg3WD9yOumic9ZFfQGAJroDstA8F2rxY2QDrPwx3cUzZNhmszUxolw0AIiJ4XWHyeah7SkXCstwFTtwv4t+NiOVY7iHZMyLrCvT15EDn5tUibuLcdGgzomYYGqT7u0rYQijZAFU6S5scnSsnuTRORfY13kcysV68hjQd7+LOhdtDqCS8TmJhvlvXDSrV7eOLEsZnOm1dBdr6W4ALsy52MP6ElZACdk/46KOdgtQtnZNK0QnNpGbkHVleL/ANzkmVU5CX8wyJby9tfy5EHkNpfH9FsDPHow5uQ3o9vf3prsYZ48X2tPkbUeaC5ly8RvtuWitvd/F0QqD77biNAMD8I37C4F8slB70EUV2TXwnD3lYp2d/atEjU6uTXcxwgjQVjTnwusiAxpEflaOpVy0k69DO0y775n4KsBcwnFkXZl1gfdyckw1ykVOQlvEMqMuqkY0bHQzXPeSwQ1AHZ07LrQwgnQXg3a6Q0VkC0OWv3FR4GTPaY8xzwW6yBG0iaWHPDTaAXOAZOQBF2+5KXKlXLxOrobMz3CaCzlaO9JpzLnAp+RmyAopKlYuLGfI8G0KUqNifn+ILEbWlVEX0RcECEFCuhAwootYpjvSRKNHbbT5tQZZtFkDfMNusqnqKSDC05gpv9klRkfzrRxJHQJcFqhusxBYX03W9xlGGTawByKLi+uBNQZ34WDEIG3XIGrYbmVEjERmFr9GYwxoyKvuOd2VOl/sYAiqowbLitXvcUYON5aQtcOsBkLSL9DHlkNQrREmiy0uaSB3pxjr3ApgV5g0wqPIvjLq4o70kLqOiK+ET1/EMlpZLlKZG3qRauNGPJivwYIFahvDQ7OU5mKBnu8bBLCV9lYsXFkv0tvOnfi7TqS0wFGahTe1Muzi47M5IlPNvfdSOm5o85b533aB7lI4mhnBkioln8eyLIMuDlyW24Rp+gaQkCoEjoQOPStcvGql0emkO4glVUUPlmeZc3H3G8Wi5uaUHtUTpmlTjui9c7qeDS7gHWq2VsbzsbTaGw9iefY4LvT0n32XTKqzF6HrejSRPGnUJKK6ng0uVsqECz06de/GleicDV37w6Q+hkx+KVRAwTApGom7WbFfrJQRF1M3xWt++fepGRe4krxrcfbOzInibqJGXVZqbkRXXGRfj+RCb02ozcE0eeccVAeuJ0+a96DrakKUEFTO5piuT624yL4exYWuX5kWKGQ1n93TADwxmbr57RMeoL7ljmGSTT+h6wu4MGNRDVT8jJUyVOZd6Hryj2M5suQJ4vKVqYs3c0zy+c27+mxy9oZ0MSk+nzzMhe+rk+e/r9AX80Hy/XVYKVNl3gWg6xc/MyUdiKSPzy6Jp6LLU1AioroelYMCxvXpyd910UNc3PphwpH//a8rLv53Lf06out/5igPcrNIPVTTH+my6B+XLz0BXSHSpj44hwVcnLwF4KuTKB90rO92AHAEv/oFaAsC5m4fMNhWPlhe3H4o5cLbfa0N+EVMHBITlu7svMWcFS3VhTpcqL3i8ieJafUhJIWk5k6SxSV57fgYmNGwkIuK7woge/udk9+e/CkX5d9/t+r8LXx93oK3359A7vlX5W9++KlAuZg4/+GH7wSlI9x6R0z8sPJ/dNL6j13gwF/snU1oE0EUgB8uspupiOBBb0LJQZCK4KEecuwt6MHTgle9ePGoZGlvgSJJG0lDD0klhEIIUUsgOWgKbQ21P/khtNiWBi9WWvpfL8XqxTez262BaDZtdV19HyQ7ebO3+XjvZTrb6D2mJwFMC+ZMMcKrstDirghkNcYBnZ96cXMwtdYtoxepMZArHbC2JkNXfxerdEOtXIPuMs48hrZInnvhruYB8pULd1KXoVTugNkiECZ2e5FELTjjYrFfZOJGRQmCqmJoSdGnC0MyU1nTbVO2UKyWHmO+wPHgsly9hddyHoqzcqW7zGZr+szyIPeiqx/rSXv/ApQ6zpbeFiHyjzza9VfSqhfqO71sZNwyF4Opr58qPJJJivSgFVzGwZycxlhzL5ALg4OHXtzv7+LXWeiIXI1A6Vrkkj5zq8y96KjKAO7eS1Cr5WevV2/20s/01WGnF+q43meOnjHrBBt6mstlXgBDQqPmAQxPVpOhiReXy934fq+XGV4ArxRtvU/gfG/xBizXsIkQM8Ui92KMW7PQ3w7dpcgCtqyDQNRhoxfiWK+ixBPmY2UMcSf1a3KKTxpqeHJDzbyAtcjC7YcRM1/Acmnh8mzlPo77H0JXag2w/bwld1cecC9YZO3KHUwu0FatyJBP5YE4wlYvbk/raz6uGs2kuBgw2BezSg4LCR8UNAz+0ou2WiVVql0/9AI/9/ZGxnCcLwGw8lvekRarqYtt3Au4Ppiqrl3mOhUBrqA5RB32efEiLlY+8wlt4PlBeGH6Ma5nir4hrU8RZmQTONukv7gJddw+D3WgMe8vHPUiDn16y3kcx4tR/dmQpWwh2KnCoRZMM77AjjNI9CkIV0Q94Xkt9IKwiJ11hDcXnaJ4JOIKMqqphhehrEskiX2GaGF9ozw8pJ7Mi4UIEFaxr+98EcxNJ0QNUUcVQXwqIXMtkhkXR5kKidKh9eli5BLOOMdHnHBfy80Q46CeANuNJRnYXRwgoyFZeMGe9/E5jyebIC8cSKteGN9A1ERYOMFf/KyvNu7y8EHuOag4z81I5EQh8WST5IXzONa5HIBQVs8V+guFCIveQvksrNHFGAorPBZ/Rl44j+Od44NvisCwQwerxr5bnxaw51lxwvMReeE8juUFdMaFD1PJzqm4IYZoNNyGNgZBBSEvnMixvEjqeaLAV1ybFuWCM5o0jKj34vXPvfi4Ip02Kx+BsMmLd4pglXHc2rTQIhzW8FNLXnyUfgckhk1edHrE6b0p1EDFl8yWMh7FFf/MOIBYrSMr0u9gBYg/7oUMENL/eFYYAo5QQ00Egwmo84JZ8EL6PQBhw/lO9Z3eZo6bEaTBE6vkhZNpvY5ocbFVMZ2EpgRd5IVDadmLEO5voxe5M4yRF/8urXrB9vV9zq8MTjVfxHY38N23yMcj0W09gLyJvpQOtiSDgXSAXyaj0bRvXZIC0WERPohyNskL27xgSx6xsZlRwYQZQB2sNS9moumYJC3u8vEXHIoA98L7UurxSwa+6Lq4LI6MrHt3pIBX96JncgQJkBe2eRGadiEeT/CuFfBmq15sLe7OYKYQCz15YAbqvRiOrvuEFx/4XT1HXvipjtjqBXvn+hlKIyx7EZgfWfTzxd4QcpiBei8ONl9Ghw0vAtEB8qIhdpzjK/DFFijWsOjFl0lpZD6ADcQeFhOfGaj3IpbeljZ7uBc+/+Q8Dkwv9uaQGHlhjxcMnoUVpFmiqE8YnmcWvOBd5CRmgVfzbyTfFzNQ78XMXkzaTsdwcnNmp2dv+8gL3wBC+cKufMFCfeZ6W8sb4v9fNPdi2Oufm/PxPDHXMzwfMAP1XvgxLWx5d4z+YtNPdeQIe71gqwWXwOM61ML1KyuQ+Cpr7kXP5ACCuQLzwYb/x8CRF2jBB4xt+Q0vNnzkxRE2e8GePe9shUQCq0hTL2LpGX7Z2sTh3vzMDwHDi1dI7INPTy0j+D31VWAnvYFebPMZ44aX5IV9XjBg1gHAlwUvdtIxfQ8Dl3YDxz8EhBdezvouZgl978vn9c77B2LoBSdt3OAnL+zxwoBBi9A+uPM4hhdAXvz7kBcEeUE4wgs6x/fX8ie8oHO/zuNPeEHPCTgPS15MyHAKyBNAOAVLXoy1wynQPgaEU7DkxfmJdhlOiNw+4cTfKf9fseQFnB+bOHdCJsZICwdR7wVBkBcEeUGQFwR5QQjIC6IR5AXRCPKCaAR5QTSCvCAaQV4QjSAviEaQF0QjyAviO3tnstNGGkXhs7q1sEB2YTseA9h4wmIwYGRsGYPNFMQoMQgS5ikEghiakEhICSQSC3ZZROo8AIu8QPb9Ar3pZR6m/VeVr1100U2nG+ihjhRUw63/DvURV+Q6ipFMLkwZyeTClJFMLkwZyeTClJFMLkwZyeTClJH+i1yMmf///j9Pt+OiZW7uoEmCkR6TF4pqqBp/rKVV6DU+TD/jFrplhreN+D3ZPFB1FALsDoOIBYrjL4oz3KUCEy6D7YHovXLxis4yI/7kXXBh77xYfgAu7OHbcNHUZcV3iDPcoTrkJv32VgewPvLhnrloRsP7yODfz8UOLQAPwIWl4TZc1FMVvkOc4S7lurYdTPLBe+UCaKUUgNeJ3n0UJaXyi1UKF17PolXctdqBwTllGNsv8wIWjn0UO07MASFPIqpw8SkxkIWi1HN6E7NoV3AgMC5+topD9T8BVZvJGpcug8i/4HlyCuDJxqOvE7UKognP0xIXAxupva/b2O/vV27z8mby1anCRSr/VZS026px8aLm501oan6V3N0UXCzPJTfbgQEHJby8xzr5NjixDsASyie2RdFWUec516l0ITIEYqfqbCCtDL5afrUPoaYYgGzslEfIRTx53To4pmsO1vPi0NQFtpq3LlAb+whgawPxWABwLeRDDRDb2zF/d2x5NLbN4+fsd84Fgglgb2Rmyv0cWE7bHJkRb5GLtsYp36yEGirYJ909ErAlT6755iWOLZ7K2AbxgeyTv/hXgXnnpV3eg9BuD3U7qvkKNVBM2e0CgmdAlXyAQMEf9qWbyxmErD1+R5u8ANjabJc29ycgNXS2FgxqXNgKZ5d+Z9fQTCO9B2ozZ1fpSAqwZWzhgm9cMKH+yfkyPTQPRbXFPOkgxRE9a7xqdGbxpY3CA7xXUtzmDPuDWUhhv6NbHsBjOgLG5BWuU+lCrP6Y3qmzwaU88zkt10NogiTgNeV4hFyELSMvPdY1Z+mmmbORc0iT7vBUhg7wiEIA3P3ooBY0DaUd/uBHse11+AqOWis94fFz9rvnovsSzY0DwIRzGXP0GNLScDF7WEKKNlBDNcAmLeAp1QPjtMKxxVMxYMx5COwPPUOfCPxRrobQNj0FX6EGCp2SF01OeR9eqsVU2oXjka5yBqGOYBMQ/iyGGYDV+RKoW6pG1VKJC7sVzW7aB9ocQGfaVRxxwVLEpRmWuiXmot22KuE9hSCUdMeBVYqja8mFBtue9jnCe5rsbQFURb6gV34M5N1R2J8BNU5XRZ0xKKvzbFrpAJgjPRc8Qi7CdlYLfXMeZzFkJoMF2gUWrnPxbLYBWXdebKufI4ILHiZnv3MuLu3ip+XjOHXAS/3taLCqzxfSSL/66W/1HcLjrgIQnOFY9dQ3GlefL9bdjiwkq8Rc8BVKoKq6JPbWunvxsx1W9wCUKXEGTe3xr2RVHxl6ZnBK+fLzhXq0exiAJwKXLPbqKace7qLTEhchQU5DRL3pheHy80UgPmVXueA9TcvUD+B8F3ZxKFrcG/Bb0DOvq1PlgmdzKFeJC/Vc8Ai5CFGdvrk6McWLWMOh3A4EaEXPhWfkfBQWi44LHiZnv3su6r4AoTYaitA5kPD7wgtQs8PZq03j8wx6xHixmuFY9VRCdqlcwNtI7yasYC74igou8j9gaeVVHdq6sEGKGssZFA3MUsRf4mKyE6/p/DoXMz2CAj9SCpJxWlEPhyhX4uIlKXoGoRGPxoX17ZBs82lc8J6mFOUgNDoiOETjF5y4vc1uL9fJXJRnM1UHAy60EXIRzAU3N1oEQFGPHQZctE+7I4dN0HHBw+Tsd85Fs7MeO875PrRQBwDL+bS8d52L4TWEC1B+VzlWPTVAWf73SFMy8q7MBV/BXChzkwMv5Au5qRiy9aioi3IGoU3aqsWTCi4uaPcmLh7RNwWHcfXwOF2UuNiSQ2LtOIQihxoXP/p+csFR4oL3ShW/hyK/CK8WdFyuHtgauE4DLubPDLjgEXIRzAU3538ORWsZIy6A5YNOt1fHBQ/z/rh4MxTHIl0Ar8W97gIwHazkIgusO/fwlfoAl3uQY9VBhegAkDKreHq4LiYeZS74igouRiP2TsA+ZZOKJ+bFCKVyBqE3EQlIVHBR7RPIfTHiotp5BSBPzbAdiiNuV4mLcaoBYIWi7jMAKxRHWqDXo3CxDt7T5HL/CKA/gc5ZhbFWYNE/9RZcpwEXvaLBrMbFEUXFdTkeIRfBXHBzw2kArVejexQHxsQzm1wPWGWVi+QuIM1Oa1wMKlzwMO+Hi47U3IwzBTTR/PbmLO2inrayr886K7mwpz6tuXfQHJmJ7jyLRDlWG1RdIdXyhp7hsbz6qWXS6WIu+IoKLjBPr4B+WgWQH/H05YYrMght0cv9iRHKMhcYdH7rG5CNuECvc/fEG3kO2OQP0c2hN/zcKb0Lvsq2Bp9CqJW6+o5sFMfl0ELTj5QBcrRby3vSlBdCSX8iXiPXI0X500d1w1bA6qeNcp0GXIz5Li9y3RoXVuezTx0ZyvEIuQjmgps7ojefvMU7nx0Jx1sm6QDobNvITZLKxbS/dWxT7lW5mBmOWgUXPEzOvjJ/d1yQXPdc9I6aJYoMyHngZYQiV/EKLpwTToocAWixkzycK8dqg9opHp4Xz+7jGfJNhsBc8BWVXOzSDhClTQANXU4amrZyBkWWQRulu2i8zIV1Wib7qiEXUu8Q/TIoAba3wySvtTMXOHG4ydZfuh1OCorfzB2HkyavZGsRBVrjvWpaUTMnfWSbEBzNkjMcQFFfCijXacAFQr+Qz6Nxgfo0+ZOU4xFyEcwFN4eDCDkPrcC4jUbeCi5SneR+7le5cD1zUiGpPXe22mhfcMHD5Oyes3v53mxMgqpjCXpJWe1IVVVlLOukdNebrWDxFcbihXmDD9bimlyBm1c4RSn3tUyWU7AaStuuUlCtlfcWKQpVo9nSWYtBncbKNnTQOHhyvKEvwqC54wZtAUtABfPEUhE2BpZl7IZhpuf/ld+n/lu0eoXv1djnbUR7Isv4SxJcfIfizn2TizuUqxrfKykvRyj4Ex6EC5z8O9+/+F+oKpSrxl9Uw9Mx/IH+S+/lmPoHyuTClMmFKZMLUyYXpkwuTBnI5MKUkUwuTBnpf8/Fd7gybB7cQs10gL+qAeVLzDvwiNxC/xkubvZmBLoew0ALNTe6Mjji/rjgFrha4d4w4mKr4ztWZV/I7fUf4eJmb0aUWmGg6bobXRkccV9ccAu6al0w4iKY/O7BuHB7PQgXDSFPLK6zJAgzyHsvOx1U88VxrLkydjH5IYvxfP1yhUfiaPP4Q9cn9mYIsW9EDWn10GqNOJjoypVTnsTqgrF9xfeBj/XJA6vOOiKkRrCLA1IukX+qcFGO0rlaPljmPE8lEbk7OBHXuHi/CCD3ASVzBhfC7g69vUSpVm2g6N4QXBS7rgUbPVRjB7fPqxa1PdFe9MNYrd9+PrJwHm0wR4vbnpTwiHAb7Lq5QQ/BhbTmvrT5NyosCVvCDBKcLzsdajNF80WaohzbQum6HnfhR/+k3zdW9kg4zs560lSveTNUaY4LLSQ2RcNvgFf0bkl+wil3HLN+Rwh2B5Dy2edtP9SWrSOq1Ah2ccBDdXbqAmxLtslyVKWr5V1dt3wFIOye+tyocRHuBpDwoWTO4ELY3aG3lyjVqg1YSZRbKKxFGrP8IqZq7OD2eVUBmrzUWVyiO9Pj/gGcRx2Mw+bM7HZQC7fBrpsb9QBcvJSjsLR1ly0JY75VYQaZLzsdBuU+YJWiHNtCzwEvzQofx5OyR8LhywGdU/rPEeG44BD1b+bX8hzwfKiZUyqfEoKLZf8VcBz5wtYR3ecIuzi8tAIkaOdaVKWr5QPQO2RBK+0CBzSn50IzZ3Ah7O7Q20tEtVoDKhcOIOt7y1wonyPcG6+qcEHvgZ/FBUfUwnnUVR0jKaDIBbfBrpsb9QBcfF4DMCFb2JLwTWzgh+my06FgB4qdRzlWeWncIosR2NfKHgmHXZu7notqcIg66T2bBWK4nJK56KBzwYBPYutIJRfs4kg6LcB6rO96lN7VIt7LPnS7+PmCuSiZM7gQsQ6L7SXMRbXGhTKY7oKeC+6NV1W5sIoz4+LKAc6jcWGHwgW3wa6bm/QAXEhOUhTnVwZfyusqF2zj8CVVLjhW4QK+hBhTmD0SasMx2YALDlEmPUWK5jglc5EQtxEH9IJf7azkgl0cnRoJ16P0rpYmSqHHDgMuNHMGF8JcsKHkZi72fHouuDdelbnIUQqQaILz6LngNth1c6PunwsMzT8Ssla8/L2jcsFOh19WVS44VscFeyRu5oJD1ElfFpRlAr/lYoJOAXTJ1YZcsIsj/IMxF+xqYS4cBSMuNHMGF8JcsKHkZi7yET0X3FtpVQMuOI+eC26DXTfGehAuhoMSACv4JqVoDpAK02WnQ2cjgF2KcqyOC/ZIVHKxruOCQ9RJ5ynKKXVcqAZMDNfBkAt2ceSpFuhzPL4exa4W5qJLQP5C4yIZBJD3oWTO4EKYC5295Ldc7IqG37HRQxg7uDde1YALziNWZS64DXbd3Kz752KcHBsth2WzSMkMMl92OrTSnjBfRDlWxwV7JJgL1Zvh6WcuyiGWobcfUetvW3gxEQyUU/YP7S8LLtB5lqqdcC+UuXg09QKAGsEujmNnz9OmzsbR61HsamEujn3hltxnmtNshlvRA98ISuYMLoS50NtLlGoruYi87+ulRTZ6qMYO7k1b1YALziNWZS64DXbd7M7BSA/CBTaD5H4Xr+Ai3kbyfGa6wukQc5LtkKIcq+OCPRLMherNsH9hLsoh6PJFgEdtRIXxipTRNA0oXDQX49JzKHPhpY8A1Ah2cWAjTXL3C31UpQOGuUAoSM6Xsjrv0aSNCs9GUDJncCHMhd5eolRbyUVslnxfwUYP1djBvfGq17ngPMqqzAW3wa6bnmEY6f65YCOFXieu5ZF8pdOhIQuPbNXHGhs1eMlTud4wZH1Z/KwKQK/saClObx15E9RFsIujud04akyC8cq8uN6coS9Eby/hass6tmgTsrCxg3vjVQ3EeXh+3EaF60aSYKSH4kIvKbwoBZKUAis0OYbXwR78KR2k2/E36Kz+llGm7pgLbA4NuX1blVBPkZ/sO/hzCuDv0Mkto0zdORewNIUC0Om0Y1+Cqf+G/m3vX5gylMmFKSOZXJgyksmFKSOZXJgyksmFKSOZXJgyksmFKSOZXJgyksmFqV/Zu38WxYE4jOPycCDccv6uvBdgOd2+Dqt5Ib4A+0DKdGl2gwQhxRZiIMTCVL6BFBY2/umsdZdtrrjT5ATDbGRT3GTh+RQTRcsvmXESjAm7IBN2QSbsgkzYBZmwCzJhF2TCLsjEVhddm3gX2H1Wuuh2B/0Ha/oDlnGXjS66g94LLHrpDRjGHTa6GPRgWW/QoVoWuuj2X2DZS58njHo2uniAdQ/sop6NLn7Cup/soh67IBMbXVhfdgI9dlGPXZAJuyATdkEm7IJM2AWZtKgLrfEhdvHFNe9CiSh8hF18cc260Eop8ZUorWDELr64Zl3IWXnQuDVELW+EOnGEUDns4rNa0YUSpZQ+v9BK1E0UkZ/myRBYP+PWaF0Es5BnVIQOriZjTGXGLj6rFV1o0f/WnZUuEtnMknQLLLe4FYiDMyfyULF32UXbND1fQPsioqtdjHMAW/c5TrMYGM2iEJjHzvF42Ipa46/XeP4Uh4fg/MY7RA4wy5bxCMP3QLOL1mjShSrWFr4SH1puwthJ7AEIM3EzrHN3LAuEkss+yWUcl/OII2N3KRFGJ3eS+t7ETbP1PFtm6YRdtEWDLrSocvBFF8PV81hOwQjFPBLtR9i481Cy9VM5j5Rd7Lx57iOWA6JsfZlH9P6AWDS7aIlGXfjloESXw5V3nMhyWnSB4SHwxQnPCd12EQA7F69pFo+u64t1tJGIXbREw3lEwxflF/OJwq3X3PUuXbwvs6SuC+jtMk28SxdhfkrYRXs0X3eKiF9dd47SHYCNhJcuJrmHoKaLt9+YJ3K4dLEQB2/sojUadwFoff3JerVLF7+j5clDNp5ikx6PubwVXbyLCitdBBI5ibwhSd+fZrKY+hKwi5Zo2oW+lKFUdV8ryeWUOEDsyjDcp+5G4qKL+UQWlS68xJUsAl7HMvU2bprIhl20RMN9rTOgOFQ8lYXMzzFUPzB9uTwMh7w+0h6Nuig3wP1iO9yM182+NF5nJ96XQ7yPj9gFsQsyYRdkwi7IhF2QCbsgE3ZBJja6WMG6Fbuoxy7IxEIXnf4jLHvsd6iWjS66q2+w6htPF/dY6WKw+vX4w5rHXyv+3+89NrrofLf9/+DfO/R/mLto5wMFOvRZfP4InbELMmEX9KddO7ZBGAaDMAoFygrufikDef8FsgcxEZaLa6HJezN8uuoSXZDogkQXJLog0QWJLkh0QaILEl2Q6IJEFyS6INEFiS5IftpF94zisvWli2oPGFotXRxlMBi2OmYXZxh7NWWwtdrPLGYXYzH6i7vrYy1mF58wYHh+u4CFLtAFukAXXHTBX7wBTc5qwpvS0SEAAAAASUVORK5CYII=)

During installation, you will need to provide a connection to the warehouse you would like the visualization to use. It will also highlight any required pipeline and data model dependencies.

The installation workflow will look something like this:

![Visualization install process](/assets/images/sample-app-install-8c42522fc02777615519062089af612d.png)

### Data model dependencies

Generally, visualizations will depend on data models. If there are dependencies, the installation flow will highlight which models are required and what models you currently have [running via Console](/docs/modeling-your-data/running-data-models-via-console/). It will also highlight any properties that you need to enable or configure for these data models.

> **Note:** If you are running the necessary data models yourself outside of Console, then you will need to manually check that your setup satisfies the requirements for each visualization. These requirements are listed within the documentation pages for each visualization.

## Warehouse connection

To install a visualization, you will need to create a connection to where your data model output or atomic table resides. You can do this as part of the installation process. The visualization will need secure credentials to fetch the required data.

You might have already set up a warehouse connection for loading and/or modeling the data. Visualizations, however, require a different connection, as you will often want them to run under a different user/role and with different permissions.

The connection process will look something like this:

![Visualization warehouse connection process](/assets/images/add-connection-d2b0cce3fe5e1fd0f0f0e921c721adf0.png)

Once you have selected a destination, provided the credentials, and run the suggested SQL script, the Console will test the connection. Upon a successful test, the visualization will be available to use.

## Using the visualization

### Is it running?

When the visualization is doing some calculations, querying the database, or otherwise still loading, you'll see the following gif in the top right:

![Gif of stick people running, swimming, etc.](data:image/gif;base64,R0lGODlhgACAAPAAAAAAAAAAACH5BAkUAAEAIf8LTkVUU0NBUEUyLjADAQAAACwAAAAAgACAAAAC/oyPqcvtD6OctNqLs968+w+G4kiW5omm6sq27gvH8kzX9o3nKcD3vA5s+Ia9oNFATAKOQKWSiXM6oTbpkzqzTrEx7ZX78ibBYfGYzDKf0So1kZ12/+Ar+Zy+s+Pr7j1f7ffnFRinRegiddg1pNho5OMIKbhGZpXXl6N3YrdlwvlF8ik6SioZWoqaauap2ur6xvoqm4oya1t6easL2Lbra1j2+5sp3ApWjKmI7Bghy1zh/DwRLd38Wi1Bje2gvc3Q7a0AHo4wTo50fb5gfs5O7h4O7y2/TY9tX40vrf/Mz+wfKZ26BAAbFVQmcOCBg4cYEnIYCKIfiXso4rFIByMcnY1sOKLxWCmhwgCWhCQaufDkN5UoS67s1JKlOJkKXa6jOdDmTJgjdRLEqc5nOaDtiJI0Gs+o0KBKkc5ryrMmVFA9p1KqGnVoVqZb0XV9ZxUWyqNflxYt67ReWEZjyVL9+TUp2rhP577NudZUTLtXpfIVu/duSrpq/7Id6zRtYcAvGQfWy60vYsgPDre9jDmz5s2cO3v+DDq06NEnCgAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/JM1/aN5/rO9/4P1AGGxGGwV0wSjzmlE8C0PZ/R2XRahV2vWdcW22V9qeHV2FlWnclp0xrddr+T8dN8WZff8/o5v3/2BzgmOMhVaAiHaAe2qEbnGCk56XBHSLlgaYl5oOmJ6Rk6GUoaSVrqeIpaqLoq2OqaBxtbNyv6eljZyprbAMvL5vuLG8wwTKxofIxcBLGM2Oz8zKm8S/0wfZ0wq62r2i1sDa6QPc49Ti6O3lkObFR9Opq8rc6sBB/v3kj/rb/PXi/Om0wB2/jh10/WQQTtumxCaGsekocMI2758QmixfZiNxbi28ixxpoIIC91NCmtZC+RXyiobCkFpsuX/2TIrEAz5IubOHNK3Hkxg897NIJuGIrHptGjSIuubPLUS9STNaVWpapTzFWsP1tM5UpUadaYW82U1bA05dhHa4V+JXg2RVoPc721RVGXQ96PkMTetbAX7t9EST8EFtzV69sLh9PFlbu4Z2SNff0OVps43GXChd1OphzNaWNsPEl+xotS8umKj9mOdvwadGcrqUnXlhAbciDbt3Gv1j0QoEfVmbUMRfvbtUqmm82+7NDaOEjDYSdaDBEayPV1wmlxz8dd88jwvnOTP48+vfr17Nu7fw8/vnz4BQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/IG1PaMc/a+537FC9Z+xIfwWEwmjkylkgl1FqFN6Y9atc6wWW2Mi/TKwELxmMwzf9E99YvddrvgQ/kcbofh828yf431FxK2ECjowcVAeKgDxsjC9phCJ2lCV1c5eImZ2bEZ16nxmRaaMRpUenFaljqxutja8NoVqzAbVWt7i5trsJuYOxuAVivsS1xqfBDZqYxAKfkquyko7TDKtxpxasctge3m7foZTq56KUdtCq2G3ojcvofomC4PQpVtr0ma594ryu4fBn8CB+orCOQgwnHMYuH7ppAGLxQRn1WkcFGiuv5p8Axu9GRNUcOE2t450+UH4y6AvwBcSwmx5bmWnFAa2kazJoScoGxOfMnTZUxgPn8uuQkUqUV6RpQmpXXUaFFWOB9yhFoV6zKpS7U+hRUVLEOvw8g6HUt1as+ZadXq7IpKo1ircuMW8srVAt1jYte1hcvP7T1aeT3+5Xu4bGKWacn6tRsWMuAbIrr0BXlYquPH/DZzXqs482WTpEZjDrx17uJ5cU2TXmvUM+MesmdTnnw7NeoRGU/Dhur69V4SYGOv/tCxRHHgx1kTtfTXuGTiMKFPl76bd3Xl0ZlP187U+m7soCtvp37de3bz4bmnX/6dfXvw49WXH1wmuO3ciK/fx0fOin6fvaVZc3VRVhtboBX4n29tJEhSfH1BqBJVFGa1Xn9CiWSgXosUZliGEYoYImoXfnXfiCR62J2AO53IYYcYZgfiUDLOuCKO/OnmoltvHZjjVZed9WKNJTYo5HHPFQmjjin6uONdRPJo5ICmzZckTRQF9WOWK00SlIK33CHTkeK0UOZ+H5FZknMEnWGOfOf5EJB7VeKw5EJ67slnn37+CWiggg5KaKGGMlIAACH5BAkUAAEALAAAAACAAIAAAAL+jI+py+0Po5y02ouz3rz7D4biSJbmiabqyrbuC8cyBNT2jHP2vud+xQvefsSG8AgoKhFI5HLZPD6LUefUV5Vec1nhFtfVfmFh75hc5p3R6do61na/X/HknJ6+s8P6fbUPlwUo8zc4I2aYqLjI2Ehk5rhSGGlSRilSd+lRJ6eZwdnjCQQaJCpBimi6gNqkqsAq6GoAG6tKy2d6aymqi6vZu3sJ7BuZN0sr3JWgSzm5zOwYxTAc3bpKzSh9ja2o/To8ZOj9DB7eN34AGtAGiH6cyXnnvp75Hj82f5+u/pVfz8TviruAAPU9GWiwYJwtCBc6+HfQmkJjD9gJtELOYoTpYFMw7oO40VlHSBMpjiLZL9Q3hxtUZgMpa0LCmCFh0qw48+ZDgjp3kuqJExXQaciGZmRltKTQpPSgGf3DsmcsjVLHUb0p0iTWhsqqSlTq8Y1LCrWOfj0zj+hZs6lSElsZFm5chjBFbpt7sZw5I2vz6rVDo+/Iv4CDtqX795TgwXoV4/XLLfBjyL1kLkbrtObhdj/JXhanVTPKX5/5TvZUGENappI3szY8+nXr2LJ9uq59lzbu3KV2i+7tGzbw4KZvE29qnPjq48h1M19+HLry0sGlV6d+3Tnz5mO3y+3kvWL48eTLmz8PoQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJpurKtu4Lx/JM1/aN5/rO9/4PDAqHQ4DRWDseiQalc8l6PoXSKgBllQKz1RJX2/taRWLwrjz2oM25dbfjdvLibAxdrrvXK3pou49nAeiHM6h0YXiVl4jEZxjG6Dj4EzmRSARo+cgUoCdxydlEFwEa2jn6UGp66gYxuYqA2vAKeyDL0FercLtwp9sb5+D7uxs820oMjHz8lSxs/PzmHL32STjNDI0Nl7vNseltpxquWUluzdh4npp+uJ7drv4eGz81b1u/R56fts6f5e8fwHACm20rKAYbwjLOulFD03BZOYYRDYqzOI3Lt/aB++x98HgvpMiRJEuaPIkypcqKIDVw3NDPBUREMwVRTDEMnUR22khk4ukJKK+P4/AVFQWO28KlP2EyferwItSpPSVRvXqTJtatGjNw/RrTZldltBKYKzZWa1iyQ9EGfdhSbFx4NUm1xSXNZSAK1dRinCjvxlqpc1caPow4seLFjBs7hqTP7169kXFm5VsXc0IVb13dxVs1RFO6oc1GBXGW3lFWZTeCBav0NWzKsl/Trv31Nu6tundf7V2Y9WjVrYkPBhzcaGfQy9kmt1vZbV+d0z1HB0w4rebNVgPbOC73+ePx5MubP48+vfr17Nu7fw8/vvzGBQAAIfkECRQAAQAsAAAAAIAAgAAAAv6Mj6nL7Q+jnLTai7PevPsPhuJIluaJphPAAuoLI+3cxrZJ5+7Ne/q/6wkvQN3wSCn+kEyHEtiMJp5LqZVqtEaxVS2Sm/UewTnxlzwzn9HqNbk9RgfhPTY9jr27n3omv99UBKiVNmh4iJiouMjY6FhX9hijJJnCVVnyhhkit/khx+LJARoqmkFaY0qEOqcqwVrq+gobK+tEW2u7gJuqu8ub62sA3Cs8TBxsi1wovMys7PysGh05TV29eQ1Vqf2H2N35jXyMKg4sw3o4jl5ueK4Aq07bEO+efts+eI9PKp/PUM/eP3j7BPYDWFDcAU0EBzZiyO4gNzC/JE681DBcNvCKGSFi8kgO5MU8HUmaEmnHGsYpHF2hXOmyZURKul7S/JjM5raRrUJSqdjlUZiZ3ohKY7TTqKCSyRTdVJp0ITakRaXCtNrUXFWsWwMUo3qV69OFkjSyNLnRLNShZR36HPsw4NmfQuchROs07Aq9+vjuxdsX8CrBd2RuMFzYLwbEbRiPcutH8WHIQxx/ogyJ7omEeOBmwmxDswrOmaOOBv3C82mLQkzDkNt6amjYpXvOpm1srcjcXnnxZopa1rrfvd8RL47ruNjg0HATd36ceW5Qyh+krE5PMva3Qbfz6+79O43wSdiShyD7PPqv6tsbKgAAOw==)

You may particularly notice this on visualizations with multiple tabs per page, as the tabs will load in order so the last tab may seem empty until this processing is completed.

### Configuration

Where the visualization has some configuration requirements it will also have a **Settings** page that will validate what is available to the visualization, and provide information for steps to take for any unfulfilled requirements.

### Chart sources

Many of the visualizations support the exporting of the SQL used to generate the charts. In some cases, there may be a specific download button, but otherwise look for the ![](data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTYiIGhlaWdodD0iMTYiIHZpZXdCb3g9IjAgMCAxNiAxNiIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZmlsbC1ydWxlPSJldmVub2RkIiBjbGlwLXJ1bGU9ImV2ZW5vZGQiIGQ9Ik03LjcwNjA3IDExLjI2NkM3Ljc5NDMzIDExLjMwOTEgNy44OTM5MyAxMS4zMzMzIDggMTEuMzMzM0M4LjE0NzI3IDExLjMzMzMgOC4yODM3OCAxMS4yODQ5IDguMzk0NDIgMTEuMjAzM0M4LjQyMjMyIDExLjE4MzYgOC40NDg3OSAxMS4xNjIgOC40NzM0OSAxMS4xMzg2TDExLjgwNjggNy44MDUyNkMxMi4wNjY4IDcuNTM4NiAxMi4wNjY4IDcuMTE4NiAxMS44MDY4IDYuODU4NkMxMS41NDAyIDYuNTkyNiAxMS4xMjAyIDYuNTkyNiAxMC44NjAyIDYuODU5MjZMOC42NjY2NyA5LjA1Nzg3VjIuNjY2NjdDOC42NjY2NyAyLjI5MzMzIDguMzY2NjcgMiA4IDJDNy42MjY2NyAyIDcuMzMzMzMgMi4yOTMzMyA3LjMzMzMzIDIuNjY2NjdWOS4wNTc1NUw1LjE0MDE2IDYuODU5MjZDNC44NzM0OSA2LjU5MzI2IDQuNDUzNDkgNi41OTMyNiA0LjE5MzQ5IDYuODU5OTNDMy45MjY4MiA3LjExOTkzIDMuOTI2ODIgNy41Mzk5MyA0LjE5MzQ5IDcuODA2Nkw3LjUyNjgyIDExLjEzOTlDNy41NzgwNiAxMS4xOTEyIDcuNjM5MDIgMTEuMjMzOSA3LjcwNjA3IDExLjI2NlpNMTQgMTIuNjY2N0MxNCAxMy43NjY3IDEzLjEgMTQuNjY2NyAxMiAxNC42NjY3SDRDMi44OTMzMyAxNC42NjY3IDIgMTMuNzY2NyAyIDEyLjY2NjdWMTEuMzMzM0MyIDEwLjk2IDIuMjkzMzMgMTAuNjY2NyAyLjY2NjY3IDEwLjY2NjdDMy4wMzMzMyAxMC42NjY3IDMuMzMzMzMgMTAuOTYgMy4zMzMzMyAxMS4zMzMzVjEyLjY2NjdDMy4zMzMzMyAxMy4wMzMzIDMuNjI2NjcgMTMuMzMzMyA0IDEzLjMzMzNIMTJDMTIuMzY2NyAxMy4zMzMzIDEyLjY2NjcgMTMuMDMzMyAxMi42NjY3IDEyLjY2NjdWMTEuMzMzM0MxMi42NjY3IDEwLjk2IDEyLjk2IDEwLjY2NjcgMTMuMzMzMyAxMC42NjY3QzEzLjcgMTAuNjY2NyAxNCAxMC45NiAxNCAxMS4zMzMzVjEyLjY2NjdaIiBmaWxsPSIjODA4MTg3Ii8+Cjwvc3ZnPgo=) icon. Click it to download the SQL used to make that chart.

Note that some data is processed further after the query to get it into the format required for plotting, which may include actions such as filtering, pivoting, etc.

### Help

The visualizations provide help text throughout: keep an eye out for the help icon to get more context or help in using some functionality.

### Log out

If you wish to log out of the visualization, you can do this from the sidebar. Note that this also logs you out of the Console.

---

# User and Marketing Analytics visualization
> User and marketing analytics visualization with dashboards for acquisition, traffic sources, retention, and engagement metrics.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/marketing-dashboards/

The User and Marketing Analytics visualization contains all the visuals you need to perform a high level analysis of your web and mobile performance. This includes dashboards covering:

- Reports on user acquisition
- Information relating to your traffic sources
- Insight into user retention
- Measurement of user engagement
- Deep dives into technology and user demographics



## Requirements

- [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) enabled

- [Campaign Attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) enabled

- [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) enabled

- [Referrer Parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) enabled

- One of:

  - (**Recommended**) Running the [Snowplow Unified dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) with:

    - `snowplow__enable_yauaa` set to `true`
    - `snowplow__list_event_counts` set to `true`
    - (optional) Conversion event(s) defined, with `snowplow__total_all_conversions` set to `true`

  - Running the [Snowplow Web dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/) with:

    - `snowplow__enable_yauaa` set to `true`
    - `snowplow__list_event_counts` set to `true`
    - (optional) Conversion event(s) defined, with `snowplow__total_all_conversions` set to `true`

- Access to the derived tables granted to the role used when setting up the visualization

## Usage

All charts have help text to explain any definitions used, and have the SQL used to produce them available to download by clicking on the icon in their title.

### Filters

The data queried and displayed can be filtered using the filter in the sidebar. Note that these filters will apply to all visuals across all pages, and changing the filters will cause the dashboard to refresh all visuals which may take a few seconds. By default the last 30 days of sessions and page views are used.

The comparison option allows you to compare two different time periods, we provide default suggestions such as the preceding period matching the starting day of the week, but you can select a custom range. Note that it is possible to select a shorter or longer period for comparison than your main filter; this can lead to unexpected results in charts as for example bar charts will use the full range, but line charts will only display up to the main range. You will get a notification if your ranges are of different lengths.

Some charts are only related to new or returning users, not both, so depending on your filters these may be blank.

### Settings

All configurations can be found in the **Settings** page. Note that all these settings are global for all users, meaning if you change them they will be changed for everyone.

Configuration options:

- Select the currency symbol used throughout the visualization
- Toggle between using campaign or channel for reporting aggregation
- Use tables produced by the web package instead of the unified package, and select which tables to use

---

# Video and Media Analytics visualization
> Video and media analytics visualization with dashboards for watch time, audience retention, and in-media ad performance.
> Source: https://docs.snowplow.io/docs/modeling-your-data/visualization/video-media/

While 91% of businesses use video as a marketing tool, few businesses deeply understand how their users interact with this content and what effect this has on business metrics. Collecting vital engagement data plays a significant role in understanding your audience, optimizing your content, and measuring the impact of your video and media marketing efforts.

Our Video and Media Analytics visualization helps you visualize insights on your video, audio and streaming content, based on data generated by our media plugins and [Media Player DBT package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/). It comes with the following dashboards:

1. **Overview**: get a high-level view across all your media and video content performance including "total watch time", "plays over time", and "top-performing content".
2. **Content**: dive deeper into individual media items through an impression to plays funnel, audience retention chart and views over time graph.
3. **In-Media Ads**: improve advertising performance with key metrics like click-through-rate and see what content returned the most ad views and clicks. Know how far viewers reach through your ads with the Ad Audience Retention chart, to understand where in the ad viewers may be skipping to improve ad creative performance.

### Requirements

- Running the [Snowplow Media dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/)
- Access to the derived tables granted to the role used when setting up the visualization
- Media events tracked using the media APIs for the [JavaScript (Media plugins)](/docs/sources/web-trackers/tracking-events/media/) or [mobile trackers](/docs/sources/mobile-trackers/tracking-events/media-tracking/). Note that there are multiple JavaScript tracker media plugins available. Implementation of a v2 plugin is required for this visualization to work as intended.

## Usage

The following steps will guide you through using the visualization.

### Configure

The **Settings** page lets you choose the schema and tables in your data warehouse. These refer to the output of the Media dbt package.

First, choose the warehouse schema where your dbt Media package produced the derived tables (e.g., `dbt_media_derived`). The visualization will suggest the most likely tables. The following derived tables will need to be configured: `base` and `ad views`.

Verify that these match the desired tables and click **Save Settings**. If the tables have the required columns, you can navigate to the dashboard pages in the sidebar and explore your data.

### Explore the dashboards

You can browse the following pages:

1. **Overview**: key metrics across all media, including number of plays, average play time, and number of unique viewers. It also shows a table with details and metrics for all media items.
2. **Content**: choose one piece of media to see metrics such as total plays and total watch time, as well as charts for plays over time by platform, an engagement funnel, and retention across the media playtime.
3. **In-Media Ads**: ads engagement, including impressions over time, clicks, and click-through rate.

### Choose the date range and filters

At the top of each page, there is an option to choose the date range. The **Content** and **In-Media Ads** pages also have filters for individual items of media. These options let you narrow down the data shown in the dashboard to what you are interested in.

---

# Set up classic alerts
> Set up cost-effective classic alerts for new failed events and daily digests using internal telemetry data.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/classic-alerts/

Snowplow can send two types of pipeline alerts to help you monitor failed events:

- **New failed event:** receive an alert within 10 minutes of a new type of event failure being detected on your pipeline.
- **Failed event digest**: receive a daily digest of all failed event activity in the previous 48-hour period.

To receive alerts you must have the failed events [monitoring](/docs/monitoring/) feature switched on in Snowplow Console.

## Subscribing to alerts

- Login to Snowplow Console
- Locate the pipeline you wish to set up alerts for in the left-hand navigation
- Click on the `Configuration` tab, then the `Pipeline alerts` section

![](/assets/images/image-febe5bcb4732a53fa5a1b94da9e47175.png)

- Click `Manage` for the alert you wish to subscribe to
- Add one or more email addresses by typing them into the input and clicking `Add recipient`
- Once you have added all recipients, click `Save Changes`

![](/assets/images/image-1-fb3fab59f7b5cc71a4e25c13ff7e4c21.png)

---

# Create failed event alerts in Console
> Configure email or Slack alerts for failed events with custom filters and thresholds in the data quality dashboard.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/creating-alerts/

Set up failed event alerts to receive notifications when failed events occur in your data pipeline.

You'll need access to the [data quality dashboard](/docs/monitoring/#data-quality-dashboard).

To create an alert, go to Snowplow Console:

1. Navigate to **Data Quality** in the left sidebar
2. Click **Manage alerts** in the top-right corner
3. Click **Create alert**

![Create alert form](/assets/images/data_quality_create_alert-afeeedce7f41da59235c86224c07e133.png)

## Configure destination

Choose how you want to receive notifications:

### Email notifications

1. Select **Email** as destination
2. Add recipient email addresses
3. Click **Add filters** to configure filters
4. Configure triggers as needed
5. Enter alert name (e.g., "mobile-app")

![Email destination configuration](/assets/images/data_quality_create_email_alert-d7a41f749266f8b2922ae2bf52587159.png)

### Slack notifications

1. Select **Slack** as destination
2. Select Slack channel from dropdown
3. Click **Add filters** to configure filters
4. Configure triggers as needed
5. Enter alert name (e.g., "web-app")

![Slack destination configuration](/assets/images/data_quality_create_slack_alert-5fb2a749236fc0c5ee189561f1518d36.png)

When no active Slack integration is found, a `Connect with Slack` button will appear instead of the list of channels.

![Connect to Slack](/assets/images/data_quality_connect_slack-19cd34be33c56ff687d46d51edee16a4.png)

A Slack consent screen will appear.

![Slack consent](/assets/images/data_quality_slack-619e081be9bee27143d1b537cda3ec1d.png)

To select channels in the UI, first add the app to those channels. In Slack:

1. Open the channel where you want notifications
2. Type `@Snowplow Notifications` and send
3. Click "Add them" when prompted

![Invite to Slack](/assets/images/data_quality_slack_invite-9dfb992f8c26e95209f7343bf6f215f0.png)

Once a Slack alert is configured you will see a confirmation notification in the selected Slack channel.

![Slack confirmation](/assets/images/data_quality_slack_confirmation-6bc7271ab78f4dd95a12da4fc7251319.png)

## Set up filters

Configure when alerts should trigger:

1. **Issue types**: select `ValidationError`, `ResolutionError`, or both
2. **Data structures**: choose specific data structures (all versions will apply)
3. **App IDs**: filter by application identifiers

![Filter configuration](/assets/images/data_quality_filters-3a3e756dad9c7851f98b9e71d6f72796.png)

## Configure triggers

Set up when alerts should be triggered based on failed event conditions:

### Trigger types

Choose from the available trigger options:

- **When above value**: Set an absolute threshold for failed events (e.g., 1,500 failed events per hour)
- **On any issue**: Alert when any failed events are detected

![Trigger configuration showing trigger type selection](/assets/images/data_quality_trigger_types-a758467a4f7b9abd6106b58ebb2bb4ce.png)

### Threshold configuration

When using "When above value" trigger:

1. **Value**: Enter the threshold number of failed events
2. **Time period**: Select the time window (10 minutes, hour, or day)
3. **Deliver**: Choose notification frequency (daily, weekly, or monthly)

![Value threshold configuration with insights panel](/assets/images/data_quality_absolute_threshold-9ac189a9c8981ce3e989281a77e5ed9a.png)

#### Insights from recent data

When configuring a threshold alert, the Insights panel displays data from the last 7 days to help you choose an appropriate threshold:

- **Average events volume** - Shows the average counts of valid and failed events (per relevant time unit)
- **Suggested threshold** - Provides a recommended threshold value based on recent patterns
- **Apply suggestion** - Click to automatically populate the threshold value field with the suggested value

Use these insights to set realistic thresholds that reflect your actual event patterns.

**Alert delivery:** If the trigger condition is met, the alert is sent once per delivery interval you selected.

## Complete setup

1. Review your configuration
2. Click **Confirm** to create the alert
3. Your alert will appear in the alerts list

## Alert frequency

Alerts are checked every 10 minutes. You'll receive notifications when new failed events match your filter criteria.

---

# Failed event alerts
> Automatically notify via email or Slack when validation errors, resolution errors, or data quality issues occur in your pipeline.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/

Failed event alerts automatically notify you when [failed events](/docs/fundamentals/failed-events/) occur in your data pipeline. Set up alerts to receive notifications via email or Slack when validation errors, resolution errors, or other data quality issues arise.

- The alerting system monitors your failed events and sends notifications based on the filters you configure.
- Alerts are checked every 10 minutes and sent to your specified destinations when matching failed events are detected.

These alert destinations are supported:

- **Email**: send notifications to one or more email addresses
- **Slack**: send notifications to specific Slack channels

## What you can filter on

Configure alerts to trigger only for specific types of failed events:

- **Issue types**: select `ValidationError`, `ResolutionError`, or both
- **Data structures**: filter by specific schemas or event types
- **App IDs**: filter by application identifiers

## Getting started

1. Navigate to the data quality dashboard
2. View your failed events overview
3. Click **[Manage](/docs/monitoring/alerts/failed-event-alerts/managing-alerts/)** alerts to set up notifications
4. **[Create](/docs/monitoring/alerts/failed-event-alerts/creating-alerts/)** and configure your first alert

![Data Quality Dashboard overview](/assets/images/dq_manage_alerts_button-6db5c87cc08cc32cdc8e7e7fac30b3b4.png)

---

# Manage failed event alerts
> Manage existing failed event alerts by editing filters, changing destinations, or deleting alerts in the data quality dashboard.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/failed-event-alerts/managing-alerts/

This page explains how to edit, delete, or review existing failed event alerts.

## View alerts

1. Navigate to **Data Quality** in the left sidebar
2. Click **Manage alerts** in the top-right corner
3. View all configured alerts with their destinations

![Manage alerts interface](/assets/images/data_quality_list_alerts-0f080579a99d99787076ffd84fbb3862.png)

## Edit an alert

1. Click the arrow next to the alert name

2. Modify destination, filters, triggers, or recipients:

   - **Destination**: Change email addresses or Slack channels
   - **Filters**: Update issue types, data structures, or App IDs
   - **Triggers**: Switch between "When above value" and "On any issue", adjust threshold values and delivery frequency

3. Click **Save** to update

## Delete an alert

1. Click the arrow next to the alert name
2. Click on the three dots button
3. Click **Delete**
4. Confirm deletion

## Alert behavior

### Trigger frequency

- **Threshold-based triggers**: Alerts are evaluated against the specified threshold and time window. Notifications are sent according to the delivery frequency (daily, weekly, or monthly) when conditions are met.
- **"On any issue" triggers**: Alerts fire immediately when any failed events are detected, subject to the configured delivery frequency.

### Multiple notifications

Alerts trigger when new failed events match your filters. You may receive multiple notifications for the same failed events in the following scenarios:

- **Rolling window detection**: alerts use rolling time windows to detect failed events. The same failed event type will continue triggering notifications as each consecutive window passes and detects the events again.
- **Overlapping alert configurations**: Multiple alerts may capture the same failed events when their filter criteria overlap. This results in duplicate notifications, especially when alerts are configured to send to the same destination (same Slack channel or email address).
- **Delivery frequency**: Alerts respect the configured delivery frequency to prevent notification spam, sending at most once per the specified period when trigger conditions are met.

---

# Set up failed event alerts
> Configure alerts for failed events using warehouse-based alerts or classic alerts with email and Slack notifications.
> Source: https://docs.snowplow.io/docs/monitoring/alerts/

Snowplow can alert you when a new failed event has occurred. There are two different implementations available to choose from:

- [Failed event alerts](/docs/monitoring/alerts/failed-event-alerts/) run queries directly against the your data warehouse every 10 minutes. This approach keeps warehouse workers continuously active, resulting in higher compute costs compared to classic alerts, but provides faster detection, are richer alert context, and are more configurable.

- [Classic alerts](/docs/monitoring/alerts/classic-alerts/) offer cost-effective monitoring by executing queries against our internal telemetry data warehouse tables. This approach minimizes your costs since the computational load runs on our systems.

## Feature comparison

| Feature                                       | Classic alerts | Failed event alerts |
| --------------------------------------------- | -------------- | ------------------- |
| Can alert on new failed events                | ✅              | ✅                   |
| Can send a digest of failed events for a week | ✅              | ✅                   |
| Notify via Email                              | ✅              | ✅                   |
| Notify via Slack                              | ❌              | ✅                   |
| Filters                                       | ❌              | ✅                   |
| Does not affect pipeline cost                 | ✅              | ❌                   |

---

# Access raw failed events from S3 or GCS buckets
> Retrieve and query raw failed event payloads directly from S3 or GCS file storage using Athena or BigQuery.
> Source: https://docs.snowplow.io/docs/monitoring/exploring-failed-events/file-storage/

On AWS and GCP, when failed events are generated on your pipeline, the raw event payload along with details about the failure are saved into file storage (S3 on AWS, GCS on Google Cloud).

> **Info:** If you followed the [Snowplow Self-Hosted quick start guide](/docs/get-started/self-hosted/quick-start/) on GCP, you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) to save failed events into GCS, as it’s currently not included in the Terraform scripts.

## Retrieving raw data

You can directly access and download examples of events that are failing from file storage. This is useful for further investigation, and also required to design a recovery operation.

**AWS:**

Login to your AWS Console account and navigate to the sub-account that contains your Snowplow pipeline. Then navigate to your S3 storage buckets.

You should find a bucket with a name ending in `-kinesis-s3-bad` and within that a folder with your pipeline name e.g. `prod1`.

![bad bucket](/assets/images/failed-evs-s3-1-84031c8dbe10799d17c6f929c7d5fd54.jpg)

![prod folder](/assets/images/failed-evs-s3-2-5d381d124833dde43bdaff289eacd9f2.jpg)

Navigate into this folder and you should see `partitioned` (search if it isn't visible).

![partitioned folder](/assets/images/failed-evs-s3-3-002938da54b3c9eb577de32ce9e4ae42.jpg)

Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.

![failed event folders](/assets/images/failed-evs-s3-4-76b13fa30ce9af8ec672e4b5e23f5a36.jpg)

You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.

![failed events](/assets/images/failed-evs-s3-5-08e23c59d143d1cb880d89e84e16a35b.jpg)

**GCP:**

Login to your Google Cloud Platform account and navigate to the project that contains your Snowplow pipeline. Then navigate to your Google Cloud Storage buckets.

You should find a bucket named with a prefix of `sp-storage-loader-bad`.

![bad bucket](/assets/images/failed-evs-gcs-1-ba52c0737b95e4a2f54777cfccef3df2.jpg)

Navigate into this folder and you should see `partitioned` (search if it isn't visible).

![partitioned folder](/assets/images/failed-evs-gcs-2-a91853d0d0df4501d25a7f4d3a964281.jpg)

Within this folder, there will be a subfolder for each type of failed event. Select the relevant type for the failed events you wish to find.

![failed event folders](/assets/images/failed-evs-gcs-3-46e49ccef31512c9375deafa5f115c96.jpg)

You can now browse the folder using date and time to find a batch of failed events that occurred on that date / time period.

![year folders](/assets/images/failed-evs-gcs-5-c30a145ebcd28f00ab3afdb08e2a7a2f.jpg)

![month folders](/assets/images/failed-evs-gcs-6-25555744ddb50b7026df16975f0350f6.jpg)

![failed events](/assets/images/failed-evs-gcs-7-26a0e6a98aea028185da83ff11409ca9.jpg)

***

## Using Athena or BigQuery

[Athena](https://aws.amazon.com/athena/) on AWS and [BigQuery](https://cloud.google.com/bigquery) on GCP are tools that let you query your failed events, using the cloud storage files as a back-end data source.

```sql
SELECT data.failure.messages FROM adapter_failures
WHERE from_iso8601_timestamp(data.failure.timestamp) > timestamp '2020-04-01'
```

This approach is handy for debugging your pipeline without the need to load your failed events into a separate database.

Before you can query this data, you need to create corresponding tables in Athena or BigQuery as we explain below. Each different failed event type (e.g. schema violations, adapter failures) has a different schema, so you will need one table per event type.

## Creating the tables

**AWS:**

Go to [the Athena dashboard](https://eu-central-1.console.aws.amazon.com/athena/home) and use the query editor. Start by creating a database (replace `{{ DATABASE }}` with the name of your pipeline, e.g. `prod1` or `qa1`):

```sql
CREATE DATABASE IF NOT EXISTS {{ DATABASE }}
```

Then run each SQL statement provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/athena) by copying them into the Athena query editor. We recommend creating all tables, although you can skip the ones you are not interested in.

> **Info:** Note that the SQL statements contain a few placeholders which you will need to edit before you can create the tables:
>
> - `{{ DATABASE }}` — as above, change this to the name of your pipeline, e.g. `prod1` or `qa1`.
> - `s3://{{ BUCKET }}/{{ PIPELINE }}` — this should point to the directory in S3 where your bad rows files are stored.

![Creating a table in Athena](/assets/images/athena-create-table-9b949019dd6014dd9c28e45da9f37ba2.png)

**GCP:**

> **Info:** If you followed the [Snowplow Self-Hosted Quick Start guide](/docs/get-started/self-hosted/quick-start/), you will need to manually deploy the [GCS Loader](/docs/api-reference/loaders-storage-targets/google-cloud-storage-loader/) to save failed events into GCS, as it’s currently not included in the Terraform scripts.

> **Note:** These instructions make use of the [`bq` command-line tool](https://cloud.google.com/bigquery/docs/bq-command-line-tool) which is packaged with the [Google cloud SDK](https://cloud.google.com/sdk/docs). Follow the SDK instructions for how to [initialize and authenticate the SDK](https://cloud.google.com/sdk/docs/initializing). Also take a look at the [BigQuery dashboard](https://console.cloud.google.com/bigquery) as you run these commands, so you can see your tables as you create them.

Create a dataset to contain your failed event tables:

```bash
bq mk --data_location=EU bad_rows_prod1
# Dataset 'my-snowplow-project:bad_rows_prod1' successfully created.
```

The `--data-location` should match the location of your bad rows bucket. Also replace `prod1` with the name of your pipeline.

Next, download the table definitions provided in the [badrows-tables repository](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery) in JSON format.

> **Info:** Each table definition contains a `{{ BUCKET }}` placeholder which needs to be changed to the GCS bucket where your bad rows files are stored (e.g. `sp-storage-loader-bad-prod1-com_acme`).

Now run `bq mk` for each table definition in turn. Use the `--external_table_definition` parameter so that BigQuery uses the bucket as the back-end data source. Here is how to run the command for the first three tables (note that you should change the dataset name `bad_rows_prod1` to match the dataset you just created):

```bash
bq mk \
  --display_name="Adapter failures" \
  --external_table_definition=./adapter_failures.json \
  bad_rows_prod1.adapter_failures

# Table 'my-snowplow-project:bad_rows_prod1.adapter_failures' successfully created.

bq mk \
  --display_name "Schema violations" \
  --external_table_definition=./schema_violations.json \
  bad_rows_prod1.schema_violations

# Table 'my-snowplow-project:bad_rows_prod1.schema_violations' successfully created.

bq mk \
  --display_name "Tracker protocol violations" \
  --external_table_definition=./tracker_protocol_violations.json \
  bad_rows_prod1.tracker_protocol_violations

# Table 'my-snowplow-project:bad_rows_prod1.tracker_protocol_violations' successfully created.
```

Run the corresponding commands for the remaining table definitions. We recommend creating all tables, although you can skip the ones you are not interested in.

> **Tip:** BigQuery has an “Auto-detect” feature to automatically generate the table definition for you by inspecting the file contents. So you might wonder why it is necessary to provide explicit schema definitions for your tables.
>
> There are two potential pitfalls when using the autogenerated schema with the Snowplow bad rows files:
>
> - _Optional fields_. BigQuery might not “notice” that a field exists, depending on the sample of data used to detect the schema.
> - _Polymorphic fields_, e.g. `error` that can be either a string or an object. BigQuery will throw an exception if it sees an unexpected value for a field. Our table definitions use the `JSON` data type for these fields.

***

## Querying the data

**AWS:**

As example of using your Athena tables, you might start by getting counts of each failed event type from the last week. Repeat this query for each table you have created:

```sql
SELECT COUNT(*) FROM schema_violations
WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
```

![Athena query](/assets/images/athena-count-3c29355bdfad7e01cc24cb3bd09105b9.png)

If you have schema violations, you might want to find which tracker sent the event:

```sql
SELECT data.payload.enriched.app_id, COUNT(*) FROM schema_violations
WHERE from_iso8601_timestamp(data.failure.timestamp) > DATE_ADD('day', -7, now())
GROUP BY data.payload.enriched.app_id
```

You can do a deeper dive into the error messages to get a explanation of the last 10 failures:

```sql
SELECT message.field AS field,
       message.value AS value,
       message.error AS error,
       message.json AS json,
       message.schemaKey AS schemaKey,
       message.schemaCriterion AS schemaCriterion
FROM schema_violations
CROSS JOIN UNNEST(data.failure.messages) AS t(message)
ORDER BY data.failure.timestamp DESC
LIMIT 10
```

**GCP:**

You can query your tables from the query editor in the [BigQuery console](https://console.cloud.google.com/bigquery). You might want to start by getting counts of each failed event type from the last week. This query will work, but it is relatively expensive because it will scan all files in the `schema_violations` directory:

```sql
SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
WHERE data.failure.timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY);
```

You can construct a more economical query by using the `_FILE_NAME` pseudo column to restrict the scan to files from the last week:

```sql
SELECT COUNT(*) FROM bad_rows_prod1.schema_violations
WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY);
```

You can repeat that query for each table you created in your bad rows dataset.

![BigQuery query](/assets/images/bigquery-count-06422ef48c38bef0a72bc39b1b739bf9.png)

If you have schema violations, you might want to find which tracker sent the event:

```sql
SELECT data.payload.enriched.app_id, COUNT(*) FROM bad_rows_prod1.schema_violations
WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
GROUP BY data.payload.enriched.app_id;
```

If you have tracker protocol failures, you can do a deeper dive into the error messages to get a explanation of the last 10 failures:

```sql
SELECT message.field AS field,
       message.value AS value,
       message.error AS error,
       message.expectation AS expectation,
       message.schemaKey AS schemaKey,
       message.schemaCriterion AS schemaCriterion
FROM bad_rows_prod1.tracker_protocol_violations,
UNNEST(data.failure.messages) AS message
WHERE DATE(PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', LTRIM(REGEXP_EXTRACT(_FILE_NAME, 'output-[0-9]+-[0-9]+-[0-9]+T[0-9]+:[0-9]+:[0-9]+'), 'output-'))) >= DATE_SUB(CURRENT_DATE, INTERVAL 7 DAY)
ORDER BY data.failure.timestamp DESC
LIMIT 10;
```

**Digging deeper**

You might notice that the `error` field in the result of the query above has the `JSON` type. This is because depending on the variety of the failed event, the `error` might be a simple string or a complex object with additional detail.

For example, the “invalid JSON” message might have this `error`:

```json
"invalid json: expected false got 'foo' (line 1, column 1)"
```

In contrast, in case of a failure to resolve Iglu server, the value in the `error` field would look like this, with “sub-errors” inside:

```json
{
  "error": "ResolutionError",
  "lookupHistory": [
    {
      "attempts": 1,
      "errors": [
        {
          "error": "RepoFailure",
          "message": "Unexpected exception fetching: org.http4s.client.UnexpectedStatus: unexpected HTTP status: 404 Not Found"
        }
      ],
      "lastAttempt": "2021-10-16T17:20:52.626Z",
      "repository": "Iglu Central"
    },
    ...
  ]
}
```

You can figure out what to expect from such a field by looking at the JSON schema for the respective type of failed events, in this case the [tracker protocol violations schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-0). The mapping between the various failed event tables and the corresponding JSON schemas is [here](https://github.com/snowplow-incubator/snowplow-badrows-tables/tree/master/bigquery).

BigQuery has a variety of JSON functions that allow you to extract data from within complex objects. For instance, if you are interested in Iglu repositories that failed to resolve, you can use something like this:

```sql
SELECT DISTINCT(JSON_VALUE(message.error.lookupHistory[0].repository))
FROM ...
WHERE ...
AND message.error.lookupHistory IS NOT NULL
```

It’s also possible, although unwieldy, to reduce all `error`s to a single string:

```sql
-- Unnest individual messages for each failed event
WITH unnested_messages AS (
  SELECT message, CASE
    -- resolution errors
    WHEN message.error.lookupHistory IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.lookupHistory[0].errors)
    -- event validation errors
    WHEN message.error.dataReports IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.dataReports)
    -- schema validation errors
    WHEN message.error.schemaIssues IS NOT NULL THEN JSON_QUERY_ARRAY(message.error.schemaIssues)
    -- other errors
    ELSE [TO_JSON(STRUCT(message.error as message))]
  END AS errors
FROM bad_rows_prod1.tracker_protocol_violations,
UNNEST(data.failure.messages) AS message
WHERE ...)

SELECT JSON_VALUE(error.message) AS error
FROM unnested_messages,
UNNEST(errors) AS error
```

In the future, we plan to simplify the schemas of failed events so that they are more uniform and straightforward to query.

***

---

# Load failed events into warehouse or lake
> Load common types of failed events to a separate table in your warehouse or lake to analyze them easily.
> Source: https://docs.snowplow.io/docs/monitoring/exploring-failed-events/

> **Note:** This feature is available since Enrich 5.0.0 and works with:
>
> - Snowflake Streaming Loader
> - BigQuery Loader (since version 2)
> - Lake Loader
> - Databricks Streaming Loader

For the common failures (validation and enrichment), you can configure continuous loading of any offending events into _a separate table_ in your warehouse or lake. This way, you can easily inspect them and decide how they might be patched up (e.g. with SQL) and merged with the rest of your data.

> **Note:** This feature is not retroactive, i.e. only failed events that occur _after it’s enabled_ will be loaded into your desired destination.

## Format

The format of the failed events loaded into your warehouse or lake is [the same as for your atomic events](/docs/fundamentals/canonical-event/). All the standard fields are present (unless themselves invalid — see below). This allows you to query and aggregate this data easily, e.g. if you want to see the number of failures per `app_id`.

There are two differences compared to regular events.

**Invalid data is removed from the event.** This principle applies to all columns:

- Any invalid standard column (e.g. `geo_country`) will be set to `null` instead of the offending value.
- Likewise, any column containing the JSON for a self-describing event (`unstruct_...`) will be set to `null` if that JSON fails validation.
- Finally, for entity columns (`contexts_`), if one entity is invalid, it will be removed from the array of entities. If all entities are invalid, the whole column will be set to `null`.

For more information about the different columns in Snowplow data, see [how Snowplow data is stored in the warehouse](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

**There is an extra column with failure details.** The column is named `contexts_com_snowplowanalytics_snowplow_failure_1`. In most cases, it will also contain the invalid data in some form. See the [next section](#example-failed-event) for an example.

## Example failed event

Here is an example of what the `contexts_com_snowplowanalytics_snowplow_failure_1` column might contain. Note that a single failed event might have more than one error. In this case, there is a required field missing, and also an unrecognized field present.

```js
[
  {
    // timestamp of the failure
    "timestamp": "2025-01-15T14:12:50.498148Z",

    // failure type and failure schema version
    "failureType": "ValidationError",
    "_schema_version": "1-0-0",

    // the component where the failure happened
    "componentName": "snowplow-enrich-kafka",
    "componentVersion": "5.1.2",

    // the schema of the offending event
    "schema": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",

    // any properties which were invalid
    "data": {
      "invalidProperty": "This schema doesn't have this property"
    },

    // there can be multiple errors per event
    "errors": [
      {
        "keyword": "required",
        "message": "$.targetUrl: is missing but it is required",
        "path": "$",
        "source": "unstruct",
        "targets": [
          "targetUrl"
        ]
      },
      {
        "keyword": "additionalProperties",
        "message": "$.invalidProperty: is not defined in the schema and the schema does not allow additional properties",
        "path": "$",
        "source": "unstruct",
        "targets": [
          "invalidProperty"
        ]
      }
    ]
  }
]
```

## Configure

To use this feature, you will first need to enable the stream that contains failed events in the [Snowplow TSV format](/docs/pipeline/enriched-tsv-format/) suitable for loading into your warehouse or lake.

The instructions below are for Snowplow CDI customers. For Snowplow Self-Hosted, you will need to configure this manually via Terraform.

> **Note:** An additional stream (Kinesis, Pub/Sub or Event Hubs on AWS, GCP and Azure respectively) will be reflected in your cloud infrastructure costs (unless you are using [Snowplow Cloud](/docs/get-started/#cdi-cloud)). That said, failed events are usually a tiny fraction of all events, so this stream will be minimally sized.

Open the _“Pipeline configuration”_ section for the desired pipeline and select _“Failed events stream”_.

![enable failed events stream](/assets/images/enable-stream-bf8168f51498a883da91d0108d639a4d.png)

Click _“Enable”_ and wait for the changes to take effect.

Now you are ready to add a loader. Click _“Add failed events loader”_, which will take you to the destinations catalog.

You can use the following loaders with the failed events stream:

- Snowflake Streaming Loader
- BigQuery Loader (since version 2)
- Lake Loader
- Databricks Streaming Loader

Pick your desired destination and follow the steps in the UI, selecting _“failed events”_ as the type of events.

![loader type selection](/assets/images/loader-type-242d12262466aff8ea2061d94f51907f.png)

Note that as with any other loader, you will first need to create a connection to your warehouse or lake, and then the loader itself.

> **Warning:** Some of the problems that cause failed events could lead them to contain personally identifiable information (PII). For example, a validation error could stem from PII placed in the wrong field, and that field might not be pseudonymized, leaving PII exposed in the error message. Or the [PII enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) itself might have failed.
>
> For this reason, we strongly recommend loading failed events into a separate schema (in case of a warehouse) or storage location (in case of a data lake) compared to your atomic events. This allows you to restrict access to failed events.
>
> Keep this in mind when creating the connection.

---

# Monitor failed events
> Monitor and manage data quality with tools for tracking failed events, data quality dashboards, and Console API access.
> Source: https://docs.snowplow.io/docs/monitoring/

[Failed events](/docs/fundamentals/failed-events/) are events the pipeline had some problem processing. Snowplow pipelines separate events that are problematic in order to keep data quality high in downstream systems.

We provide tooling for monitoring, [loading](/docs/monitoring/exploring-failed-events/), and [recovering](/docs/monitoring/recovering-failed-events/) failed events.

Snowplow offers two different ways to monitor failed events in [Snowplow Console](https://console.snowplowanalytics.com):

- The data quality dashboard that surfaces failed events directly from your warehouse in a secure manner, making debugging easier
- The default view

You can also use the Console API to monitor failed events.

## Monitoring dashboard comparison

The data quality dashboard and the default view are both useful in monitoring for failed events. Which one to choose depends on your requirements. This table summarises the differences:

| Feature               | Data quality dashboard                | Default view                                 |
| --------------------- | ------------------------------------- | -------------------------------------------- |
| **Requirements**      | Deployed Failed Events Loader         | No additional infrastructure needed          |
| **Detail provided**   | Complete failed events JSON           | Aggregated data with redacted error messages |
| **PII security**      | Data never leaves your infrastructure | Redacted data flows through Snowplow systems |
| **Warehouse support** | Snowflake and BigQuery only           | All warehouses                               |

## Data quality dashboard

The data quality dashboard allows you to see failed events directly from your warehouse. Your browser connects directly to an API running within your infrastructure, such that no failed event information flows through Snowplow. The discussion of architecture is important as it highlights the trade-offs between the two ways of monitoring failed events. The aforementioned API is a simple proxy that connects to your warehouse and serves the failed events to your browser. The connection is fully secure, using an encrypted channel (HTTPS) and authenticating/authorizing via the same mechanism used by Console.

In order for you to be able to deploy and use the data quality dashboard, you need to sink failed events to the warehouse via a [Failed Events Loader](/docs/monitoring/exploring-failed-events/#configure). The data quality dashboard currently supports Snowflake and Bigquery connections.

![](/assets/images/dqd-architecture-40884e74674c075988a723be4d45b4e9.png)

In this setup, you expose an additional interface (Data Quality API) to the public internet, and all failed events information is served via that interface.

The user experience of the data quality dashboard is similar to the default view, but offers substantially more information to support resolution of tracking issues. The overview page looks as follows:

![](/assets/images/dqd-overview-5f84e3168038e22581b570180c86214e.png)

As in the default view, it is possible to change the time horizon from the 7-day default, to the last hour, last day, or last 30 days. You get a corresponding overview of event volumes, both the successful and the failed ones, for the whole time period but also split per day/hour/minute in the bar chart right below.

Complementing the graphical overview, there is a table that lists all errors (at this moment only validation and resolution errors are supported). You can quickly see, for each type of failed event, a short description of the root cause, the offending data structure, first and last seen timestamps, and the total volume.

Clicking on a particular error type will take you to a detailed view:

![](/assets/images/dqd-details-845806f880ee86b5a6418a243e41a9e7.png)

The detailed view also shows a description of the root cause and the application version ([web](/docs/events/ootb-data/app-information/#entity-definitions), [mobile](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/)). It provides a sample of the failed events in their entirety, as found in your warehouse.

Some columns are too wide to fit in the table: click on them to see the full pretty-printed, syntax-highlighted content. The most useful column to explore is probably `CONTEXTS_COM_SNOWPLOWANALYTICS_SNOWPLOW_FAILURE_1`, which contains the actual error information encoded as a JSON object:

![](/assets/images/cell-content-444ee01b40d0096460610ead840479f8.png)

Finally, you can click on the **View SQL query** button to see the SQL query that was used to fetch the failed events from your warehouse:

![](/assets/images/sql-query-b5c2f610878be42fa64bc59788c373df.png)

### Missing warehouse permissions

When deploying a loader with the data quality add-on (API), you may encounter permission errors that prevent the dashboard from querying your warehouse.

**BigQuery:**

If your service account lacks the required [permission](https://cloud.google.com/iam/docs/service-accounts) to create BigQuery jobs, you may receive the following error:

- Error code: `21xxx`
- Description: `Missing permission 'bigquery.jobs.create' on Bigquery...`

Check if your service account has the [required role](https://cloud.google.com/bigquery/docs/access-control#bigquery):

```bash
gcloud projects get-iam-policy <PROJECT_ID> \
  --flatten="bindings[].members" \
  --filter="bindings.members:<SERVICE_ACCOUNT_EMAIL>" \
  --format="table(bindings.role)"
```

To fix the error, grant the required role to your service account (recommended):

```bash
gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
  --role="roles/bigquery.jobUser"
```

Alternatively, if you need more granular control, create a custom role with only the `bigquery.jobs.create` permission:

```bash
gcloud iam roles create customBigQueryJobCreator \
  --project=<PROJECT_ID> \
  --title="BigQuery Job Creator" \
  --description="Create BigQuery jobs for Data Quality Dashboard" \
  --permissions="bigquery.jobs.create"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
  --role="projects/<PROJECT_ID>/roles/customBigQueryJobCreator"
```

**Snowflake:**

If your role lacks `USAGE` [privilege](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#warehouse-privileges) on the active warehouse, queries cannot be performed. You may receive the following error:

- Error code: `11xxx`
- Description: `Missing required privileges on Snowflake: No active warehouse selected in the current session...`

Verify current warehouse privileges for your role:

```sql
SHOW GRANTS ON WAREHOUSE <WAREHOUSE_NAME>;
SHOW GRANTS TO ROLE <ROLE_NAME>;
```

If necessary, [grant](https://docs.snowflake.com/en/user-guide/security-access-control-overview) the `USAGE` privilege on the warehouse:

```sql
GRANT USAGE ON WAREHOUSE <WAREHOUSE_NAME> TO ROLE <ROLE_NAME>;
```

Ensure the grant is properly applied:

```sql
-- Verify the grant
SHOW GRANTS ON WAREHOUSE <WAREHOUSE_NAME>;
```

***

### Query timeouts

Long-running queries, a large volume of failed events, or resource pool exhaustion can cause the data quality dashboard to time out when fetching failed events. You may receive the following errors:

- Error codes: `12xxx` or `22xxx`
- Description: `Query exceeded timeout` or `Query execution time limit exceeded`

**BigQuery:**

Diagnose the cause of the error by running:

```sql
-- Check recent query performance
SELECT
  job_id,
  user_email,
  total_slot_ms,
  total_bytes_processed,
  TIMESTAMP_DIFF(end_time, start_time, SECOND) as duration_seconds
FROM `<PROJECT_ID>.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
  AND state = 'DONE'
  AND statement_type = 'SELECT'
ORDER BY total_slot_ms DESC
LIMIT 10;
```

**Snowflake:**

Diagnose the cause of the error by running:

```sql
-- Check query history
SELECT
  query_id,
  query_text,
  warehouse_name,
  execution_time,
  queued_overload_time,
  bytes_scanned
FROM table(information_schema.query_history())
WHERE start_time >= DATEADD('day', -1, CURRENT_TIMESTAMP())
  AND execution_status = 'SUCCESS'
ORDER BY execution_time DESC
LIMIT 10;
```

***

To fix these errors, try:

- Reducing query scope

  - For Console, try switching to using legacy failed events based on telemetry data
  - If using the API, try using smaller time windows, e.g., "Last hour" or "Last day" instead of "Last 30 days"
  - Query specific error types or schemas rather than all failed events

- Optimizing warehouse performance

  - Review your warehouse configuration and query patterns
  - Consider implementing partitioning, clustering, or other optimization strategies
  - Monitor resource usage, and adjust warehouse size as needed

## Default view

The default view is a relatively simple interface that shows the number of failed events over time alongside a coarse-grained description of the problem at hand. In certain cases, the error message can be somewhat cryptic in terms of diagnosing the root cause of the error. The reason is that this information flows through Snowplow infrastructure, therefore we have redacted it substantially before it leaves your pipeline, to ensure that PII does not traverse Snowplow systems. The aggregates are served by the Console's backend:

![](/assets/images/aggregator-architecture-cf7c92f332ecd3224d85b0f12ac07087.png)

In this setup, you expose no additional interface to the public internet, and all failed events information is served by the Console's APIs.

Below is an example view of the failed events screen in Snowplow Console:

![](/assets/images/image-1024x1024-55016853cb4f2d0d6dd5804da7bff6b9.png)

This interface is intended to give you a quick representation of the volume of event failures so that action can be taken. The UI focuses on schema violation and enrichment errors at present.

We've filtered on these two types of errors to reduce noise like bot traffic that causes adapter failures for instance. All failed events can be found in your S3 or GCS storage targets partitioned by type and then date/time.

At the top there is a data quality score that compares the volume of failed events to the volume that were successfully loaded into a data warehouse.

The bar chart shows how the total number of failed events varies over time, color-coding validation and enrichment errors. The time window is 7 days be default but can be extended to 14 or 30 days.

In the table, failed events are aggregated by the unique type of failure (validation, enrichment) and the offending schema.

By selecting a particular error you are able to get more detail:

![](/assets/images/image-1-1024x1009-88e9d195e3a0ea372e494280daba3bf1.png)

The detailed view shows the error message as well as other useful metadata (when available), like `app_id`, to help diagnose the source and root cause of the error; as well as a bar chart indicating how the volume of failures varies over time for this particular failed event.

## Console API

The API that powers the Console view and dashboard is publicly available, and can be invoked with a valid token to feed your own monitoring systems.

Before you can invoke the Failed Events API, you will need to [authenticate with an API key](/docs/account-management/).

A full specification of the API can be found in [our swagger docs](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/Metrics/getOrganizationsOrganizationidMetricsV1PipelinesPipelineidFailed-events). It is worth pointing out that, as is the case in the UI, the data returned only contains schema validation errors and enrichment failures.

---

# Recover failed events
> Fix and reprocess failed events using SQL recovery or manual recovery from buckets.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/

Event recovery allows you to fix [failed events](/docs/fundamentals/failed-events/) in some scenarios, so that you can ensure your data is complete and accurate.

A typical recovery process runs a script over a set of failed events to solve the issues (e.g. misspelled attribute name), and then attempts to re-process these events if necessary.

The best way to recover failed events is to [set up loading failed events into a separate table in your warehouse or lake](/docs/monitoring/exploring-failed-events/). [Use SQL to recover](/docs/monitoring/recovering-failed-events/recover-with-sql/) the events.

If you don't have a Failed Events Loader, you can [manually recover](/docs/monitoring/recovering-failed-events/manual/) failed events from their S3 or GCS buckets.

---

# Event recovery configuration examples
> Example recovery configurations including pass-through flows, multiple flow scenarios, and advanced multi-operation recovery patterns.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/configuration/configuration-examples/

The most basic configuration that will run for the particular failure type: `adapter-failure`, in a particular version, and mark all others as `failed` because of missing configuration mappings would look like this:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",
  "data": {
    "iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-0": [
      {
        "name": "pass-through-flow",
        "conditions": [],
        "steps": []
      }
    ]
  }
}
```

In above scenario we would be resubmitting all the received `adapter_failures` without any modifications.

## Multiple flows example

The next example shows a way of setting up multiple configuration flows for specific failure types. Configurations will be matched top to bottom and the first from the top will be chosen.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",
  "data": {
    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [
      {
        "name": "main-flow",
        "conditions": [
          {
            "op": "Test",
            "path": "$.payload.raw.vendor",
            "value": {
              "regex": "com.snowplow\\.*"
            }
          }
        ],
        "steps": [
          {
            "op": "Replace",
            "path": "$.raw.refererUri",
            "match": "(?U)^.*$",
            "value": "https://console.snplow.com/"
          }
        ]
      },
      {
        "name": "pass-through",
        "conditions": [],
        "steps": []
      }
    ]
  }
}
```

In above scenario we would only process the `1-0-0` version of `enrichment_failures`. We would be modifying those for which vendor in raw payload starts with `com.snowplow`. For those rows that match that field value, we replace the full `refererUri` contents with `https://console.snplow.com/`. All other `enrichment_failures` will be resubmitted as they were originally (pass-through scenario).

## Putting it all together

Below is an advanced example making use of most of the features and performing multiple operations on a very specific subset of failed events.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",
  "data": {
    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [
      {
        "name": "main-flow",
        "conditions": [
          {
            "op": "Test",
            "path": "$.processor.artifact",
            "value": {
              "value": "beam-enrich"
            }
          },
          {
            "op": "Test",
            "path": "$.payload.raw.vendor",
            "value": {
              "regex": "com.snowplow\\.*"
            }
          },
          {
            "op": "Test",
            "path": "$.processor.artifact",
            "value": {
              "size": {
                "eq": 11
              }
            }
          },
          {
            "op": "Test",
            "path": "$.processor.artifact",
            "value": {
              "size": {
                "gt": 3
              }
            }
          },
          {
            "op": "Test",
            "path": "$.processor.artifact",
            "value": {
              "size": {
                "lt": 30
              }
            }
          }
        ],
        "steps": [
          {
            "op": "Replace",
            "path": "$.raw.refererUri",
            "match": "(?U)^.*$",
            "value": "https://console.snplow.com/"
          },
          {
            "op": "Remove",
            "path": "$.raw.parameters.aid",
            "match": "(?U)^.*$"
          },
          {
            "op": "Replace",
            "path": "$.raw.headers",
            "match": "X-Forwarded-Proto:.*",
            "value": "X-Forwarded-Proto: http"
          },
          {
            "op": "Replace",
            "path": "$.raw.parameters.cx.data.[?(@.schema=~iglu:org.w3/PerformanceTiming/jsonschema/1-0-0)].data.loadEventEnd",
            "match": "(?U)^.*$",
            "value": "1"
          },
          {
            "op": "Cast",
            "path": "$.raw.parameters.cx.data.[?(@.schema=~iglu:org.w3/PerformanceTiming/jsonschema/1-0-0)].data.domComplete",
            "from": "Numeric",
            "to": "Boolean"
          },
          {
            "op": "Replace",
            "path": "$.raw.parameters.cx.data.[1].data.domComplete",
            "match": "false",
            "value": "true"
          },
          {
            "op": "Cast",
            "path": "$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete",
            "from": "Boolean",
            "to": "Numeric"
          }
        ]
      },
      {
        "name": "impossible-flow",
        "conditions": [
          {
            "op": "Test",
            "path": "$.processor.artifact",
            "value": {
              "value": "lorem-ipsum"
            }
          }
        ],
        "steps": []
      }
    ]
  }
}
```

---

# Configure an event recovery job
> Define recovery configuration using steps for field modifications, conditions for matching events, and flows to orchestrate recovery operations.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/configuration/

Configuration for event recovery involves the use of reusable components that help map onto specific failure types. The following components are available:

### Steps

Steps are individual modifications applied to specific failed event payloads. The steps operate on specific field values and can replace, remove/nullify or cast JSON type values.

Notice

Recovery operates on an individual event's `payload` field. As an example to reach the `vendor` field that’s located within the payload, the `path` would be `$.vendor`.

Steps are constructed in JSON object format consisting of four individual parts:

- `op` – transformation operation to perform: Replace, Remove, Cast
- `path` – JSON Path to the object where the operation is meant to happen. The path can contain specific field names (ie. `vendor`) , array ids (ie. `[1]`) or filters (by regex: `$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete`).
- `match` – an expression applied when replacing field’s values with new value
- `value` – a new value to be set
- `from` / `to` - used for casting types _from_ one type _to_ another

```json
// Replace
{
  "op": "Replace",
  "path": "$.raw.parameters.aid",
  "match": "(?U)^.*$",
  "value": "https://console.snplow.com/"
}

// Remove
{
  "op": "Remove",
  "path": "$.raw.parameters.aid",
  "match": "(?U)^.*$"
}

// Cast
{
  "op": "Cast",
  "path": "
$.raw.parameters.cx.data.[?(@.data.navigationStart=~([0-9]+))].data.domComplete",
  "from": "Numeric",
  "to": "Boolean"
}
```

### Conditions

Conditions are boolean expressions that operate on failed event fields in order to match on specific structure or values.

Note that conditions can be applied to arbitrary fields and therefore a condition's path scopes its entry point as the failed events “root”.

Much like steps, conditions are constructed with the following JSON object format consisting of four individual parts:

- `op` – transformation operation to perform: Replace, Remove, Cast & Test (tests that the specified value is set in the document. If the test fails, then the patch as a whole should not apply.)
- `path` – JSON Path to object, the path can contain specific field names (ie. `raw`) , array ids (ie. `[1]`) or filters (by regex: `[?(@.schema=~iglu://\\.*)]`).
- `value` – a value matcher to match against: match a regular expression, compare directly (object equality), check size for equality, less or greater than

```json
// Compare values
{
  "op": "Test",
  "path": "$.processor.artifact",
  "value": {
    "value": "beam-enrich"
  }
}

// Match against regex
{
  "op": "Test",
  "path": "$.payload.raw.vendor",
  "value": {
    "regex": "com.snowplow\\.*"
  }
}

// Compare sizes
{
  "op": "Test",
  "path": "$.processor.artifact",
  "value": {
    "size": {
      "eq": 11
    }
  }
}
{
  "op": "Test",
  "path": "$.processor.artifact",
  "value": {
    "size": {
      "gt": 3
    }
  }
}
{
  "op": "Test",
  "path": "$.processor.artifact",
  "value": {
    "size": {
      "lt": 30
    }
  }
}
```

### Flows

Flows are sequences of Steps applied one by one. Flows are mapped onto specific steps and conditions.

```json
{
  "name": "main flow",
  "conditions": [],
  "steps": []
}
```

## I/O

In principle data is sourced from bucket storage and delivered back into a collector. Recoverable and unrecoverable failed events are stored in bucket storage. For cloud-specific locations see below tables.

#### AWS

|                |                      |                       |                              |
| -------------- | -------------------- | --------------------- | ---------------------------- |
| input          | output               | failed output         | unrecoverable output         |
| S3 (`--input`) | Kinesis (`--output`) | S3 (`--failedOutput`) | S3 (`--unrecoverableOutput`) |

#### GCP

|                 |                     |                        |                               |
| --------------- | ------------------- | ---------------------- | ----------------------------- |
| input           | output              | failed output          | unrecoverable output          |
| GCS (`--input`) | PubSub (`--output`) | GCS (`--failedOutput`) | GCS (`--unrecoverableOutput`) |

---

# Extend and customize an event recovery job
> Add custom recovery steps, conditions, and bad row type support by extending the event recovery application.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/extending/

There are several extension points for this application:

- Steps
- Conditions
- Bad Row types

## Steps

By definition steps allow performing modifications on existing bad row data points' payloads. We defined a configuration DSL for Steps that is turned into actions with `Inspectable` definitions. `Inspectable`s are data structures on which Steps can be performed. Therefore in order to add a new `Step`:

- a `config.Step` DSL structure for it has to be defined
- an implementation of action defined as per DSL has to be defined for `Inspectable`.

The latter can be described in form of `transform` function that builds upon a recursive generic JSON-transforming structure.

## Conditions

Conditions are a lot like steps but as they are triggered earlier before `Inspectable`s are created. They do operate upon JSON data and need to implement behavior for basic JSON types. Therefore in order to add a new `Condition`:

- a `config.Condition` DSL structure for it has to be defined
- functions for performing `Condition` application for basic/composite JSON types have to be defined

## Bad Row types

Once new recoverable Bad Row types are added they need to be turned into `Recoverable` by supplying appropriate `Recoverable` instances. If a new `Payload` format is added it has to be turned into `Inspectable` as well.

It is important to note that we only represent recoverable bad rows as `Recoverable` instances. Not all `BadRow`s are recoverable and we strongly believe in not representing invalid states as possible.

---

# Introduction to manual failed event recovery
> Understand the recovery workflow, prerequisites, and configuration requirements for manually recovering failed events from storage.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/getting-started/

Event recovery at its core, is the ability to fix events that have failed and replay them through your pipeline.

After inspecting failed events either in [Snowplow Console](/docs/monitoring/), or in the [partitioned failure buckets](/docs/monitoring/exploring-failed-events/file-storage/), you can determine which events are possible to recover based on what the fix entails.

With recovery it is possible to:

- replace values - e.g. correct a typo in a schema name for validation
- remove values - e.g. remove improperly encoded values from a URL string
- cast JSON types - e.g. change a property's type from `string` to `integer`

If your failed events would not be fixed by applying the above, they currently would be considered unrecoverable. Due to the fact that there might be a mix of recoverable and unrecoverable data in your storage, event recovery uses configuration in order to process only a subset of the failed events.

### What you'll need to get started

The typical flow for recovery and some prerequisites to consider would be: **Understanding the failure issue**

- Familiarity with the [failed event types](/docs/fundamentals/failed-events/)
- Access to S3 or GCS buckets with failed events

**Configuring a recovery**

- Understanding the [configuration structure](/docs/monitoring/recovering-failed-events/manual/configuration/)
- Comfort with using Regular Expressions (RegEx)

**Testing the configuration**

- Ability to edit/run a Scala script locally

**Run the recovery**

- AWS sub account or GCP project admin access in order to create a recovery user

**Monitor the recovery**

- Access to DataFlow UI (GCP) or EMR reporting (AWS)

---

# Manual event recovery workflows
> Explore data structures, build configurations, validate recovery flows, and test jobs locally using scripting utilities.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/hints-workflows/

Most of the following workflows come from testing/working with elaborate bad row structures and configurations and rely heavily on the supplied scripting utility. To set it up locally follow [the testing documentation](/docs/monitoring/recovering-failed-events/manual/testing/) or starting with an empty ammonite script file (ie. `test.sc`), set up Snowplow's repository and dependencies:

```scala
interp.repositories() ++= Seq(coursierapi.MavenRepository.of("http://maven.snplow.com/releases/"))
@
import $url.{`https://raw.githubusercontent.com/snowplow-incubator/snowplow-event-recovery/feature/recovery-typeclasses/scripts/Recovery.sc` => Recovery}, Recovery._
import $ivy.`com.snowplowanalytics::snowplow-event-recovery-core:0.2.0-rc10`, com.snowplowanalytics.snowplow.event.recovery._, config._, json._
```

## Exploring data structures

The new bad row format is designed to be, readable and self-contained, however some fields are not captured in [JSON schemas for bad row format](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.badrows).

The unspecified fields can vary from plain JSON types through JSON objects to inline JSON structures and even Base64-encoded.

To explore the data structures we are able to use the convenience scripts as described in Testing section.

Given we got a bad row JSON structure we can turn it into navigable JSON object that we can step through and encode/decode using for example:

```scala
val badrow = """{"schema":"iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0","data":{"processor":{"artifact":"beam-enrich","version":"1.0.0-rc5"},"failure":{"timestamp":"2020-02-17T09:28:18.100Z","messages":[{"enrichment":{"schemaKey":"iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-0","identifier":"api-request"},"message":{"error":"Error accessing POJO input field [user]: [java.lang.NoSuchMethodException: com.snowplowanalytics.snowplow.enrich.common.outputs.EnrichedEvent.user_id-foo()]"}}]},"payload":{"enriched":{"app_id":"console","platform":"web","etl_tstamp":"2020-02-17 09:28:18.095","collector_tstamp":"2020-02-17 09:28:16.560","dvce_created_tstamp":"2020-02-17 09:28:16.114","event":"page_view","event_id":"2dfeb9b7-5a87-4214-8a97-a8b23176856b","txn_id":null,"name_tracker":"msc-gcp-stg1","v_tracker":"js-2.10.2","v_collector":"ssc-1.0.0-rc4-googlepubsub","v_etl":"beam-enrich-1.0.0-rc5-common-1.0.0","user_id":null,"user_ipaddress":"18.194.133.57","user_fingerprint":null,"domain_userid":"d6c468de-0aed-4785-9052-b6bb77b6dddb","domain_sessionidx":13,"network_userid":"510b2f05-27e3-4fd3-b449-a2702926da5e","geo_country":"DE","geo_region":"HE","geo_city":"Frankfurt am Main","geo_zipcode":"60313","geo_latitude":50.1188,"geo_longitude":8.6843,"geo_region_name":"Hesse","ip_isp":null,"ip_organization":null,"ip_domain":null,"ip_netspeed":null,"page_url":"https://console.snowplowanalytics.com/","page_title":"Snowplow CDI","page_referrer":null,"page_urlscheme":"https","page_urlhost":"console.snowplowanalytics.com","page_urlport":443,"page_urlpath":"/","page_urlquery":null,"page_urlfragment":null,"refr_urlscheme":null,"refr_urlhost":null,"refr_urlport":0,"refr_urlpath":null,"refr_urlquery":null,"refr_urlfragment":null,"refr_medium":null,"refr_source":null,"refr_term":null,"mkt_medium":null,"mkt_source":null,"mkt_term":null,"mkt_content":null,"mkt_campaign":null,"contexts":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"39a9934a-ddd3-4581-a4ea-d0ba20e63b92\"}},{\"schema\":\"iglu:org.w3/PerformanceTiming/jsonschema/1-0-0\",\"data\":{\"navigationStart\":1581931694397,\"unloadEventStart\":1581931696046,\"unloadEventEnd\":1581931694764,\"redirectStart\":0,\"redirectEnd\":0,\"fetchStart\":1581931694397,\"domainLookupStart\":1581931694440,\"domainLookupEnd\":1581931694513,\"connectStart\":1581931694513,\"connectEnd\":1581931694665,\"secureConnectionStart\":1581931694572,\"requestStart\":1581931694665,\"responseStart\":1581931694750,\"responseEnd\":1581931694750,\"domLoading\":1581931694762,\"domInteractive\":1581931695963,\"domContentLoadedEventStart\":1581931696039,\"domContentLoadedEventEnd\":1581931696039,\"domComplete\":0,\"loadEventStart\":0,\"loadEventEnd\":0}}]}","se_category":null,"se_action":null,"se_label":null,"se_property":null,"se_value":null,"unstruct_event":null,"tr_orderid":null,"tr_affiliation":null,"tr_total":null,"tr_tax":null,"tr_shipping":null,"tr_city":null,"tr_state":null,"tr_country":null,"ti_orderid":null,"ti_sku":null,"ti_name":null,"ti_category":null,"ti_price":null,"ti_quantity":0,"pp_xoffset_min":0,"pp_xoffset_max":0,"pp_yoffset_min":0,"pp_yoffset_max":0,"useragent":"Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","br_name":null,"br_family":null,"br_version":null,"br_type":null,"br_renderengine":null,"br_lang":"en-US","br_features_pdf":0,"br_features_flash":0,"br_features_java":0,"br_features_director":0,"br_features_quicktime":0,"br_features_realplayer":0,"br_features_windowsmedia":0,"br_features_gears":0,"br_features_silverlight":0,"br_cookies":1,"br_colordepth":"24","br_viewwidth":1918,"br_viewheight":982,"os_name":null,"os_family":null,"os_manufacturer":null,"os_timezone":"Europe/Berlin","dvce_type":null,"dvce_ismobile":0,"dvce_screenwidth":1920,"dvce_screenheight":1080,"doc_charset":"UTF-8","doc_width":1918,"doc_height":982,"tr_currency":null,"tr_total_base":null,"tr_tax_base":null,"tr_shipping_base":null,"ti_currency":null,"ti_price_base":null,"base_currency":null,"geo_timezone":"Europe/Berlin","mkt_clickid":null,"mkt_network":null,"etl_tags":null,"dvce_sent_tstamp":"2020-02-17 09:28:16.507","refr_domain_userid":null,"refr_dvce_tstamp":null,"derived_contexts":"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0\",\"data\":{\"useragentFamily\":\"Firefox\",\"useragentMajor\":\"72\",\"useragentMinor\":\"0\",\"useragentPatch\":null,\"useragentVersion\":\"Firefox 72.0\",\"osFamily\":\"Linux\",\"osMajor\":null,\"osMinor\":null,\"osPatch\":null,\"osPatchMinor\":null,\"osVersion\":\"Linux\",\"deviceFamily\":\"Other\"}}]}","domain_sessionid":"96958bf6-a8bf-4be8-9c67-fd957b6bc8d2","derived_tstamp":"2020-02-17 09:28:16.167","event_vendor":"com.snowplowanalytics.snowplow","event_name":"page_view","event_format":"jsonschema","event_version":"1-0-0","event_fingerprint":"5acdc8f85f9530081d1a71ec430c8756","true_tstamp":null},"raw":{"vendor":"com.snowplowanalytics.snowplow","version":"tp2","parameters":[{"name":"e","value":"pv"},{"name":"duid","value":"d6c468de-0aed-4785-9052-b6bb77b6dddb"},{"name":"vid","value":"13"},{"name":"eid","value":"2dfeb9b7-5a87-4214-8a97-a8b23176856b"},{"name":"url","value":"https://console.snowplowanalytics.com/"},{"name":"aid","value":"console"},{"name":"cx","value":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiIzOWE5OTM0YS1kZGQzLTQ1ODEtYTRlYS1kMGJhMjBlNjNiOTIifX0seyJzY2hlbWEiOiJpZ2x1Om9yZy53My9QZXJmb3JtYW5jZVRpbWluZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJuYXZpZ2F0aW9uU3RhcnQiOjE1ODE5MzE2OTQzOTcsInVubG9hZEV2ZW50U3RhcnQiOjE1ODE5MzE2OTYwNDYsInVubG9hZEV2ZW50RW5kIjoxNTgxOTMxNjk0NzY0LCJyZWRpcmVjdFN0YXJ0IjowLCJyZWRpcmVjdEVuZCI6MCwiZmV0Y2hTdGFydCI6MTU4MTkzMTY5NDM5NywiZG9tYWluTG9va3VwU3RhcnQiOjE1ODE5MzE2OTQ0NDAsImRvbWFpbkxvb2t1cEVuZCI6MTU4MTkzMTY5NDUxMywiY29ubmVjdFN0YXJ0IjoxNTgxOTMxNjk0NTEzLCJjb25uZWN0RW5kIjoxNTgxOTMxNjk0NjY1LCJzZWN1cmVDb25uZWN0aW9uU3RhcnQiOjE1ODE5MzE2OTQ1NzIsInJlcXVlc3RTdGFydCI6MTU4MTkzMTY5NDY2NSwicmVzcG9uc2VTdGFydCI6MTU4MTkzMTY5NDc1MCwicmVzcG9uc2VFbmQiOjE1ODE5MzE2OTQ3NTAsImRvbUxvYWRpbmciOjE1ODE5MzE2OTQ3NjIsImRvbUludGVyYWN0aXZlIjoxNTgxOTMxNjk1OTYzLCJkb21Db250ZW50TG9hZGVkRXZlbnRTdGFydCI6MTU4MTkzMTY5NjAzOSwiZG9tQ29udGVudExvYWRlZEV2ZW50RW5kIjoxNTgxOTMxNjk2MDM5LCJkb21Db21wbGV0ZSI6MCwibG9hZEV2ZW50U3RhcnQiOjAsImxvYWRFdmVudEVuZCI6MH19XX0"},{"name":"tna","value":"msc-gcp-stg1"},{"name":"cs","value":"UTF-8"},{"name":"cd","value":"24"},{"name":"page","value":"Snowplow CDI"},{"name":"stm","value":"1581931696507"},{"name":"tz","value":"Europe/Berlin"},{"name":"tv","value":"js-2.10.2"},{"name":"vp","value":"1918x982"},{"name":"ds","value":"1918x982"},{"name":"res","value":"1920x1080"},{"name":"cookie","value":"1"},{"name":"p","value":"web"},{"name":"dtm","value":"1581931696114"},{"name":"lang","value":"en-US"},{"name":"sid","value":"96958bf6-a8bf-4be8-9c67-fd957b6bc8d2"}],"contentType":"application/json","loaderName":"ssc-1.0.0-rc4-googlepubsub","encoding":"UTF-8","hostname":"gcp-sandbox-prod1.collector.snplow.net","timestamp":"2020-02-17T09:28:16.560Z","ipAddress":"18.194.133.57","useragent":"Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","refererUri":"https://console.snowplowanalytics.com/","headers":["Timeout-Access: ","Host: gcp-sandbox-prod1.collector.snplow.net","User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","Accept: */*","Accept-Language: en-US, en;q=0.5","Accept-Encoding: gzip, deflate, br","Origin: https://console.snowplowanalytics.com","Referer: https://console.snowplowanalytics.com/","Cookie: sp=510b2f05-27e3-4fd3-b449-a2702926da5e","X-Cloud-Trace-Context: 958285ba723e212998af29cec405e002/12535615945289151925","Via: 1.1 google","X-Forwarded-For: 18.194.133.57, 35.201.76.62","X-Forwarded-Proto: https","Connection: Keep-Alive","application/json"],"userId":"510b2f05-27e3-4fd3-b449-a2702926da5e"}}}}"""
parse(badrow).foreach(println)
```

## Building configurations

After being able to explore the data structure and figuring out what amendments to introduce, this needs to be turned into a set of individual building blocks. Configuration building blocks can be tested individually using the scripting utility. To test particular steps you can use following pattern:

```scala
operations.cast("""{"int": "1"}""", "$.int", CastType.String, CastType.Array)
operations.replace("""{"int": "1"}""", "$.int", "(?U)^.*$", "new")
```

## Validating configurations

The rules about setting up configurations are fairly strict to prevent unnecessary delay in empty job evaluations. To validate a built configuration it is easy to simply run it through a validator using the scripting utility:

```scala
val cfg = """{ "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/3-0-0", "data": { "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-*": [{"name": "passthrough", "conditions": [], "steps": []}]}}"""
configs.validate(cfg)
```

## Testing jobs locally

After validating the configuration it makes sense to try out the full configuration locally using a sample, scrambled data:

```scala
jobs.test(cfg, badrow)
jobs.testMany(cfg, List(badrow))
```

---

# Event recovery without Failed Events Loader
> Manually recover failed events from S3 or GCS buckets using low-level recovery tools for Snowplow Self-Hosted pipelines.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/

Snowplow pipelines are "non-lossy", this means if something is wrong with an event during any part of the pipeline, the event is stored in a separate storage environment rather than just discarded. See the [failed events section](/docs/fundamentals/failed-events/) for more information on the types of failures that may occur.

Besides allowing for the inspection of failed events to fix the root cause of the problem, you have the option to recover data by running a recovery process to correct an issue and "re-play" the events through your pipeline again.

Snowplow Event Recovery is available for pipelines running on AWS and GCP.

---

# Monitor event recovery jobs
> Track recovery job progress using EMR metrics, Google Dataflow UI, and business processing metrics for failed, recovered, and unrecoverable events.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/monitoring/

When running a recovery job in your production pipeline, you'll likely find it useful to keep an eye on it's progress.

In order to verify the process is running properly there are several locations in your **infrastructure** that can be monitored, depending on your runtime environment. These are: datasinks (for recovery job processed output): `failedOutput` (S3/GCS bucket), `unrecoverableOutput` (S3/GCS bucket), `output` (Kinesis/PubSub streams) and job runners (tracking job status and processing in real-time).

Beyond monitoring infrastructure, each job exposes “business” processing metrics that summarise failed, unrecoverable and recovered bad rows that have been processed.

### Amazon EMR

On EMR recovery job exposes run summary using a [built-in reporting library delivering](https://spark.apache.org/docs/latest/monitoring.html) count summaries for:

- `event-recovery.driver.summary.Recovered`
- `event-recovery.driver.summary.Failed`
- `event-recovery.driver.summary.Unrecoverable`

The metrics can be accessed trough variety of ways (sinks) and can be configured upon cluster creation parameters.

To enable desired sink set EMR’s `spark-metrics` classification parameters following [possible values](https://github.com/apache/spark/blob/master/conf/metrics.properties.template) using EMR’s classifiers.

To expose metrics over http accessible at `http://${SPARK_HOST}:4040/metrics/json`:

```json
  {
    "classification": "spark-metrics",
    "properties": {
        "spark.metrics.namespace": "event-revovery",
        "*.sink.servlet.class": "org.apache.spark.metrics.sink.MetricsServlet",
        "*.sink.servlet.period": "1",
        "*.sink.servlet.unit": "seconds",
        "*.sink.servlet.path": "/metrics/json",
        "master.sink.servlet.path": "/metrics/master/json",
        "applications.sink.servlet.path": "metrics/applications/json",
        "*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"

    }
```

To push metrics to output logs to console appender:

```json
  {
    "classification": "spark-metrics",
    "properties": {
        "spark.metrics.namespace": "$name",
        "*.sink.console.class": "org.apache.spark.metrics.sink.ConsoleSink",
        "*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"

    }
```

For more sinks see [Spark documentation on monitoring](https://spark.apache.org/docs/latest/monitoring.html).

### Google Dataflow

On Google Dataflow recovery job exposes realtime run metrics using a dataflow’s native functions. The metrics are:

- `recovered`
- `failed`
- `unrecoverable`

The metrics can be accessed through the web UI directly in Dataflow.

---

# Run event recovery on Flink
> Deploy event recovery jobs on Apache Flink using EMR with fault tolerance and CloudWatch metrics monitoring.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/flink/

The Flink job reads bad rows from an S3 location and stores the recovered payloads in Kinesis, unrecovered and unrecoverable in other S3 buckets.

#### Building

To build the fat jar, run:

```bash
sbt flink/assembly
```

#### Running

Event recovery jobs are usually ran using our [dataflow-runner](https://github.com/snowplow/dataflow-runner) application. A configuration template for running is [available in the repository](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner).

To run a transient EMR cluster and execute the job using the templates, download [Flink dataflow-runner templates](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner) from the repository and run:

```bash
dataflow-runner run-transient \
  --emr-playbook flink-playbook.json.tmpl \
  --emr-config flink-cluster.json.tmpl \
  --vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY,interval,$INTERVAL
```

Where:

- `BUCKET_INPUT` - S3 bucket containing bad events to be recovered (eg. geoffs-bad-events)
- `BUCKET_INPUT_DIRECTORY` - directory in the `BUCKET_INPUT` to use as the source for bad events
- `AWS_REGION` - region in which the job is being ran (eg. eu-central-1)
- `AWS_SUBNET` - network subnet to run the job in (eg. subnet-435010347a21886ab)
- `AWS_IAM_ROLE` - AWS IAM role to assume while running the job (eg. geoffs-recovery-role)
- `AWS_KEYPAIR` - AWS EC2 key pair to use for the instances (eg. geoffs-keypair)
- `JOB_OWNER` - tag to use to mark the owner of the job (eg. goeff)
- `RECOVERY_VERSION` - application version (eg. 0.6.0)
- `RECOVERY_CONFIG` - recovery job config as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `IGLU_RESOLVER` - iglu resolver configuration as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `KINESIS_OUTPUT` - Kinesis stream to output recovered events to
- `INTERVAL` - job checkpointing interval (we recommend starting with 10 minutes and tuning to your job's characteristics)

The job uses [checkpointing](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/checkpointing/) to allow for job restarts. Kinesis connector which is a part of the checkpointing mechanism [guarantees](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#kinesis-sinks-and-fault-tolerance) at-least-once delivery semantics and provides backpressure to the job.

#### Monitoring

EMR Flink Job emits a wide range of metrics through Cloudwatch Agent which is installed during cluster bootstrap using a provided script (see project repo's dataflow-runner directory). Custom metrics include:

- number of unrecoverable events in the run: `taskmanager_container_XXXX_XXX__Process_0_events_unrecoverable`
- number of failed recovery attempts: `taskmanager_container_XXXX_XXX__Process_0_events_failed`
- number of recovered events: `taskmanager_container_XXXX_XXX__Process_0_events_recovered`
- total number of events submitted for processing: `taskmanager_container_XXXX_XXX__Process_0_numRecordsIn`

The metrics are delivered to `snowplow/event-recovery` Cloudwatch namespace.

Given a one-second aggregation in Cloudwatch the diagram should show an always increasing metrics that should balance themselves up to total sum.

> **Note:** Flink's `numRecordsOut` metric for sinks does not reflect an actual number of records saved in the sink.

---

# Run event recovery on GCP Dataflow with Beam
> Deploy dockerized event recovery jobs on Google Cloud Dataflow using Apache Beam runtime.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/gcp-beam/

The Beam job is a dockerized purpose built Dataflow job that reads data from a GCS location specified through a pattern and stores the recovered payloads in a PubSub topic, unrecovered and unrecoverable in other GCS buckets.

## Permissions

In order to run the job a `service-account` with appropriate credentials must be set up in the project.

In order to be sure not to disrupt any service accounts that may be required for normal pipeline operations, it's recommended to create a new service account for recovery.

To create a new service account in your GCP console navigate to [IAM & Admin > Service](https://console.cloud.google.com/iam-admin/serviceaccounts) [Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts). Create a new account and grant `full project owner` permissions.

Generate a new key for the service account and save down to your local machine.

## Building

To build the docker image, run:

```bash
sbt beam/docker:publishLocal
```

## Running

To run on Apache Beam in GCP Dataflow run it through a docker-deployment the options to set are shown below.

```bash
-v {{path-to-local-key-file}}.json:/snowplow/config/credentials.json \
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/config/credentials.json \
  snowplow/snowplow-event-recovery-beam:0.6.1 \
--runner=DataFlowRunner \
--project={{your-gcp-project-name}} \
--jobName=event-recovery-rt-pipeline \
--region={{your-project-region}} \
--gcpTempLocation=gs://sp-storage-loader-tmp-{{pipeline_tag_e.g._prod1}}-{{pipeline_name}}/temp \
--inputDirectory=gs://sp-storage-loader-bad-{{pipeline_tag_e.g._prod1}}-{{project_name}}/partitioned/** \
--outputTopic=projects/{{project_name}}/topics/{{recovery-topic}} \
--failedOutput=gs://{{create_or_choose_a_failed_output_folder}}/failed/ \
--unrecoverableOutput=gs://{{create_or_choose_an_unrecoverable_folder}}/unrecoverable/ \
--config={{base64-encoded-string of your recovery configuration}} \
--resolver={{base64-encoded-string of a schema resolver file (Iglu Central ok for default)}}
```

As this is a self-contained docker image the first flag sets the key for the deployment in the folder for use by the job.

Note that for zone GCP prefers you may need to use -a or -b zones (e.g. europe-west2-b).

Also note, that no `"` are needed for any of the strings.

---

# Run event recovery jobs
> Define and encode recovery configuration, then deploy recovery jobs using Spark, Flink, or Beam runtimes.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/

Now that we've discussed configuring the recovery, let's dive in to running it on your pipeline.

First we'll need to define configuration:

## Define configuration

The configuration consists of the `resolver-config.json` that your pipeline uses to resolve events against corresponding schema, for example:

```json
{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",
  "data": {
    "cacheSize": 0,
    "repositories": [
      {
        "name": "Iglu Central",
        "priority": 0,
        "vendorPrefixes": [
          "com.snowplowanalytics"
        ],
        "connection": {
          "http": {
            "uri": "https://iglucentral.com"
          }
        }
      }
    ]
  }
}
```

Also, the`job-config.json` which describes the recovery job to be done, for example:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/recoveries/jsonschema/4-0-0",
  "data": {
    "iglu:com.snowplowanalytics.snowplow.badrows/enrichment_failures/jsonschema/1-0-0": [
      {
        "name": "passthrough",
        "conditions": [],
        "steps": []
      }
    ]
  }
}
```

## Encode configuration

Configuration is supplied to recovery jobs as a Base64-encoded string. You can use your encoding plugin of choice, but here is an example [ammonite](http://ammonite.io/) script you can use to encode your configuration:

```scala
import java.util.Base64
import java.nio.charset.StandardCharsets
import ammonite.ops._
import $ivy.`io.circe::circe-parser:0.13.0`, io.circe._, io.circe.syntax._, io.circe.parser._

val load = (path: String)  => read! os.Path(path, base = os.pwd)
val encode = (str: String)  => Base64.getEncoder.encodeToString(str.getBytes(StandardCharsets.UTF_8))

@main def run(config: String): Unit =
  parse(load(config)) match {
    case Right(data)  => println(encode(data.asJson.noSpaces))
    case Left(err)  => println(s"Invalid JSON input in: ${err.getMessage}")
  }
```

And then just run (assuming ammonite is on your path and above script is called `encode.sc`:

```bash
amm ./encode.sc resolver-config.json
amm ./encode.sc job-config.json
```

Once the configuration files are encrypted it's time to deploy the recovery job on your pipeline.

## Deploy the job

There are several runtimes that recovery process can be deployed to:

---

# Run event recovery on Spark (legacy)
> Deploy event recovery jobs on Apache Spark using EMR and dataflow-runner.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/running/spark/

> **Danger:** Spark module is now deprecated. Kinesis integration is not as reliable as we'd like. We suggest migrating to [Flink module](/docs/monitoring/recovering-failed-events/manual/running/flink/).

The Spark job reads bad rows from an S3 location and stores the recovered payloads in Kinesis, unrecovered and unrecoverable in other S3 buckets.

#### Building

To build the fat jar, run:

```bash
sbt spark/assembly
```

#### Running

Event recovery jobs are usually ran using our [dataflow-runner](https://github.com/snowplow/dataflow-runner) application. A configuration templates for running is [available in the repository](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner).

To run a transient EMR cluster and execute the job using the templates, download [Spark dataflow-runner templates](https://github.com/snowplow-incubator/snowplow-event-recovery/blob/master/.dataflow-runner) from the repository and run:

```bash
dataflow-runner run-transient \
  --emr-playbook spark-playbook.json.tmpl \
  --emr-config spark-cluster.json.tmpl \
  --vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY
```

Where:

- `BUCKET_INPUT` - S3 bucket containing bad events to be recovered (eg. geoffs-bad-events)
- `BUCKET_INPUT_DIRECTORY` - directory in the `BUCKET_INPUT` to use as the source for bad events
- `AWS_REGION` - region in which the job is being ran (eg. eu-central-1)
- `AWS_SUBNET` - network subnet to run the job in (eg. subnet-435010347a21886ab)
- `AWS_IAM_ROLE` - AWS IAM role to assume while running the job (eg. geoffs-recovery-role)
- `AWS_KEYPAIR` - AWS EC2 key pair to use for the instances (eg. geoffs-keypair)
- `JOB_OWNER` - tag to use to mark the owner of the job (eg. goeff)
- `RECOVERY_VERSION` - application version (eg. 0.6.0)
- `RECOVERY_CONFIG` - recovery job config as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `IGLU_RESOLVER` - iglu resolver configuration as described in [Configuration](/docs/monitoring/recovering-failed-events/manual/configuration/)
- `KINESIS_OUTPUT` - Kinesis stream to output recovered events to

---

# Test an event recovery configuration locally
> Validate recovery configuration and run small-scale test recoveries locally using the snowplow-event-recovery CLI tool.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/testing/

Once you know how you want to configure the recovery job, it's worthwhile to test your configuration locally, before deploying an actual recovery job.

For that purpose we provide a `snowplow-event-recovery` command line interface. Latest versions are available through [GitHub release page](https://github.com/snowplow-incubator/snowplow-event-recovery/releases/latest).

The CLI provides commands for validating (`snowplow-event-recovery validate`) and running test recoveries (`snowplow-event-recovery run`). See `snowplow-event-recovery --help` for the most up-to-date options descriptions.

### Validate

Validation checks configuration schema to verify that actual job can be ran. To validate the configuration, run:

```text
snowplow-event-recovery validate --config /opt/scenarios.json
```

Which will result in:

```text
OK! Config valid
```

Or an detailed error message:

```text
ERROR! Invalid config

Instance is not valid against its schema:
* At $.iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-*[0].conditions[0].op:
  - $.iglu:com.snowplowanalytics.snowplow.badrows/adapter_failures/jsonschema/1-0-*[0].conditions[0].op: does not have a value in the enumeration [Test]
```

### Test

To run a small-scale recovery test, outputting both good and failed events into files, execute:

```text
snowplow-event-recovery run --config $PWD/scenarios.json -o /tmp -i $PWD/input/ --resolver $PWD/resolver.json
```

The input folder (`--input` or `-i`) contains a line-delimited files (by default `.txt` files are parsed, but the glob extension can be configured using `--glob` or `-g` flag) minimized json failed events as seen in [example](https://github.com/snowplow-incubator/snowplow-event-recovery/tree/master/examples).

With output being saved into `/tmp` directory as a stringified payload objects into `good.txt` and json-serialized failed events into `bad.txt`.

Which for successful recovery will print the progress and output summary:

```text
Config loaded
Enriching event...
Enriched event 2dfeb9b7-5a87-4214-8a97-a8b23176856b
...
Total Lines: 156, Recovered: 142
```

By default no enrichments are configured, but local-only (ones that do not depend on upstream resource storage) enrichments can be enabled by setting `--enrichments` flag:

```text
snowplow-event-recovery run --config $PWD/scenarios.json -o /tmp -i $PWD/input/ --resolver $PWD/resolver.json --enrichments $PWD/enrichments

Total Lines: 3, Recovered: 0
OK! Successfully ran recovery.
```

---

# Troubleshoot the manual event recovery process
> Diagnose and resolve process failures, recovery failures, and unexpected behavior in event recovery jobs.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/troubleshooting/

In order to verify the process is running properly there are several locations that can be monitored, depending on your runtime environment these are: datasinks (for recovery job processed output): `failedOutput` (S3/GCS bucket), `unrecoverableOutput` (S3/GCS bucket), `output` (Kinesis/PubSub streams) and job runners (tracking job status and processing in real-time).

## Possible Failures

There are two main reasons why recovery process may fail. Process Failures preventing recovery process to proceed and Recovery Failures that mean recovery flow cannot be applied to given bad row/line. Another set of issues is unexpected behavior of the application. The three failure types need to be handled differently.

### Process Failures

Process failures are usually caused by misconfiguration. They usually happen as the job is submitted to its runner (Apache Beam, Spark, Flink). Reported failures may differ, but usually they boil down to:

#### Missing Resources

This means that resources set as input or output for recovery do not exist or are inaccessible to application.

**Action:** Check the existence of resources provided to job configuration and their access policies.

#### Configuration Issues

The largest group of issues causing early process termination are configuration issues. They are reported in CLI as the job is submitted.

**Action:** Make sure the configuration is well formed and it is conforming its schema.

### Recovery Failures

Definitely the largest and the most common issue with automated recovery process are recovery failures. These are runtime errors that prevent application from recovering given lines / bad rows. These failures will produce values under `failedOutput` directory.

Recovery error is yet another bad row type that is able to be recovered. For convenience recovery errors use original configurations and do not require another recovery flows configured especially for them.

Recovery error contain additional information pointing to:

- recovery flow name
- number of recoveries attempted
- original bad row

#### Missing recovery configuration

Happens when trying to recover a line for which no recovery flow configuration can be matched.

**Action:** Check whether supplied configuration contains a proper mapping for problematic bad row instance including its schema key and conditions that are applied before choosing configuration.

#### Replacement Failures

Actual replacement specified with Replace Step has failed. Reason will be supplied along with exported `RecoveryError` bad row under `unrecoveredOutput`.

**Action:** consult the exported message. This likely means that configuration supplied to modify the source bad row needs adjusting. Either with another recovery flow (when faulty row is a special case) or direct adjustment to recovery flow supplied with `RecoveryError`.

#### Cast Failures

Much like Replacement Failures, Cast Failures occur when modifying actual value with a Cast Step has failed. The reason is supplied along with exported `RecoveryError` bad row under `unrecoveredOutput`.

**Action:** consult the exported message. This likely means that configuration supplied to modify the source bad row needs adjusting. Either with another recovery flow (when faulty row is a special case) or direct adjustment to recovery flow supplied with `RecoveryError`.

#### Unrecoverable Bad Row Type

Some bad row types are marked as unrecoverable and therefore even with an existing configuration they will not be transformed. They are outputted to `unrecoverable`.

**Action:** If you think that specific bad row type should be recoverable consult Extending Recovery and consider submitting a Pull Request.

#### Invalid JSON Format

Happens whenever input data is not a proper JSON format.

**Action:** Check that input JSON file contains a proper JSON object in each line.

#### Invalid Data Format

Happens whenever input data is not a valid `BadRow` JSON format.

**Action:** Check that input JSON file contains a proper `BadRow` JSON object in each line.

#### Codec Failures

This set of failures should not really occur but means that there was an error serializing data with thrift (ThriftFailure) or Base64 coding. Some of these issues might be related to runtime locale.

**Action:** Consult supplied error message outputted along with BadRow into `unrecoveredOutput`. Verify that bad row producer source has the same locale as recovery worker – UTF-8.

### Unexpected Behavior

This is hopefully the rarest but the least specific set of issues that may occur. In case any unexpected behavior happens consider submitting a GitHub issue.

---

# Recover loaded events from a failed events table with SQL
> Learn how to recover failed events from your failed events table back into your good events table using SQL queries.
> Source: https://docs.snowplow.io/docs/monitoring/recovering-failed-events/recover-with-sql/

This guide will walk you through the process of recovering [failed events](/docs/fundamentals/failed-events/) from your failed events table back into your good events table. Failed events occur when [events](/docs/fundamentals/events/) don't pass validation during processing, typically due to [schema](/docs/fundamentals/schemas/) violations or enrichment failures.

## Recovery process

Recovering failed events involves identifying the failure type, reconstructing the data, and reinserting it into your good events table. The process varies depending on the specific failure that occurred.

The main steps are:

1. **Identify the type of failure**: there are many possible reasons an event can fail and end up in your failed events table. The failed events table only includes schema violations and enrichment failures, but there are still many different types:

   - Missing schema - the schema referenced (or specific version of that schema) isn't present in your prod Iglu repository.
   - Missing required property.
   - Extra properties that are not allowed in this schema version.
   - Referencing an incorrect schema version.
   - Incorrect data type.
   - Breaching a validation rule (a `min`, `max`, `minLength`, `maxLength`, `pattern`, `format`, `enum`, etc.)

2. **Reconstruct the offending row**: how you do this will depend on the type of failure that has occurred, and your specific business logic. For example, if you have a missing schema, because you started tracking before pushing the schema to Prod, then you should push the schema to Prod, and then essentially just select the values from the failure entity. However, if you have a string that is too long for the `maxLength` validation rule for example, then you must choose how to resolve this (trim to the max allowed value, or push a new schema with an increased limit, and change the schema version in the offending row from the original to the new version).

3. **Create a prep `SELECT` statement**: create a statement that addresses the issue. You can test this by attempting to `UNION` your prep table to a `SELECT` from the good `events` table to check for type inconsistencies.

4. **Create an `INSERT` command**: add the repaired row back to the main `events` table, leaving all other values unaltered. You can either type this out by hand, or if you don't want to do this (your events table may have hundreds of columns which is time consuming and error prone to type out manually) you can use the `INFORMATION_SCHEMA.COLUMNS` metadata table in your warehouse to construct the `INSERT` command programmatically.

## Identifying the nature of the failed event

In your failed events table, there is a column called `contexts_com_snowplowanalytics_snowplow_failure_1` which contains information about why the event failed. All the other columns are as-is when the event was tracked and enriched.

The following query will show you all the failures in your failed events table, and make the fields easier to read:

**BigQuery:**

```sql
SELECT
    app_id,
    event_name,
    event_version,
    contexts_com_snowplowanalytics_snowplow_failure_1,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
    contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,
    SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,
        '/')[0], ':')[1] AS schema_vendor,
    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,
    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version
  FROM `snowplow_failed.events` -- Use your failed events schema
  WHERE 1=1
    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day) -- include a date range for when the failed events occurred
    AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
```

**Snowflake:**

```sql
SELECT
    app_id,
    event_name,
    event_version,
    contexts_com_snowplowanalytics_snowplow_failure_1,
    contexts_com_snowplowanalytics_snowplow_failure_1[0]:data AS data,  -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
    contexts_com_snowplowanalytics_snowplow_failure_1[0]:errors AS errors,
    contexts_com_snowplowanalytics_snowplow_failure_1[0]:failure_type AS failure_type,
    contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema AS schema,
    contexts_com_snowplowanalytics_snowplow_failure_1[0]:timestamp AS timestamp,

    SPLIT_PART(SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 1), ':', 2) AS schema_vendor,
    SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 2) AS schema_name,
    SPLIT_PART(contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema::STRING, '/', 4) AS schema_version

FROM SNOWPLOW_FAILED.EVENTS -- Use your failed events schema
WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAY' -- include a date range for when the failed events occurred
  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL;
```

**Databricks:**

```sql
SELECT
  app_id,
  event_name,
  event_version,
  contexts_com_snowplowanalytics_snowplow_failure_1,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
  contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,
  SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,
      '/')[0], ':')[1] AS schema_vendor,
  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,
  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version
FROM snowplow_failed.events -- Use your failed events schema
WHERE 1=1
  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7) -- include a date range for when the failed events occurred
  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
```

***

You can add additional filters to this query to isolate the specific failures you're interested in recovering:

**BigQuery:**

```sql
WITH
  failed_events AS (
  SELECT
    app_id,
    event_name,
    event_version,
    contexts_com_snowplowanalytics_snowplow_failure_1,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,
    contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,
    SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,
        '/')[0], ':')[1] AS schema_vendor,
    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,
    SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version
  FROM `snowplow_failed.events` -- Use your failed events schema
  WHERE 1=1
    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)
    AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
  ORDER BY timestamp DESC)
  SELECT
    event_name,
    schema,
    schema_name,
    schema_version,
    schema_vendor,
    data,
    errors,
    data.user_name,
    data.user_id
  FROM failed_events
  WHERE 1=1
    AND schema_name = 'user_entity'
    AND app_id = 'test-app'
  ORDER BY timestamp DESC
```

**Snowflake:**

```sql
WITH failed_events AS (
  SELECT
    app_id,
    event_name,
    event_version,
    f.value AS failure_context,
    f.value:data AS data,
    f.value:errors AS errors,
    f.value:failure_type AS failure_type,
    f.value:schema AS schema,
    f.value:timestamp AS timestamp,
    SPLIT_PART(SPLIT_PART(f.value:schema::string, '/', 1), ':', 2) AS schema_vendor,
    SPLIT_PART(f.value:schema::string, '/', 2) AS schema_name,
    SPLIT_PART(f.value:schema::string, '/', 4) AS schema_version
  FROM SNOWPLOW_FAILED.EVENTS,  -- Use your failed events schema
       LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_failure_1) f -- using flatten allows for instances where multiple entities have failed on the same event
  WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAYS'
)
SELECT
  event_name,
  schema,
  schema_name,
  schema_version,
  schema_vendor,
  data,
  errors,
  data:user_name AS user_name,
  data:user_id AS user_id,
FROM failed_events
WHERE schema_name = 'user_entity'
  AND app_id = 'test-app'
ORDER BY timestamp DESC;
```

**Databricks:**

```sql
WITH
failed_events AS (
SELECT
  app_id,
  event_name,
  event_version,
  contexts_com_snowplowanalytics_snowplow_failure_1,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].data AS data,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].errors AS errors,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].failure_type AS failure_type,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].schema AS schema,
  contexts_com_snowplowanalytics_snowplow_failure_1[0].timestamp AS timestamp,
  SPLIT(SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema,
      '/')[0], ':')[1] AS schema_vendor,
  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[1] AS schema_name,
  SPLIT(contexts_com_snowplowanalytics_snowplow_failure_1[0].schema, '/')[3] AS schema_version
FROM snowplow_failed.events -- Use your failed events schema
WHERE 1=1
  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)
  AND contexts_com_snowplowanalytics_snowplow_failure_1 IS NOT NULL
ORDER BY timestamp DESC)
SELECT
  event_name,
  schema,
  schema_name,
  schema_version,
  schema_vendor,
  data,
  errors,
  get_json_object(data, '$.user_name') as user_name,
  get_json_object(data, '$.user_id') as user_id
FROM failed_events
WHERE 1=1
  AND schema_name = 'user_entity'
  AND app_id = 'test-app'
ORDER BY timestamp DESC
```

***

Here we are filtering for our `test-app` app ID, and the specific schema failures we're looking for (`WHERE schema_name = 'user_entity'`).

The `data` field is a JSON object of the data that was included in the event, untouched from when it was tracked. We can access these using dot notation on the `data` object - e.g. `data.user_name`.

![Failed events query results showing ResolutionError failures](/assets/images/failed-events-results-table-7e2bc8645de5d51b4d45e42af3bfdd93.png)

In this example, we can see that in the `schema_version` field, the version is `2-0-1` when the most up to date version of this schema is `2-0-0` and we've therefore had a `ResolutionError` in the `failure_type` column - meaning that Iglu wasn't able to find the schema that was included in the event. Therefore, we need to reconstruct the entity columns, with the correct schema version and the correct corresponding column name for that version. This is a relatively straightforward fix, as we don't have to mutate the data in place at all.

## Reconstructing the failed event row

Even though this is a fairly straightforward fix (essentially in this case it is a passthrough) we must use some knowledge of how Snowplow constructs data in our good `events` table in order to correctly reconstruct the event before inserting.

Key principles for reconstructing events:

- The properties types defined in the JSON schema for an event or entity are deterministically converted into column types in the data warehouse.

- Custom events are analogous to JSON objects - key-value pairs of data on the event. The values in this object can be primitive types (strings, numbers, boolean values etc) or complex types (objects and arrays). The keys become the column names.

- Custom entities are stored as arrays of objects - this is because it is possible (and common) to attach multiple of the same entity to a single event - e.g. an array of `product` entities being attached to a `product_list` event. So even if there is only one of a given entity attached to an event, they are always stored as arrays in the data warehouse. Each entity object is semantically identical to that of a custom event i.e. a key-value object.

- The `contexts_com_snowplowanalytics_snowplow_failure_1` column is itself an array as it is an entity, and a single event can have multiple failures, each that should be addressed before reinserting.

- The target column names also follow a deterministic naming convention:

  - Custom event columns are of the format `unstruct_event_{vendor_name}_{schema_name}_{version_number | major_version_numer}`.
  - Custom entity columns are of the format `contexts_{vendor_name}_{schema_name}_{version_number | major_version_numer}`.
  - Column names also are "snake-ified", so camelCase names in your schemas will be converted to snake\_case, and dots (`.`) in your vendor name will also be snake-ified.

## Creating a prep query

In this example, the `user_entity` is obviously an entity, so will be in an array. Depending on the data warehouse you are using, the `data` object will be stored differently.

- BigQuery stores the `data` object as a JSON type.
- Snowflake uses a VARIANT type.
- Databricks uses a `STRING` type to represent the JSON object.

Regardless of the data warehouse in question, the repaired object should be an array containing a single struct value, and the individual values must be cast to their correct types from the value stored in the failed event object.

**BigQuery:**

```sql
WITH prep AS (
  SELECT
    contexts_com_snowplowanalytics_snowplow_failure_1
  FROM `snowplow_failed.events`
  WHERE 1=1
    AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)
    AND app_id = 'test-app'
    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1' -- filter for my offending schema
)
SELECT
  ARRAY( -- since this is an entity, it must be an array
    SELECT
      STRUCT(
        '2-0-0' AS _schema_version, -- set the schema version to the correct entity version for BigQuery Loader V2
        INT64(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_id) AS user_id, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
        STRING(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_name) AS user_name -- use the `INT64(...)` and the `STRING(...)` functions to cast the raw JSON values to their appropriate types
      )
    )
  AS contexts_com_example_user_entity_2_0_0 -- correctly name the column name to be inserted
FROM prep;
```

**Snowflake:**

```sql
WITH prep AS (
    SELECT contexts_com_snowplowanalytics_snowplow_failure_1
    FROM SNOWPLOW_FAILED.EVENTS
    WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL '7 DAY'
    AND app_id = 'test-app'
    AND contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema = 'iglu:com.example/user_entity/jsonschema/2-0-1'
  )
SELECT
  ARRAY_CONSTRUCT( -- since this is an entity, it must be an array
   OBJECT_CONSTRUCT(
   '_schema_version', '2-0-0', -- set the schema version to the correct entity version for Snowflake Streaming Loader
   'user_id', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_id::INT, -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
   'user_name', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_name::STRING  -- use the `...::INT` and the `...::STRING` functions to cast the raw JSON values to their appropriate types
   )
   ) AS CONTEXTS_COM_EXAMPLE_USER_ENTITY_2
FROM prep
```

**Databricks:**

```sql
WITH prep AS (
SELECT
  contexts_com_snowplowanalytics_snowplow_failure_1
FROM snowplow_failed.events
WHERE 1=1
  AND DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)
  AND app_id = 'test-app'
  AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1' -- filter for my offending schema
)
SELECT
ARRAY( -- since this is an entity, it must be an array
    NAMED_STRUCT(
      '_schema_version', '2-0-0', -- set the schema version to the correct entity version for Databricks
      'user_id', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, '$.user_id') as BIGINT), -- use the `[0]` syntax to extract the first (and only) instance of this entity from the failure array
      'user_name', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, '$.user_name') as STRING) -- CAST as BIGINT and STRING to cast the raw JSON string values to their appropriate types
    )
  )
AS contexts_com_example_user_entity_2_0_0 -- correctly name the column name to be inserted
FROM prep;
```

***

## Reinserting the repaired rows

Now we have successfully identified our problematic rows and corrected them, we can now insert them back to the events table. While you can hand write an `INSERT` statement for the rows, the Snowplow events tables tend to have very large and complex table schemas, with hundreds or thousands of columns, with many complex nested columns. This process is time consuming and error prone, so we recommend using the `INFORMATION_SCHEMA` tables to programmatically construct the `INSERT` command before running it.

There are some complexities to bear in mind when doing this:

- There may well be columns that are not in the failed events table that are in the good events table. Your insert command must include `CAST(NULL AS ... )` in place for all of those columns.
- You must ensure that you have the correct structure for your repaired columns as otherwise they will not insert and the warehouse will reject the inserted rows.
- In newer versions of Snowplow data warehouse loaders (such as Snowflake Streaming Loader and the BigQuery Loader V2), entity columns also have a `_schema_version` column. This must also be set for reprocessed entity columns in order for the insert to be successful.
- If the recovered events are net-new to your good events table (say you've encountered failures on the first time using this schema) then that column won't exist in your good events table yet. We'd recommend sending a hand crafted good event for that schema to your prod pipeline **first**. This will allow the Snowplow loader to correctly create the column in your good events table so it is ready to be inserted into.

> **Info:** We would recommend using a staging table to insert the repaired rows before running this process on your main events table. This will allow you to review the repaired rows before inserting them into your main events table. To do so you can take a cut of your main events table, and run the `INSERT` command against that table, and verify you get the behavior you expect.

Below are scripts for BigQuery and Snowflake to create you the `INSERT` command:

**BigQuery:**

```sql
-- Schema-aware INSERT generator for Snowplow failed events reprocessing

WITH target_table_columns AS ( -- this CTE finds all the columns in the good events table
  SELECT
    ordinal_position,
    column_name,
    data_type,
    is_nullable
  FROM `snowplow_good.INFORMATION_SCHEMA.COLUMNS`
  WHERE table_name = 'events'
    AND table_schema = 'snowplow_good'
),

failed_table_columns AS ( -- this CTE finds all the columns in the failed events table
  SELECT
    column_name,
    data_type,
    is_nullable
  FROM `snowplow_failed.INFORMATION_SCHEMA.COLUMNS`
  WHERE table_name = 'events'
    AND table_schema = 'snowplow_failed'
),

schema_comparison AS ( -- this CTE joins the previous two, and labels them if they are missing
  SELECT
    t.ordinal_position,
    t.column_name,
    t.data_type AS target_type,
    f.data_type AS failed_type,
    t.is_nullable,
    CASE
      WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'
      ELSE 'MATCH'
    END AS status
  FROM target_table_columns t
  LEFT JOIN failed_table_columns f ON t.column_name = f.column_name
),

column_mappings AS (   -- this CTE checks if columns are missing,
  SELECT               -- handles those missing columns by inserting NULLs
    ordinal_position,  -- and also inserts your repaired event columns
    column_name,
    target_type,
    status,
    CASE
      -- Handle your specific replaced entity column
      WHEN column_name = 'contexts_com_example_user_entity_2_0_0' THEN -- isolate the target column
      -- paste in your reprocess logic as a string (including newline characters (`\n`) if you wish)
        '''array(select struct(
        \'2-0-0\' AS _schema_version,
        int64(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_id) AS user_id,
        string(contexts_com_snowplowanalytics_snowplow_failure_1[0].data.user_name) AS user_name
        )) AS contexts_com_example_user_entity_2_0_0'''

      -- Exclude the failure entity from source
      WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN
        'cast(null as array<struct<schema string, data json>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'

      -- Handle the load_tstamp column so that the inserted rows have the correct timestamp
      WHEN column_name = 'load_tstamp' THEN 'current_timestamp() AS load_tstamp'

      -- Handle missing columns with appropriate NULL casting
      WHEN status = 'MISSING_IN_FAILED' THEN
        CASE
          -- Entity columns
          WHEN STARTS_WITH(lower(column_name), 'contexts_') THEN
            FORMAT('cast(null as %s) AS %s', target_type, column_name)
          -- Custom event columns
          WHEN STARTS_WITH(lower(column_name), 'unstruct_') THEN
            FORMAT('cast(null as %s) AS %s', target_type, column_name)
          ELSE
            FORMAT('cast(null as %s) AS %s', target_type, column_name)
        END

      -- Handle type mismatches with explicit casting
      WHEN status = 'TYPE_MISMATCH' THEN
        FORMAT('cast(%s as %s) AS %s', column_name, target_type, column_name)

      -- Pass through matching columns
      ELSE column_name
    END AS select_expression
  FROM schema_comparison
),

select_clause AS ( -- this CTE lists all the columns to go in the `INSERT` statement
  SELECT STRING_AGG(select_expression, ',\n  ' ORDER BY ordinal_position) AS columns_sql
  FROM column_mappings
),

generated_insert AS (
  SELECT FORMAT("""INSERT INTO `snowplow_good.events`
WITH prep AS (
  SELECT
    * -- include all columns from the failed events table
  FROM `snowplow_failed.events`
  WHERE DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), INTERVAL 7 day)
    AND app_id = 'test-app'
    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = 'iglu:com.example/user_entity/jsonschema/2-0-1'
)
SELECT
  %s
FROM prep;""",
    columns_sql
  ) AS insert_statement
  FROM select_clause
)
-- Output the complete INSERT statement
SELECT
  insert_statement AS generated_sql
FROM generated_insert
```

**Snowflake:**

```sql
-- Schema-aware INSERT generator for Snowplow failed events reprocessing

WITH target_table_columns AS ( -- this CTE finds all the columns in the good events table
  SELECT
    ordinal_position,
    column_name,
    data_type,
    is_nullable
  FROM SNOWPLOW_GOOD_DATABASE.INFORMATION_SCHEMA.COLUMNS -- Use the database which has your SNOWPLOW_GOOD schema
  WHERE lower(table_name) = 'events'
    AND lower(table_schema) = 'snowplow_good'
),

failed_table_columns AS ( -- this CTE finds all the columns in the failed events table
  SELECT
    column_name,
    data_type,
    is_nullable
  FROM SNOWPLOW_FAILED_DATABASE.INFORMATION_SCHEMA.COLUMNS   --Use the database which has your SNOWPLOW_FAILED schema
  WHERE lower(table_name) = 'events'
    AND lower(table_schema) = 'snowplow_failed'
),

schema_comparison AS ( -- this CTE joins the previous two, and labels them if they are missing
  SELECT
    t.ordinal_position,
    t.column_name,
    t.data_type AS target_type,
    f.data_type AS failed_type,
    t.is_nullable,
    CASE
      WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'
      ELSE 'MATCH'
    END AS status
  FROM target_table_columns t
  LEFT JOIN failed_table_columns f ON t.column_name = f.column_name
),

column_mappings AS (   -- this CTE checks if columns are missing,
  SELECT               -- handles those missing columns by inserting NULLs
    ordinal_position,  -- and also inserts your repaired event columns
    column_name,
    target_type,
    status,
    CASE
      -- Handle your specific replaced entity column
      WHEN column_name = 'CONTEXTS_COM_EXAMPLE_USER_ENTITY_2' THEN  -- isolate the target column
      -- paste in your reprocess logic as a string (newline characters (`\n`) also work)
        'ARRAY_CONSTRUCT(
        OBJECT_CONSTRUCT(
        ''_schema_version'', ''2-0-0'', -- set the schema version to the correct entity version for Snowflake Streaming Loader
        ''user_id'', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_id::INT,
        ''user_name'', contexts_com_snowplowanalytics_snowplow_failure_1[0]:data:user_name::STRING
        )
        ) AS CONTEXTS_COM_EXAMPLE_USER_ENTITY_2'

      -- Exclude the failure entity from source
      WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN
        'CAST(NULL AS ARRAY<OBJECT<schema STRING, data VARIANT>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'

      -- Handle the load_tstamp column so that the inserted rows have the correct timestamp
      WHEN column_name = 'load_tstamp' THEN 'CURRENT_TIMESTAMP() AS load_tstamp'

      -- Handle missing columns with appropriate NULL casting
      WHEN status = 'MISSING_IN_FAILED' THEN
        CASE
          -- Entity columns
          WHEN STARTSWITH(lower(column_name), 'contexts_') THEN
            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
          -- Custom event columns
          WHEN STARTSWITH(lower(column_name), 'unstruct_') THEN
            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
          -- Standard columns
          ELSE
            CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
        END

      -- Handle type mismatches with explicit casting
      WHEN status = 'TYPE_MISMATCH' THEN
        CONCAT('CAST(', column_name, ' AS ', target_type, ') AS ', column_name)

      -- Pass through matching columns
      ELSE column_name
    END AS select_expression
  FROM schema_comparison
),

select_clause AS (  -- this CTE lists all the columns to go in the `INSERT` statement
  SELECT LISTAGG(select_expression, ',\n  ') WITHIN GROUP (ORDER BY ordinal_position) AS columns_sql
  FROM column_mappings
),

generated_insert AS (
  SELECT CONCAT(
    'INSERT INTO SNOWPLOW_GOOD.EVENTS
    WITH prep AS (
    SELECT * -- include all columns from the failed events table
    FROM SNOWPLOW_FAILED.EVENTS
    WHERE load_tstamp::DATE > CURRENT_DATE - INTERVAL ''7 DAY''
    AND app_id = ''test-app''
    AND contexts_com_snowplowanalytics_snowplow_failure_1[0]:schema = ''iglu:com.example/user_entity/jsonschema/2-0-1''
)
SELECT
  ',
    columns_sql,
    '
    FROM prep;'
  ) AS insert_statement
  FROM select_clause
)

-- Output the complete INSERT statement
SELECT
  insert_statement AS generated_sql
FROM generated_insert;
```

**Databricks:**

```sql
WITH target_table_columns AS ( -- Find all columns in the good events table
SELECT
  ordinal_position,
  column_name,
  full_data_type,
  is_nullable
FROM snowplow_good.information_schema.columns
WHERE table_name = 'events'
  AND table_schema = 'snowplow_good'
),

failed_table_columns AS ( -- Find all columns in the failed events table
SELECT
  column_name,
  full_data_type,
  is_nullable
FROM snowplow_failed.information_schema.columns
WHERE table_name = 'events'
  AND table_schema = 'snowplow_failed'
),

schema_comparison AS ( -- Join and label missing columns
SELECT
  t.ordinal_position,
  t.column_name,
  t.full_data_type AS target_type,
  f.full_data_type AS failed_type,
  t.is_nullable,
  CASE
    WHEN f.column_name IS NULL THEN 'MISSING_IN_FAILED'
    WHEN t.full_data_type != f.full_data_type THEN 'TYPE_MISMATCH'
    ELSE 'MATCH'
  END AS status
FROM target_table_columns t
LEFT JOIN failed_table_columns f ON t.column_name = f.column_name
),

column_mappings AS ( -- Handle missing columns and insert repaired event columns
SELECT
  ordinal_position,
  column_name,
  target_type,
  status,
  CASE
    -- Handle your specific replaced entity column
    WHEN column_name = 'contexts_com_example_user_entity_2_0_0' THEN
      -- Paste in your reprocess logic as a string
      'ARRAY(NAMED_STRUCT(
      ''_schema_version'', ''2-0-0'',
      ''user_id'', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, ''$.user_id'') AS BIGINT),
      ''user_name'', CAST(GET_JSON_OBJECT(contexts_com_snowplowanalytics_snowplow_failure_1[0].data, ''$.user_name'') AS STRING)
      )) AS contexts_com_example_user_entity_2_0_0'

    -- Exclude the failure entity from source
    WHEN column_name = 'contexts_com_snowplowanalytics_snowplow_failure_1' THEN
      'CAST(NULL AS ARRAY<STRUCT<schema:STRING, data:STRING>>) AS contexts_com_snowplowanalytics_snowplow_failure_1'

    -- Handle the load_tstamp column
    WHEN column_name = 'load_tstamp' THEN 
      'CURRENT_TIMESTAMP() AS load_tstamp'

    -- Handle missing columns with appropriate NULL casting
    WHEN status = 'MISSING_IN_FAILED' THEN
      CASE
        -- Entity columns (contexts)
        WHEN LOWER(column_name) LIKE 'contexts_%' THEN
          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
        -- Custom event columns (unstruct)
        WHEN LOWER(column_name) LIKE 'unstruct_%' THEN
          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
        -- Standard columns
        ELSE
          CONCAT('CAST(NULL AS ', target_type, ') AS ', column_name)
      END

    -- Handle type mismatches with explicit casting
    WHEN status = 'TYPE_MISMATCH' THEN
      CONCAT('CAST(', column_name, ' AS ', target_type, ') AS ', column_name)

    -- Pass through matching columns
    ELSE column_name
  END AS select_expression
FROM schema_comparison
),

select_clause AS ( -- List all columns for the INSERT statement
SELECT 
  CONCAT_WS(',\n  ', COLLECT_LIST(select_expression)) AS columns_sql
FROM (
  SELECT ordinal_position, select_expression
  FROM column_mappings
  ORDER BY ordinal_position
)
),

generated_insert AS (
SELECT 
  CONCAT(
    'INSERT INTO snowplow_good.events\n',
    'WITH prep AS (\n',
    '  SELECT\n',
    '    * -- include all columns from the failed events table\n',
    '  FROM snowplow_failed.events\n',
    '  WHERE DATE(load_tstamp) > DATE_SUB(CURRENT_DATE(), 7)\n',
    '    AND app_id = ''test-app''\n',
    '    AND contexts_com_snowplowanalytics_snowplow_failure_1[0].schema = ''iglu:com.example/user_entity/jsonschema/2-0-1''\n',
    ')\n',
    'SELECT\n  ',
    columns_sql,
    '\nFROM prep;'
  ) AS insert_statement
FROM select_clause
)

-- Output the complete INSERT statement
SELECT
insert_statement AS generated_sql
FROM generated_insert;
```

***

This will print out the correctly formatted `INSERT` statement. You can now copy this and run it against your data warehouse, and it will reinsert the newly repaired failed events back into your good events table.

---

# Configure the Collector
> View and configure Snowplow collector settings via Console UI or API, including cookie domains, paths, and HTTP headers.
> Source: https://docs.snowplow.io/docs/pipeline/collector/

Once your [event collector](/docs/fundamentals/) is set up, along with [trackers](/docs/sources/) to submit events to them, you may want to verify your collector configuration. This can take two forms, depending on your needs.

Read more about the technical aspects of the collector [here](/docs/api-reference/stream-collector/).

## Viewing collector configuration in Snowplow Console

The easiest way to access collector configuration is to view it within the Snowplow Console. To do that, after you log in click on _Pipeline Configuration_ under the respective pipeline's navigation menu:

![](/assets/images/image-1-8ca1a1d304dc7770448eb2698869a7ee.png)

Selecting the pipeline configuration tab

You can then view your configuration, with the default values displayed for empty fields:

![](/assets/images/image-2-d20f02e67f447d2499aeba129631d9ab.png)

Example collector configuration

This view is consuming the respective API that you can also access, as discussed in the next section.

## Consuming the collector configuration API

As a Snowplow customer you already benefit from 24x7 monitoring of pipeline collector health. If you wish to add collector monitoring to your internal monitoring systems nevertheless, the maintainable way to do this is to retrieve collector endpoints and other configuration values via the available API, then invoke your health checks on them.

### Authorization

To start using the API you’ll need authorization credentials.

First, you’ll need to:

- [Create a new user account for your API authorization in console](https://console.snowplowanalytics.com/users)
- Request the `clientid` and `clientsecret` for the API service from the [admin section in console](https://console.snowplowanalytics.com/credentials).

Once you have these you can exchange credentials for a token.

Here is an example CURL to use to fetch the token:

```bash
curl --request POST \
  --url 'https://id.snowplowanalytics.com/oauth/token' \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data grant_type=password \
  --data username=USER@DOMAIN.COM \
  --data password='PASSWORD' \
  --data audience=https://snowplowanalytics.com/api/ \
  --data client_id='YOUR_CLIENT_ID' \
  --data client_secret='YOUR_CLIENT_SECRET'
```

This token will be needed in any request to the API in the form of `Authorization: Bearer {{token}}`

### Getting started

You can have a look at and interact with the collector configuration endpoint in the normative [API specification](https://console.snowplowanalytics.com/api/msc/v1/docs/index.html?url=/api/msc/v1/docs/docs.yaml#/configuration/getOrganizationsOrganizationidResourcesV1PipelinesPipelineidConfigurationCollector).

### Authorizing in the API documentation

To be able to post sample requests in the documentation you need to click the `Authorize` button at the top of the document and authorize with your token. The value for the token field in each individual requests is overwritten by this authorization.

The endpoint for the API is:

`https://console.snowplowanalytics.com/api/msc/v1/organizations/{organizationId}/resources/v1/pipelines/{pipelineId}/configuration/collector`

and the only supported HTTP verb is `GET`.

Your organization ID is a UUID and can be found in the location bar of your browser when you are using the Console:

![This image has an empty alt attribute; its file name is image-3.png](/assets/images/image-3-b70ed2227e15788771771ffbbca92100.png)

Organization ID is the UUID in-between the two forward slashes.

Similarly, the pipeline ID can be found in the location bar as the second UUID there:

![This image has an empty alt attribute; its file name is image-4.png](/assets/images/image-4-716927604c876d4c0652077b2cc6071c.png)

The Pipeline ID is the second UUID in the location bar.

### Invoking the API

Invoking this API will return an object of the following form:

```json
{
    "cookieDomains":
    {
        "fallback": "mydomain.com",
        "domains": []
    },
    "paths": {
        "post": {
            "paths": []
        },
        "webhook": {
            "paths": []
        },
        "redirect": {
            "paths": [],
            "enabled": true
        }
    },
    "cookieAttributes": {
        "secure": true,
        "sameSite": "None",
        "httpOnly": false
    },
    "blockUnencrypted": false
}
```

The `cookieDomains` object is always expected to be available and holds two properties:

- `domains` is a non-optional list of domains for which collector cookies are being set. The value may be an empty list.
- `fallback` represents the default fallback cookie domain. The property is optional and if unavailable it defaults to `none`.

The `paths` object is always expected to be available and holds the following three non-optional properties:

- `post` contains a non-optional list of paths that accept Tracker Protocol 2 compatible POST requests

- `redirect` contains

  - a non-optional flag weather redirect-based tracking is enabled, and
  - a non-optional list of paths that support this kind of tracking

- `webhook` contains a non-optional list of paths for tracking events sent via a `GET` or `POST` request containing an [Iglu](https://github.com/snowplow/iglu)-compatible event payload.

The `cookieAttributes` object is always expected to be available and contains three non-optional properties:

- `secure` is a boolean value indicating whether secure connections should be enforced;

- `httpOnly` is a boolean value set to true if the cookie must be inaccessible to non-HTTP requests; and

- `sameSite` is a string that can take one of the following values:

  - `lax`
  - `none`
  - `strict`

Finally, `blockUnencrypted` is an optional boolean property indicating whether un-encrypted traffic should be allowed or not. If not available, the default is `false` (i.e. "do not block").

## Configuring the collector for Snowplow Self-Hosted users

After you have installed the [Collector](/docs/api-reference/stream-collector/) (either following the [Quick Start guide](/docs/get-started/self-hosted/) or [manually](/docs/api-reference/stream-collector/setup/)), you can follow the [reference page](/docs/api-reference/stream-collector/configure/) to configure it.

## HTTP request headers

The Collector will collect any standard HTTP headers and the values of these headers can be extracted during Enrichment. The [HTTP header extractor enrichment](/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/) can be configured for the headers you wish to extract.

Additionally, the following two headers can be sent on requests:

| Header       | Allowed Values     | Description                                                                                                |
| ------------ | ------------------ | ---------------------------------------------------------------------------------------------------------- |
| Content-Type | `application/json` | See [MDN Content-Type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type)             |
| SP-Anonymous | `*`                | Enables Server Side Anonymization, preventing the User IP Address and Network User ID from being collected |

### Cookie header

The Collector will collect any cookie information sent in the `Cookie` HTTP header. Cookies can be attached to events using the [Cookie extractor enrichment](/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/)

---

# Enriched stream TSV file format
> Understand the Tab Separated Values format of enriched Snowplow data streams with field definitions and analytics SDK references.
> Source: https://docs.snowplow.io/docs/pipeline/enriched-tsv-format/

The Snowplow pipeline outputs the enriched stream in a Tab Separated Values (TSV) format. As TSV files do not contain header information, this page exists to help users of the enriched stream understand what each value represents.

Additionally, Snowplow has a number of Analytics SDKs available which help parse the TSV records into JSON:

- [Analytics SDK Scala](https://github.com/snowplow/snowplow-scala-analytics-sdk)
- [Analytics SDK Python](https://github.com/snowplow/snowplow-python-analytics-sdk)
- [Analytics SDK .NET](https://github.com/snowplow/snowplow-dotnet-analytics-sdk)
- [Analytics SDK Javascript](https://github.com/snowplow-incubator/snowplow-js-analytics-sdk/)
- [Analytics SDK Golang](https://github.com/snowplow/snowplow-golang-analytics-sdk)

For explanations of what each field represents, please see the [Canonical Event Model](/docs/fundamentals/canonical-event/).

## TSV header

Copy this if you need a header to work with the Snowplow TSV files.

```text
app_id	platform	etl_tstamp	collector_tstamp	dvce_created_tstamp	event	event_id	txn_id	name_tracker	v_tracker	v_collector	v_etl	user_id	user_ipaddress	user_fingerprint	domain_userid	domain_sessionidx	network_userid	geo_country	geo_region	geo_city	geo_zipcode	geo_latitude	geo_longitude	geo_region_name	ip_isp	ip_organization	ip_domain	ip_netspeed	page_url	page_title	page_referrer	page_urlscheme	page_urlhost	page_urlport	page_urlpath	page_urlquery	page_urlfragment	refr_urlscheme	refr_urlhost	refr_urlport	refr_urlpath	refr_urlquery	refr_urlfragment	refr_medium	refr_source	refr_term	mkt_medium	mkt_source	mkt_term	mkt_content	mkt_campaign	contexts	se_category	se_action	se_label	se_property	se_value	unstruct_event	tr_orderid	tr_affiliation	tr_total	tr_tax	tr_shipping	tr_city	tr_state	tr_country	ti_orderid	ti_sku	ti_name	ti_category	ti_price	ti_quantity	pp_xoffset_min	pp_xoffset_max	pp_yoffset_min	pp_yoffset_max	useragent	br_name	br_family	br_version	br_type	br_renderengine	br_lang	br_features_pdf	br_features_flash	br_features_java	br_features_director	br_features_quicktime	br_features_realplayer	br_features_windowsmedia	br_features_gears	br_features_silverlight	br_cookies	br_colordepth	br_viewwidth	br_viewheight	os_name	os_family	os_manufacturer	os_timezone	dvce_type	dvce_ismobile	dvce_screenwidth	dvce_screenheight	doc_charset	doc_width	doc_height	tr_currency	tr_total_base	tr_tax_base	tr_shipping_base	ti_currency	ti_price_base	base_currency	geo_timezone	mkt_clickid	mkt_network	etl_tags	dvce_sent_tstamp	refr_domain_userid	refr_device_tstamp	derived_contexts	domain_sessionid	derived_tstamp	event_vendor	event_name	event_format	event_version	event_fingerprint	true_tstamp
```

## TSV properties

The following table lists the properties of the TSV file.

| Property Index | Property Name              |
| -------------- | -------------------------- |
| 0              | app\_id                    |
| 1              | platform                   |
| 2              | etl\_tstamp                |
| 3              | collector\_tstamp          |
| 4              | dvce\_created\_tstamp      |
| 5              | event                      |
| 6              | event\_id                  |
| 7              | txn\_id                    |
| 8              | name\_tracker              |
| 9              | v\_tracker                 |
| 10             | v\_collector               |
| 11             | v\_etl                     |
| 12             | user\_id                   |
| 13             | user\_ipaddress            |
| 14             | user\_fingerprint          |
| 15             | domain\_userid             |
| 16             | domain\_sessionidx         |
| 17             | network\_userid            |
| 18             | geo\_country               |
| 19             | geo\_region                |
| 20             | geo\_city                  |
| 21             | geo\_zipcode               |
| 22             | geo\_latitude              |
| 23             | geo\_longitude             |
| 24             | geo\_region\_name          |
| 25             | ip\_isp                    |
| 26             | ip\_organization           |
| 27             | ip\_domain                 |
| 28             | ip\_netspeed               |
| 29             | page\_url                  |
| 30             | page\_title                |
| 31             | page\_referrer             |
| 32             | page\_urlscheme            |
| 33             | page\_urlhost              |
| 34             | page\_urlport              |
| 35             | page\_urlpath              |
| 36             | page\_urlquery             |
| 37             | page\_urlfragment          |
| 38             | refr\_urlscheme            |
| 39             | refr\_urlhost              |
| 40             | refr\_urlport              |
| 41             | refr\_urlpath              |
| 42             | refr\_urlquery             |
| 43             | refr\_urlfragment          |
| 44             | refr\_medium               |
| 45             | refr\_source               |
| 46             | refr\_term                 |
| 47             | mkt\_medium                |
| 48             | mkt\_source                |
| 49             | mkt\_term                  |
| 50             | mkt\_content               |
| 51             | mkt\_campaign              |
| 52             | contexts                   |
| 53             | se\_category               |
| 54             | se\_action                 |
| 55             | se\_label                  |
| 56             | se\_property               |
| 57             | se\_value                  |
| 58             | unstruct\_event            |
| 59             | tr\_orderid                |
| 60             | tr\_affiliation            |
| 61             | tr\_total                  |
| 62             | tr\_tax                    |
| 63             | tr\_shipping               |
| 64             | tr\_city                   |
| 65             | tr\_state                  |
| 66             | tr\_country                |
| 67             | ti\_orderid                |
| 68             | ti\_sku                    |
| 69             | ti\_name                   |
| 70             | ti\_category               |
| 71             | ti\_price                  |
| 72             | ti\_quantity               |
| 73             | pp\_xoffset\_min           |
| 74             | pp\_xoffset\_max           |
| 75             | pp\_yoffset\_min           |
| 76             | pp\_yoffset\_max           |
| 77             | useragent                  |
| 78             | br\_name                   |
| 79             | br\_family                 |
| 80             | br\_version                |
| 81             | br\_type                   |
| 82             | br\_renderengine           |
| 83             | br\_lang                   |
| 84             | br\_features\_pdf          |
| 85             | br\_features\_flash        |
| 86             | br\_features\_java         |
| 87             | br\_features\_director     |
| 88             | br\_features\_quicktime    |
| 89             | br\_features\_realplayer   |
| 90             | br\_features\_windowsmedia |
| 91             | br\_features\_gears        |
| 92             | br\_features\_silverlight  |
| 93             | br\_cookies                |
| 94             | br\_colordepth             |
| 95             | br\_viewwidth              |
| 96             | br\_viewheight             |
| 97             | os\_name                   |
| 98             | os\_family                 |
| 99             | os\_manufacturer           |
| 100            | os\_timezone               |
| 101            | dvce\_type                 |
| 102            | dvce\_ismobile             |
| 103            | dvce\_screenwidth          |
| 104            | dvce\_screenheight         |
| 105            | doc\_charset               |
| 106            | doc\_width                 |
| 107            | doc\_height                |
| 108            | tr\_currency               |
| 109            | tr\_total\_base            |
| 110            | tr\_tax\_base              |
| 111            | tr\_shipping\_base         |
| 112            | ti\_currency               |
| 113            | ti\_price\_base            |
| 114            | base\_currency             |
| 115            | geo\_timezone              |
| 116            | mkt\_clickid               |
| 117            | mkt\_network               |
| 118            | etl\_tags                  |
| 119            | dvce\_sent\_tstamp         |
| 120            | refr\_domain\_userid       |
| 121            | refr\_device\_tstamp       |
| 122            | derived\_contexts          |
| 123            | domain\_sessionid          |
| 124            | derived\_tstamp            |
| 125            | event\_vendor              |
| 126            | event\_name                |
| 127            | event\_format              |
| 128            | event\_version             |
| 129            | event\_fingerprint         |
| 130            | true\_tstamp               |

---

# ASN lookup enrichment
> Flag bot traffic by checking autonomous system numbers against known bad ASN lists.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/

> **Note:** This enrichment is available since version 6.9.0 of Enrich.

This enrichment checks the autonomous system number (ASN) attached to an event against a configurable list of ASNs associated with bots, cloud providers, or abusive networks. When a match is found, the enrichment sets `likelyBot` to `true` on the ASN [entity](/docs/fundamentals/entities/) added by the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/).

This is useful for automatically flagging non-human traffic. Many bots and scrapers originate from well-known cloud hosting or data center ASNs, and community-maintained lists such as [cpuchain/bad-asn-list](https://github.com/cpuchain/bad-asn-list) track these.

> **Note:** This enrichment requires the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) to be enabled with ASN data (either the free GeoLite2 ASN database or the paid GeoIP2 ISP database). It runs immediately after IP lookup and reads the ASN entity that IP lookup produces.

## Configuration

- [Enrichment schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/asn_lookups/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/asn_lookups.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

The enrichment accepts three optional parameters:

| Parameter         | Type             | Description                                                    |
| ----------------- | ---------------- | -------------------------------------------------------------- |
| `botAsnsFile`     | object           | A reference to a remotely hosted CSV file containing bad ASNs. |
| `botAsns`         | array            | An inline list of ASN entries to treat as bots.                |
| `bypassPlatforms` | array of strings | Platform codes for which the enrichment is skipped.            |

### `botAsnsFile`

Points to a CSV file with ASN numbers. The file should have a header row and use the format `number,name` (e.g., `174,"COGENT-174 - Cogent Communications, US"`). Only the number column is used for matching; the name column is for human readability and can be empty.

| Field      | Type   | Description                                                                                                        |
| ---------- | ------ | ------------------------------------------------------------------------------------------------------------------ |
| `uri`      | string | Base URI where the file is hosted. Supports `http:`, `s3:`, and `gs:` schemes. Must not end with a trailing slash. |
| `database` | string | The CSV filename.                                                                                                  |

### `botAsns`

An inline array of ASN objects. These are combined with any entries from `botAsnsFile`.

| Field  | Type    | Required | Description                                                              |
| ------ | ------- | -------- | ------------------------------------------------------------------------ |
| `asn`  | integer | Yes      | The autonomous system number.                                            |
| `name` | string  | No       | A human-readable label. Used only for clarity in the configuration file. |

### `bypassPlatforms`

An array of values that correspond to the `platform` field on events. Events with a matching platform skip this enrichment entirely, because it is expected for those platforms to originate from cloud or data center ASNs.

For example, server-side tracking (`"srv"`) and IoT (`"iot"`) events typically come from cloud providers, so flagging them as bots would produce false positives.

### Example configuration

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/asn_lookups/jsonschema/1-0-0",
    "data": {
        "name": "asn_lookups",
        "vendor": "com.snowplowanalytics.snowplow.enrichments",
        "enabled": true,
        "parameters": {
            "botAsnsFile": {
                "uri": "s3://my-private-bucket/third-party/bad-asn",
                "database": "bad-asn-list.csv"
            },
            "botAsns": [
                {
                    "asn": 123,
                    "name": "ASN 123"
                },
                {
                    "asn": 456
                }
            ],
            "bypassPlatforms": ["srv", "iot"]
        }
    }
}
```

> **Tip:** You can use the community-maintained [cpuchain/bad-asn-list](https://github.com/cpuchain/bad-asn-list) as a starting point for `botAsnsFile`. Host the CSV file in your own cloud storage to avoid depending on an external service at pipeline runtime.

The enrichment modifies the existing ASN entity to add a bot signal when a match is found.

## Output

This enrichment modifies the ASN entity (`iglu:com.snowplowanalytics.snowplow/asn/jsonschema/1-0-1`) that the [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) attaches to events where ASN information is available. When the event's ASN matches an entry in the bot list, the enrichment sets `likelyBot` to `true`:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/asn/jsonschema/1-0-1",
    "data": {
        "number": 16509,
        "organization": "Amazon.com, Inc.",
        "likelyBot": true
    }
}
```

If the ASN does not match any entry in the bot list, or the event's platform is in `bypassPlatforms`, the `likelyBot` field is not added to the entity.

| Field       | Type    | Description                                                                                                                      |
| ----------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `likelyBot` | boolean | Set to `true` or `false` for whether the ASN matches the bot list. Absent if the enrichment is skipped due to `bypassPlatforms`. |

---

# Bot detection enrichment
> Consolidate bot indicators from multiple enrichments into a single entity for easier filtering and analysis.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/

> **Note:** This enrichment is available since version 6.9.0 of Enrich.

Multiple enrichments can independently detect bots: [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/), [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/), and the [ASN lookup](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/). Without this enrichment, you would need to check each source separately during data modeling to determine whether an event came from a bot.

The bot detection enrichment consolidates these indicators into a single [entity](/docs/fundamentals/entities/). It reads the output of the contributing enrichments and produces a `bot_detection` entity with a simple `bot` boolean and a list of which sources flagged the event. This lets you filter bot traffic in your data models, or drop bot events entirely using a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/examples/#filtering-out-bots).

## How bot indicators are combined

The enrichment uses "any positive = bot" logic. If any of the enabled sources flags the event as coming from a bot, the event is classified as a bot. A negative result from one source does not override a positive result from another. This is because none of the existing enrichments can produce a strong "not a bot" result.

Each source contributes a indicator as follows:

| Source     | Flagged as bot when                                                                                                                       |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| YAUAA      | `deviceClass` is `"Robot"`, `"Robot Mobile"`, or `"Robot Imitator"`, or `agentClass` is `"Robot"` or `"Robot Mobile"` in the YAUAA entity |
| IAB        | `spiderOrRobot` is `true` in the IAB entity                                                                                               |
| ASN lookup | `likelyBot` is `true` in the ASN entity                                                                                                   |

For example, if YAUAA detects a bot based on user agent but IAB does not, the event is still classified as a bot. Similarly, if the ASN lookup flags the event based on a known bad ASN, that result stands regardless of what YAUAA or IAB report.

> **Note:** It is safe to enable all three sources (`useYauaa`, `useIab`, `useAsnLookups`) even if some of the underlying enrichments are not enabled. If a contributing enrichment is not enabled, its entity will not be present and that source is silently skipped.

## Configuration

- [Enrichment schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/bot_detection_enrichment_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/bot_detection_enrichment_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

The enrichment accepts three required boolean parameters that control which sources are consulted:

| Parameter       | Type    | Description                                                                                                                             |
| --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `useYauaa`      | boolean | Consult the [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) output for bot indicators.           |
| `useIab`        | boolean | Consult the [IAB enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) output for bot indicators.               |
| `useAsnLookups` | boolean | Consult the [ASN lookup enrichment](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/) output for bot indicators. |

### Example configuration

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/bot_detection_enrichment_config/jsonschema/1-0-0",
    "data": {
        "name": "bot_detection_enrichment_config",
        "vendor": "com.snowplowanalytics.snowplow.enrichments",
        "enabled": true,
        "parameters": {
            "useYauaa": true,
            "useIab": true,
            "useAsnLookups": true
        }
    }
}
```

The enrichment produces a single entity that summarizes all bot indicators for the event.

## Output

When enabled, this enrichment always attaches a `bot_detection` entity (`iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1-0-0`) to every event, even when no bot is detected.

| Field        | Type             | Description                                                                                                                 |
| ------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `bot`        | boolean          | `true` if any enabled source flagged the event as a bot, `false` otherwise.                                                 |
| `indicators` | array of strings | Which sources flagged the event as a bot. Possible values: `"yauaa"`, `"iab"`, `"asnLookups"`. Empty when `bot` is `false`. |

### Example: bot detected by multiple sources

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1-0-0",
    "data": {
        "bot": true,
        "indicators": ["yauaa", "iab"]
    }
}
```

### Example: no bot detected

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/bot_detection/jsonschema/1-0-0",
    "data": {
        "bot": false,
        "indicators": []
    }
}
```

---

# Campaign attribution enrichment
> Link events to marketing campaigns by extracting attribution data from query string parameters.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/

This enrichment can be used to link events to marketing campaigns, using the query string parameters.

When using online marketing campaigns to drive traffic to our website, it is usually possible to find information in the query string parameters to identify the particular campaign, medium and more.

A link for an online advertisement that brings users back to our site might look like:

```markup
https://www.acme.com/spring_offer_product?utm_source=influencer&utm_medium=blog&utm_campaign=spring_offer
```

This could result in the following fields being added to the enrich event:

| field          | value          |
| -------------- | -------------- |
| `mkt_source`   | `influencer`   |
| `mkt_medium`   | `blog`         |
| `mkt_campaign` | `spring_offer` |

The configuration of the enrichment defines which parameters in the URL (e.g. `utm_source`) map to which fields in the event (e.g. `mkt_source`) — see examples below.

In addition, this enrichment automatically knows about Google (corresponding to the `gclid` query string parameter), Microsoft (`msclkid`), and DoubleClick (`dclid`). For example, if the query string contains `&gclid=abc`, this will be the result:

| field         | value    |
| ------------- | -------- |
| `mkt_clickid` | `abc`    |
| `mkt_network` | `Google` |

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/campaign_attribution/jsonschema/1-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/campaign_attribution.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### Basic usage

Here’s an example for standard Google parameters. It specifies that the `utm_medium` parameter in the query string will map to the `mkt_medium` field in the event, and so on.

```json
    "parameters":{
      "mapping":"static",
      "fields":{
        "mktMedium":[
          "utm_medium"
        ],
        "mktSource":[
          "utm_source"
        ],
        "mktTerm":[
          "utm_term"
        ],
        "mktContent":[
          "utm_content"
        ],
        "mktCampaign":[
          "utm_campaign"
        ]
      }
    }
```

And here’s an example for Omniture (only the `mkt_campaign` field will be populated):

```json
    "parameters":{
      "mapping":"static",
      "fields":{
        "mktMedium":[
        ],
        "mktSource":[
        ],
        "mktTerm":[
        ],
        "mktContent":[
        ],
        "mktCampaign":[
          "cid"
        ]
      }
    }
```

### Supporting multiple parameters

What if some of your links use `utm_campaign=...` and some use `legacy_campaign=...`? In this case, you can configure more than one parameter name in the array, like so:

```json
    "parameters":{
      "mapping":"static",
      "fields":{
        ...
        "mktCampaign":[
          "utm_campaign",
          "legacy_campaign"
        ]
        ...
      }
    }
```

The same applies to other configuration options, namely `mktMedium`, `mktSource`, `mktTerm` and `mktContent`.

> **Note:** If the query string includes multiple acceptable parameters (e.g. both `utm_campaign` and `legacy_campaign`), the **first one listed in the configuration** will be used (_not_ the first one present in the query string).

Results:

| query string                           | value of `mkt_campaign` |
| -------------------------------------- | ----------------------- |
| `utm_campaign=abc`                     | `abc`                   |
| `legacy_campaign=abc`                  | `abc`                   |
| `legacy_campaign=abc&utm_campaign=def` | `def`                   |

### Click and network attribution

You can define which URL parameters are used to populate the `mkt_clickid` field (the defaults include `gclid`, `msclkid` and `dclid`). For each parameter, _the same configuration setting_ also defines what network — the `mkt_network` field — it corresponds to (by default, `gclid` corresponds to `Google`, `msclkid` to `Microsoft` and `dclid` to DoubleClick).

> **Tip:** You can configure any parameter names or network names, including your custom ones.

In the next example, we will customize the `mktClickId` configuration:

- First, we add support for `wbraid` and `gbraid` [parameters](https://support.google.com/analytics/answer/11367152?hl=en), which will be mapped to `Google` as the corresponding `mkt_network`.
- Second, we override the `msclkid` parameter, so that it maps to `NotMicrosoft` as the marketing network (instead of the default `Microsoft`).
- Third, we add a custom `xyzid` parameter that maps to the `XYZ` network.
- Other default mappings for `gclid` and `dclid` remain unaffected.

```json
    "parameters":{
      "mapping":"static",
      "fields":{
        ...
        "mktClickId": {
          "wbraid": "Google",
          "gbraid": "Google",
          "msclkid": "NotMicrosoft",
          "xyzid": "XYZ"
        }
        ...
      }
    }
```

> **Warning:** You should not use more than one click parameter in the query string (e.g. both `wbraid` and `gbraid`). If you do, one of them will be picked arbitrarily.

Results:

| query string            | value of `mkt_clickid` | value of `mkt_network` |
| ----------------------- | ---------------------- | ---------------------- |
| `wbraid=abc`            | `abc`                  | `Google`               |
| `msclkid=abc`           | `abc`                  | `NotMicrosoft`         |
| `xyzid=abc`             | `abc`                  | `XYZ`                  |
| `wbraid=abc&gbraid=def` | `abc` or `def` ⚠️      | `Google`               |

## Output

This enrichment populates the following fields of the atomic event :

| Field          | Purpose                                                                                                                                                                                                                                                           |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mkt_medium`   | The advertising or marketing medium, for example: `banner`, `email newsletter`.                                                                                                                                                                                   |
| `mkt_source`   | Identifies the advertiser, site, publication, etc. that is sending traffic to your property, for example: `newsletter4`, `billboard`.                                                                                                                             |
| `mkt_term`     | Identifies keywords (terms).                                                                                                                                                                                                                                      |
| `mkt_content`  | Used to differentiate similar content, or links within the same ad. For example, if you have two call-to-action links within the same email message, you can use `mkt_content` and set different values for each so you can tell which version is more effective. |
| `mkt_campaign` | The individual campaign name, slogan, promo code, etc. for a product.                                                                                                                                                                                             |
| `mkt_clickid`  | Click ID which resulted in the redirect/follow request                                                                                                                                                                                                            |
| `mkt_network`  | The advertising network name, either default determined from parameter for Click ID or custom specifically stated                                                                                                                                                 |

If the enrichment is not activated, those fields will not be populated.

---

# Cookie extractor enrichment
> Extract name-value pairs from first-party cookies and attach them to events as derived contexts.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/

This enrichment extracts name-value pairs from cookies set on the collector domain, attaching them to the event as derived contexts.

A powerful attribute of using a custom collector domain is the ability to capture values in first-party cookies set by other services such as ad servers or content management software (CMS). By capturing these cookie fields and attaching them to the event, you can use the data to better identify users of your website.

## Examples

The following table provides examples of how the cookie and collector domains interact, indicating whether or not the cookies can be accessed with this enrichment.

| Collector Domain  | Cookie Domain | Cookies Extracted |
| ----------------- | ------------- | ----------------- |
| c.snowplow\.io    | acme.com      | ❌                 |
| t.acme.com        | c.acme.com    | ❌                 |
| t.acme.com        | acme.com      | ✅                 |
| sp.track.acme.com | acme.com      | ✅                 |

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/cookie_extractor_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/cookie_extractor_config.json)

In the configuration we specify the list of cookie keys for which we want to extract the value and attach it to the event.

The example configuration is capturing Scala Stream Collector’s own “sp” cookie value but in practice we would probably want to extract other more valuable cookies available on the company domain.

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).
>
> To test first-party cookies, you will need to run Micro locally and [configure a custom DNS resolution rule](/docs/testing/snowplow-micro/local/remote-usage/#locally-resolving-an-existing-domain-name-to-micro).

## Input

This enrichment uses the `Cookie` HTTP header.

## Output

For **each** key listed in the configuration, a context with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.ietf/http_cookie/jsonschema/1-0-0) is added to the enriched event. Each context contains only one key/value pair.

---

# Cross-navigation enrichment
> Parse cross-navigation data from query string parameters to track user journeys across domains and platforms.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/

This enrichment parses the extended cross-navigation format in the `_sp` querystring parameter, and attaches a `cross_navigation` entity to an event.

Check out the [cross-navigation](/docs/events/cross-navigation/) page to learn why this can be useful.

The extended cross-navigation format is `_sp={domainUserId}.{timestamp}.{sessionId}.{subjectUserId}.{sourceId}.{sourcePlatform}.{reason}`. The `domainUserId` and `timestamp` fields will always be present, but some of the other fields may be null or empty. They're configured within the tracker.

If this enrichment isn't enabled, Enrich parses the `_sp` querystring parameter according to the short format, `_sp={domainUserId}.{timestamp}`

> **Note:** The pipeline will always populate the `refr_domain_userid` and `refr_dvce_tstamp` enriched event fields, even if this enrichment isn't enabled.

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/cross_navigation_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/cross_navigation_config.json)

```json
loading...
```

[View on GitHub](https://github.com/snowplow/enrich/blob/master/config/enrichments/cross_navigation_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Input

This enrichment extracts the `_sp` querystring parameter from the `page_url` field from the Snowplow event.

## Output

This enrichment adds a new derived entity to the enriched event based on [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/cross_navigation/jsonschema/1-0-0).

Enrich always populates the `refr_domain_userid` and `refr_dvce_tstamp` enriched event fields.

---

# Currency conversion enrichment
> Convert transaction values to a base currency using Open Exchange Rates API for standardized reporting.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/

This enrichment uses [Open Exchange Rates](https://openexchangerates.org/) to convert the currencies used in transactions. It requires an Open Exchange Rates account and API key.

When transactional data is collected in multiple currencies, it can be useful to convert it in the one that is used for reporting for instance. This could help to lower discrepancies when reporting revenue amounts across multiple currencies.

> **Warning:** This is an older and less actively maintained enrichment, and as such it has several limitations.
>
> - **It can only use the exchange rate from the end of the day prior to the event’s `collector_tstamp`.** You can’t apply a different rate (e.g. the one in effect when the event occurred), and you can’t pick a different timestamp field (e.g. `dvce_sent_tstamp`).
> - **It only works with the [`tr_` and `ti_` fields](/docs/fundamentals/canonical-event/#legacy-ecommerce-fields).** These fields are not very convenient to use and exist for legacy reasons. One of their significant downsides is that you have to send a separate event for the transaction itself and then an event for each of the order items in that transaction (as opposed to including all items in a single event).
>
> Over the years, it has become more idiomatic to use dedicated [entities](/docs/fundamentals/entities/) for order items in ecommerce transactions.
>
> We recommend to manage currency conversion downstream instead of using this enrichment. For example, you could bring currency exchange rate information into your data warehouse and join that data with your Snowplow data.

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/currency_conversion_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/currency_conversion_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

| **Field**      | **Description**                                                                                                                                                                                                |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `accountType`  | Level of Open Exchange Rates account. Must be “developer”, “enterprise”, or “unlimited”.                                                                                                                       |
| `apiKey`       | Open Exchange Rates API key                                                                                                                                                                                    |
| `baseCurrency` | Currency to convert all transaction values to                                                                                                                                                                  |
| `rateAt`       | Determines which exchange rate will be used. **Currently only “EOD\_PRIOR” is supported**, meaning that the enrichment uses the exchange rate from the end of the day prior to the event’s `collector_tstamp`. |

## Input

This enrichment uses the following fields :

- `tr_currency`
- `tr_total`
- `tr_tax`
- `tr_shipping`
- `ti_currency`
- `ti_price`

## Output

This enrichment updates the following fields of the atomic event :

| Field              | Purpose                                                                                          |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| `base_currency`    | Base currency code according to [ISO\_4217](https://en.wikipedia.org/wiki/ISO_4217#Active_codes) |
| `tr_total_base`    | Total amount of transaction in base currency                                                     |
| `tr_tax_base`      | Tax applied in base currency                                                                     |
| `tr_shipping_base` | Shipping cost in base currency                                                                   |
| `ti_price_base`    | Item price in base currency                                                                      |

---

# Custom API Request enrichment
> Enrich events with data from external HTTP APIs by making custom API requests during event processing.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/

The API Request Enrichment lets you perform dimension widening on a Snowplow event via your own or third-party proprietary http(s) API.

This enrichment gives you the flexibility to add additional data points for your events by pulling in data from other sources. Using a common key like a user ID or an email address for example, you might be able to add relevant information about a user to each event before it gets written to your database.

The configuration for this enrichment is all about connecting to your data source and fetching the relevant data points you want to enrich your events with.

If you’d like support in setting up or configuring this enrichment please contact us at <support@snowplowanalytics.com>.

## Configuration

- [schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-2)
- [example](https://github.com/snowplow/enrich/blob/master/config/enrichments/api_request_enrichment_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

The example configuration is using an imaginary api.acme.com RESTful service to widen Snowplow event with context containing information about users.

The configuration JSON for this enrichment contains four sub-objects:

1. `inputs` specifies the datapoint(s) from the Snowplow event to use as keys when performing your API lookup
2. `api` defines how the enrichment can access your API
3. `outputs` lets you tune how you convert the returned JSON into one or more self-describing JSONs ready to be attached to your Snowplow event
4. `cache` improves the enrichment's performance by storing values retrieved from the API

### Configuration

To go through each of sub-objects in more detail:

#### `inputs`

Specify an array of `inputs` to use as keys when performing your API lookup. Each input consists of a `key` and a source: either `pojo` if the datapoint comes from the Snowplow enriched event POJO, or `json` if the datapoint comes from a self-describing JSON inside one of the three JSON fields. The `key` can be referred to later in the `api.http.uri` property. Note that key name can contain only alphanumeric symbols, hyphens and underscores.

For `pojo`, the field name must be specified. A field name which is not recognized as part of the POJO will be ignored by the enrichment.

For `json`, you must specify the field name as either `unstruct_event`, `contexts` or `derived_contexts`. You must then provide two additional fields:

- `schemaCriterion` lets you specify the self-describing JSON you are looking for in the given JSON field. You can specify all SchemaVers (`*-*-*`), only the SchemaVer MODEL (e.g. `1-*-*`), MODEL plus REVISION (e.g. `1-1-*`) or a full MODEL-REVISION-ADDITION version (e.g. `1-1-1`)
- `jsonPath` lets you provide the [JSON Path statement](http://goessner.net/articles/JsonPath/) to navigate to the field inside the JSON that you want to use as the input.

The lookup algorithm is short-circuiting: the first match for a given key will be used.

#### `api`

The `api` section lets you configure how the enrichment should access your API. At the moment only `http` is supported, with this option covering both HTTP and HTTPS - the protocol on the `uri` field will determine which to use. Before R113, only `GET` was supported as the HTTP `method` for the lookup. Now it supports both `GET` and `POST`.

For the `uri` field, specify the full URI including the protocol. You can attach a querystring to the end of the URI. You can also embed the keys from your `inputs` section in the URI, by wrapping the key in `{{}}` brackets thus:

```json
"uri": "http://api.acme.com/users/{{client}}/{{user}}?format=json"
```

If a key required in the `uri` was not found in any of the `inputs`, then the lookup will not proceed, but this will **not** be flagged as a failure.

Make sure your `uri` is actually valid URI and does not contain any special symbols or spaces. Enrichment will "urlize" content of input extracted from event with `java.net.URLEncoder.encode` function, but safety of `uri` is on user's behalf.

Currently the only supported `authentication` option is `http-basic`: provide a `username` and/or a `password` for the enrichment to use to connect to your API using [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Some APIs use only the `username` or `password` field to contain an API key; in this case, set the other property to the empty string `""`.

If your API is unsecured (because for example it is only accessible from inside your private subnet, or using IP address whitelisting), then configure the `authentication` section like so:

```json
"authentication": { }
```

#### `outputs`

This enrichment assumes that your API returns a JSON, which will contain one or more _entities_ that you want to add to your event as derived contexts. Within the `outputs` array, each entry is a `json` sub-object that contains a `jsonPath` configuration field that lets you specify which part of the returned JSON you want to add to your enriched event. `$` can be used if you want to attach returned JSON as is.

If the JSON Path specified cannot be not found within the API's returned JSON, then the lookup (and thus the overall event) will be flagged as a failure.

The enrichment adds the returned JSON into the `derived_contexts` field within a Snowplow enriched event. Because all JSONs in the `derived_contexts` field must be self-describing JSONs, use the `schema` field to specify the Iglu schema URI that you want to attach to the event.

Example:

```json
GET http://api.acme.com/users/northwind-traders/123?format=json
{
  "metadata": {
    "whenCreated": 1448371243,
    "whenUpdated": 1448373431
  },
  "record": {
    "name": "Bob Thorpe",
    "id": "123"
  }
}
```

With this configuration:

```json
"outputs": [ {
  "schema": "iglu:com.acme/user/jsonschema/1-0-0",
  "json": {
    "jsonPath": "$.record"
  }
} ]
```

This would be added to the `derived_contexts` array:

```json
{
  "schema": "iglu:com.acme/user/jsonschema/1-0-0",
  "data": {
    "name": "Bob Thorpe",
    "id": "123"
  }
}
```

The `outputs` array must have at least one entry in it.

#### `cache`

A Snowplow enrichment can run many millions of time per hour, effectively launching a DoS attack on a data source if we are not careful. The `cache` configuration attempts to minimize the number of lookups performed.

The cache is an LRU (least-recently used) cache, where less frequently accessed values are evicted to make space for new values. The `uri` with all keys populated is used as the key in the cache. Configure the `cache` as follows:

- `size` is the maximum number of entries to hold in the cache at any one time (minimum value for is `1`)
- `ttl` is the number of seconds that an entry can stay in the cache before it is forcibly evicted. This is useful to prevent stale values from being retrieved in the case that your API can return different values for the same key over time.

To disable `ttl` so keys could be stored in cache until job is done `0` value should be used.

#### `ignoreOnError`

When set to `true`, no failed event will be emitted if the API call fails and the enriched event will be emitted without the context added by this enrichment.

### Data sources

The data source for this enrichment is the entire `enriched/good` event. More precisely, input data can be accessed in any of four forms:

- Snowplow Plain Old Java Object produced during common enrichment process
- [Snowplow self-describing event](/docs/fundamentals/events/#self-describing-events)
- [Entities](/docs/fundamentals/entities/) (contexts) attached to event by tracking SDK
- Derived entities (contexts) attached to event by [other enrichments](/docs/pipeline/enrichments/available-enrichments/)

More precise usage of these data sources is described in inputs section.

### Algorithm

This enrichment uses any 3rd party RESTful service to fetch data in JSON format. In most cases however you probably want to use your own private server to maintain acceptable performance since third-party service may cause serious slowdown of your enrichment process.

Here are some clues on how this enrichment will handle some exceptional cases:

- If the provided JSONPath is invalid - all events attempting to be enriched will be sent to `enriched/bad`
- If more than one context (derived or custom) matches `schemaCriterion` - the first one will be picked, even if the following have higher SchemaVer
- If the input's value is found in more than one source - the last one will be picked, so try to put the more precise input last (for example to get longitude/latitude pair use data from IP Lookup enrichment first and GPS-derived longitude/latitude second)
- If any of the input key wasn't found - the HTTP request won't be sent and a new context won't be derived, but the event will be processed as usual
- If the output's JSONPath wasn't found - the event will be sent to `enriched/bad` bucket
- If the server returned any non-successful response or timed-out - the event will be sent to `enriched/bad` bucket
- If the server response returned JSON which is invalidated by the schema provided in the output - the event will be sent to `shredded/bad`
- If the input JSONPath will match a non-primitive value in context or unstructured event, the enrichment will try to stringify it. An array will be concatenated with commas, `null` will be transformed to string "null", an object will be stringified and will result in invalid URL

### Data generated

This enrichment adds a new context to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0).

As during the API Request enrichment process the new context is added to `derived_contexts` of the enriched/good event, the data generated will end up in its own table determined by the custom `schema` key in `output` configuration sub-object.

---

# JavaScript enrichment code examples
> Example JavaScript enrichment implementations for common use cases and transformation patterns.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/examples/

Suppose you want to extend your events with a label that distinguishes traffic from internal and external systems. This is a good use case for [adding an entity](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#adding-extra-entities-to-the-event).

```js
const internalIPs = new Set([
  "1.2.3.4",
  "2.3.4.5"
]);

function process(event) {
  const ip = event.getUser_ipaddress();
  const traffic_source = internalIPs.has(ip) ? 'internal' : 'external';

  return [{
    schema: "iglu:com.my-company/traffic_source/jsonschema/1-0-0",
    data: { traffic_source }
  }];
}
```

> **Note:** This example is not copy-paste-able. You will need to define your own schema.

## Filtering out bots

You can use the [IAB enrichment](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) to identify bots. However, it does not remove the bot-generated events from your data — it only adds relevant information to the events (in the `derived_contexts` field), e.g.:

```json
{
    "schema": "iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0",
    "data": {
        "spiderOrRobot": false,
        "category": "BROWSER",
        "reason": "PASSED_ALL",
        "primaryImpact": "NONE"
    }
}
```

If you wish to discard the bot events (sending them to [failed events](/docs/fundamentals/failed-events/)), you can use the JavaScript enrichment:

```js
function process(event) {
  const entities = JSON.parse(event.getDerived_contexts());
  if (entities) {
    for (const entity of entities.data) {
      if (entity.schema.startsWith('iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1') &&
          entity.data.spiderOrRobot) {
        throw "Filtered a spider/bot event";
      }
    }
  }
}
```

> **Note:** You need to use Enrich 3.8.0+ (or Snowplow Micro 1.7.1+) to be able to access the derived entities via `event.getDerived_contexts()`. In prior versions, this function always returns `null`.

For special cases, you can also dispense with the IAB enrichment and add something simpler:

```js
const botPattern = /.*Googlebot.*/;

function process(event) {
  const useragent = event.getUseragent();

  if (useragent !== null && botPattern.test(useragent)) {
    throw "Filtered event produced by Googlebot";
  }
}
```

## Omitting information based on a condition

Let’s say you want to omit certain user information depending on which country the user is in. While [PII pseudonymization enrichment](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/) is very useful for dealing with PII, it does not support such conditions. JavaScript enrichment to the rescue!

```js
function process(event) {
  if (event.getGeo_country() === 'US') {
    event.setGeo_latitude(null);
    event.setGeo_longitude(null);
  }
}
```

---

# Custom JavaScript enrichment
> Write custom JavaScript code to transform and enrich events with flexible data manipulation logic.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/

With this enrichment, you can write a JavaScript function to be executed for each event. Use this enrichment to apply your own business logic to your events, including:

- [Adding extra data](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#adding-extra-entities-to-the-event) to the event in the form of [entities](/docs/fundamentals/entities/)
- [Modifying event fields](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#modifying-event-fields-directly) directly
- [Discarding the event](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#discarding-the-event) so that it goes to [failed events](/docs/fundamentals/failed-events/) rather than your data destination

## Configuration

- [schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-1)
- [example](https://github.com/snowplow/enrich/blob/master/config/enrichments/javascript_script_enrichment.json)

> **Note:** Notice that in the configuration file you will need to provide the JavaScript code of your enrichment encoded in base64.

## Output

This enrichment is the only one that can both update fields of the atomic event in-place or/and add derived contexts to the enriched event.

---

# Test JavaScript enrichments locally
> Test JavaScript enrichment code locally using Node.js before deploying to production pipelines.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/

You can (and should!) test your enrichment with [Snowplow Micro](/docs/testing/snowplow-micro/) before adding it to your production pipeline.

## Basic setup

While it's possible to use Micro [through Console](/docs/testing/snowplow-micro/console/), running it locally has the benefit that you can add logging statements and look at their output.

If you haven't run Micro on your machine before, take a look at the [usage guide](/docs/testing/snowplow-micro/local/). Adding the JavaScript enrichment to Micro is similar to [adding other enrichments](/docs/testing/snowplow-micro/local/enrichments/).

> **Tip:** Testing `.js` files directlyWhile you can add the full enrichment configuration in a JSON file (with your JavaScript encoded in base64), you can also just drop a `.js` file in your enrichments folder. Micro will interpret it as JavaScript enrichment code. This way, you don’t need to keep converting your enrichment to base64 as you iterate on it. _(Make sure to either use a `.json` or a `.js` file, but not both.)_

For example, if your enrichment code is in `script.js` in the current directory, you can execute the following command to run it in Micro:

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/script.js,destination=/config/enrichments/script.js \
snowplow/snowplow-micro:4.1.1
```

Next, point your tracking code to `localhost:9090` (see the [usage guide](/docs/testing/snowplow-micro/local/#sending-events-to-micro) for more details). Now the events you send to Micro will go through your enrichment code.

## Debugging

When debugging a JavaScript enrichment with Micro, there are a few sources of useful information: the [dashboard](/docs/testing/snowplow-micro/ui/) (available since Micro 2.0.0), the logs and the [REST API](/docs/api-reference/snowplow-micro/api/).

### Dashboard

The dashboard will show you which events are enriched successfully and which fail.

For each failed event, you can also click on the failure badge to look at the detailed failure message.

### Logs

The logs will quickly show you any failure. For example, if you forgot to add your schema:

```text
[WARN] EventLog - BAD {
  "schemaKey" : "iglu:com.my-company/my-schema/jsonschema/1-0-0",
  "error" : {
    "error" : "ResolutionError",
    ...
```

Or if you made a typo in a function name:

```text
[WARN] EventLog - BAD {
  "error" : "Error during execution of JavaScript function: [TypeError: event.getLongtitude is not a function in <eval> at line number 2]"
}
```

> **Note:** If you `throw` in your JavaScript code to [discard the event](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#discarding-the-event), you will get a similar failure message in the logs, but it would be expected.

### Tracing

You can also add temporary `print` statements to your code to inspect certain values.

> **Warning:** Make sure to remove all debug printing when adding the enrichment to a production pipeline.

Here’s an example:

```js
print(`--- URL is: ${event.getPage_url()}`);
```

This will be printed directly to the output from Micro (in the terminal — currently this information is not available in the Micro dashboard).

> **Info:** There is no `console.log`Note that `console.log` and `console.dir` familiar to JavaScript developers _will not_ work in this context.

If you want to pretty-print an object or array, the best approach is to use `JSON.stringify`:

```js
const myEvent = JSON.parse(event.getUnstruct_event());
print(JSON.stringify(myEvent.data, null, 2));
```

### REST API

Using Micro’s [REST API](/docs/api-reference/snowplow-micro/api/), you can verify that your code modifies event fields or adds entities as expected. The most useful endpoint is `/micro/good`, which will return all successfully processed events:

```bash
curl localhost:9090/micro/good
```

For example, if your code adds some entities, you will find them in the output from the REST API, under `derived_contexts` field for each event:

```json
...
"derived_contexts": {
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "...",
      "data": { ... }
    },
    ...
  ]
}
...
```

## Iterating on code and schemas

If your enrichment [adds extra entities to the event](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/#adding-extra-entities-to-the-event), you might be iterating on the schemas for those entities at the same time you are iterating on your enrichment code.

You can simplify your development and testing flow by [adding schemas to Micro](/docs/testing/snowplow-micro/local/schemas/#adding-schemas-directly-to-micro) as local files. For example, your current directory might look like this:

```text
script.js
schemas
└── com.my-company
    └── my-schema
        └── jsonschema
            └── 1-0-0
```

This command will add both the script and the schema to Micro:

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/schemas,destination=/config/iglu-client-embedded/schemas \
--mount type=bind,source=$(pwd)/script.js,destination=/config/enrichments/script.js \
snowplow/snowplow-micro:4.1.1
```

## Testing with more complete events

Here are a few more tips on getting the most out of Micro when testing your JavaScript enrichment.

### Enriched fields

If your enrichment uses fields filled by other enrichments, you will want to [enable those](/docs/testing/snowplow-micro/local/enrichments/) as well.

### IP addresses

If your enrichment relies on the `user_ipaddress` field or the `geo_*` fields, with the default setup you will discover that `user_ipaddress` is your local one (e.g. something like `192.168.0.42`). Subsequently, the `geo_*` fields are all `null`. That’s because you have both Micro and your tracking code running locally on the same machine.

If you are using a web browser to test your site or app, you can spoof a specific IP address using a browser plugin that sets an `X-Forwarded-For` header. For example, here are plugins for [Chrome](https://chrome.google.com/webstore/detail/x-forwarded-for-header/hkghghbnihliadkabmlcmcgmffllglin/) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/x-forwarded-for-injector/). Install the plugin and set the IP address to your liking.

Alternatively, check out the section on [exposing Micro via a public domain name](/docs/testing/snowplow-micro/local/remote-usage/#exposing-micro-via-a-public-domain-name) — with this approach you can get a public URL for your Micro, to which you can point your tracking code. Now the interaction between your tracking and your Micro will go through the internet, and you will get a realistic IP address in your events.

### Cookies

If your enrichment relies on cookies, you may find it difficult to reproduce all the cookies set by your website or app in a local setup.

For a solution, check out the section on [locally resolving an existing domain name to Micro](/docs/testing/snowplow-micro/local/remote-usage/#locally-resolving-an-existing-domain-name-to-micro). With this approach you can hook into an existing website or app, receiving all cookies in your Micro.

> **Tip:** If the values of cookie-based fields (e.g. `network_userid`) are not what you expect, make sure you [configure Micro](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-collector-configuration) to use the same cookie name as your website or app (the default is `micro`). For example, to set it to `sp`:
>
> ```bash
> docker run ... \
> snowplow/snowplow-micro:4.1.1 \
> -Dcollector.cookie.name=sp
> ```

---

# Write JavaScript enrichment functions
> Write custom JavaScript functions to transform events with access to event properties and utility functions.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/writing/

Your JavaScript enrichment code should contain a function called `process`:

- This function will receive each event as its only argument
- It can optionally return an array of [entities](/docs/fundamentals/entities/) to be added to the event (under `derived_contexts`)
- Any uncaught exceptions will result in [failed events](/docs/fundamentals/failed-events/)

```js
function process(event) {
  // do something with the event...
  ...

  // add entities to the event
  return [ ... ];
}
```

You can only have _one_ JavaScript enrichment, and hence a single `process` function for your pipeline. However, you can split more complex logic into multiple helper functions and variables as you see fit, as long as you comply with the above interface.

> **Tip:** JavaScript enrichment uses the [Nashorn Engine](https://docs.oracle.com/en/java/javase/11/nashorn/introduction.html) and since version 3.0.0 of Enrich, many features of ECMAScript 6 are supported. For a list of those features, please refer to [this OpenJDK proposal](http://openjdk.java.net/jeps/292). Regarding the features the proposal says “might be feasible” in the future, as of 2023 our testing shows that classes and generators don't work, but tail calls do.

## Best practices

Before we dive into it, here are a few general tips:

- Make sure your code works for _all_ of your events, not just the particular types of events you are interested in. Remember, unhandled exceptions will result in [failed events](/docs/fundamentals/failed-events/).
- Don’t try to share state across multiple enriched events. Enrichments are run inside a highly parallel application with multiple independent instances, so this will not work.
- In your enrichment code, avoid CPU-intensive tasks (e.g. encryption) and IO-intensive tasks (e.g. requests to an external service) without thoroughly benchmarking the impact they might have on your event processing time.
- The enrichment code has access to the Java standard library and therefore to the filesystem of the machine it’s running on. Proceed with caution when copying code from untrusted sources.

## Inspecting the event fields

Regardless of what you want to do with the event, you will likely want to inspect some of the data within. For instance, to get the `app_id` field:

```js
function process(event) {
  const appId = event.getApp_id();

  ...
```

There are getter methods available for each of the [standard event fields](/docs/pipeline/enriched-tsv-format/?fields=table) — just capitalize the first letter of the field and prepend it with `get`, for example `event.getUser_ipaddress()` or `event.getGeo_country()`.

> **Note:** One exception is `refr_device_tstamp`, where the getter method is called `getRefr_dvce_tstamp` and _not_ `getRefr_device_tstamp`.

## Inspecting self-describing events and entities

If your event is a [self-describing event](/docs/fundamentals/events/#self-describing-events), you might want to access its fields. Here’s how to do it:

```js
function process(event) {
  ...

  const myEvent = JSON.parse(event.getUnstruct_event());

  if (myEvent) {
    // the schema of your self-describing event
    const mySchema = myEvent.data.schema;

    // the custom data in your self-describing event
    const myData = myEvent.data.data;

    ...
```

> **Note:** For events other than self-describing events, `getUnstruct_event()` will return `null`. The pattern above works, as `null` is a valid input for `JSON.parse`.

You can access any [entities](/docs/fundamentals/entities/) in the event in a similar fashion:

```js
function process(event) {
  ...

  const entities = JSON.parse(event.getContexts());

  if (entities) {
    // loop through the entities
    for (const entity of entities.data) {
      if (entity.schema.startsWith('iglu:org.my-company/my-schema/jsonschema/1')) {
        // work with the entity
        const myField = entity.data.myField;

        ...
      }
    }
  }

  ...
```

> **Note:** For events with no entities attached, `getContexts()` will return `null`. The pattern above works, as `null` is a valid input for `JSON.parse`.

> **Note:** For derived entities (added by other enrichments), you can use `event.getDerived_contexts()` in the same way as above. Note that this is only supported since Enrich 3.8.0 (and Snowplow Micro 1.7.1). In prior versions, this function always returns `null`.

## Adding extra entities to the event

> **Tip:** Adding entities is the preferred way of augmenting your events with extra information, because it preserves the original event fields intact.
>
> In some cases, you might choose to update existing fields instead of adding entities. However, keep in mind that if you overwrite a field, you won’t have access to its original value in your data warehouse or lake.

The (optional) return value of the `process` function is an array of extra entities to add to the event. So adding entities is as simple as returning them!

```js
function process(event) {
  ...

  return [
    {
      schema: 'iglu:com.my-company/traffic-source/jsonschema/1-0-0',
      data: {
        traffic_source: 'internal'
      }
    }
  ];
}
```

The entities you add with this method will be _derived_ entities, similar to what other enrichments add. You will find them in the `derived_contexts` field of the event.

**Behavior for special values (e.g. NaN)**

Your array of entities will be passed to `JSON.stringify()` before being attached to the event. This is irrelevant for you, unless your entities have `NaN` values (they will become `null`), or `undefined` values (they will be dropped), or circular references (an exception will be thrown).

If you are still iterating on the schema while writing the JavaScript code, you might find the setup described in the [testing guide](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/#iterating-on-code-and-schemas) very useful.

> **Warning:** Make sure that the schemas of your entities are defined and accessible to your pipeline.

## Modifying event fields directly

Sometimes you will want to modify the original event fields directly.

> **Warning:** Keep in mind that the old value of a modified field will not be available in your data warehouse or lake. However, that might be your goal.

Just like with getters, there are setter methods available for each of the [standard event fields](/docs/pipeline/enriched-tsv-format/?fields=table):

```js
function process(event) {
  event.setMkt_source('Facegoog');
  event.setGeo_latitude(null);
  event.setGeo_longitude(null);

  ...
```

> **Note:** One exception is `refr_device_tstamp`, where the setter method is called `setRefr_dvce_tstamp` and _not_ `setRefr_device_tstamp`.

## Modifying self-describing events and entities

If you want to modify the self-describing event fields or the entities attached to the event, you will need to reverse the steps you took to fetch them.

For self-describing events:

```js
function process(event) {
  ...

  // unpack the self-describing event
  const myEvent = JSON.parse(event.getUnstruct_event());

  if (myEvent && myEvent.data.schema === ...) {
    // update a field inside
    myEvent.data.data.myField = 'new value';

    // pack the self-describing event back
    event.setUnstruct_event(JSON.stringify(myEvent));
  }

  ...
```

For entities:

```js
function process(event) {
  ...

  // unpack the entities
  const entities = JSON.parse(event.getContexts());

  if (entities) {
    // loop through the entities
    for (const entity of entities.data) {
      if (entity.schema === ...) {
        // update a field inside
        entity.data.myField = entity.data.myField + 1;
      }
    }

    // pack the entities back
    event.setContexts(JSON.stringify(entities));
  }

  ...
```

> **Note:** You might be tempted to update derived entities in a similar way by using `event.setDerived_contexts()`. However, this is not supported (the function exists, but has no effect). Instead, refer to the [Erasing derived contexts](#erasing-derived-contexts) section.

## Erasing derived contexts

Starting with Enrich 5.4.0, it is possible to erase derived contexts.

This feature can be used to update existing derived contexts as well. The way to do that is shown in the below example:

```js
function process(event) {
  const derived = JSON.parse(event.getDerived_contexts())

  // erase the existing contexts from the event
  event.eraseDerived_contexts()

  // modify the contexts
  for (const entity of derived.data) {
    if (entity.schema === ...) {
      // update a field inside
      entity.data.myField = entity.data.myField + 1
    }
  }

  // returned the updated array of derived contexts, which will replace the original one
  return derived.data
}
```

## Discarding the event

Sometimes you don’t want the event to appear in your data warehouse or lake, e.g. because you suspect it comes from a bot and not a real user.

Starting with Enrich 5.3.0, it is possible to drop an event by calling `event.drop()` in JavaScript enrichment code:

```js
const botPattern = /.*Googlebot.*/;

function process(event) {
  const useragent = event.getUseragent();

  if (useragent !== null && botPattern.test(useragent)) {
    event.drop();
  }
}
```

This mechanism can be used to drop not only good events, but also invalid events. The dropped events will not be sent to any stream or destination, thus lowering the infrastructure costs.

> **Warning:** There is no way to recover dropped events therefore use it with caution.

Another way to discard events is throwing an exception in your JavaScript code, which will send the event to [failed events](/docs/fundamentals/failed-events/):

```js
const botPattern = /.*Googlebot.*/;

function process(event) {
  const useragent = event.getUseragent();

  if (useragent !== null && botPattern.test(useragent)) {
    throw "Filtered event produced by Googlebot";
  }
}
```

> **Warning:** This will create an “enrichment failure” failed event, which may be tricky to distinguish from genuine failures in your enrichment code, e.g. due to a mistake.

## Accessing Java methods

Because the JavaScript enrichment runs inside the Enrich application, it has access to the Java standard library, as well as _some_ Java libraries (the ones used by Enrich). You can call Java methods via their fully qualified path, for example:

```js
function process(event) {
  ...

  const salt = 'pepper';
  const hashedIp = org.apache.commons.codec.digest.DigestUtils.sha256Hex(event.getUser_ipaddress() + salt);

  ...
}
```

## Passing an object of parameters

Starting with Enrich 4.1.0, it is possible to pass an object of parameters to the JS enrichment.

You can pass these parameters in the enrichment configuration, for example:

```json
{
	"schema": "iglu:com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-1",

	"data": {

		"vendor": "com.snowplowanalytics.snowplow",
		"name": "javascript_script_config",
		"enabled": true,
		"parameters": {
			"script": "script",
			"config": {
				"foo": 3,
				"nested": {
					"bar": "test"
				}
			}
		}
	}
}
```

The parameter object can be accessed in JavaScript enrichment code via the _second_ parameter of the `process` function, for example:

```js
function process(event, params) {
  event.setApp_id(params.nested.bar);
  return [];
}
```

> **Tip:** This is useful when you want to quickly reconfigure the enrichment without updating the JavaScript code.

## Accessing the HTTP request headers

Starting with Enrich 5.1.1, it is possible to access the HTTP headers that came with the original request to the Collector. The array of headers is passed in the _third_ argument to the `process` function. Each header is a string in the format `HEADER:value`.

> **Tip:** Per the HTTP specification, headers are not case sensitive, so we recommend using a case insensitive match, like what is shown below. It is also a good practice to trim whitespace from the value, as there might or might not be some space around the `:` character.

For example, if you want to pass some token via a custom `X-JWT` header and then use it in the enrichment:

```js
function process(event, params, headers) {
  for (header of headers) {
    const token = header.match(/X-JWT:(.+)/i)
    if (token) {
      const value = token[1].trim()
      // do something with the value
    }
  }
}
```

---

# Custom SQL query enrichment
> Query relational databases during enrichment to attach lookup data from MySQL, PostgreSQL, or other SQL databases.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/

The SQL Query Enrichment lets you perform dimension widening on a Snowplow event via your own internal relational database.

If you have data points that you’d like to use to enrich your event data collected with Snowplow that live in a data base, this enrichment will help you to query for the fields you want to add.

Currently supported database types:

- MySQL, plus variants which speak MySQL (e.g. MariaDB, Amazon Aurora)
- PostgreSQL, plus variants which speak PostgreSQL

We don’t recommend to use this enrichment with analytical databases which support minimal (50-100) concurrent queries (e.g. Redshift).

For help with configuring this enrichment and getting it live on your pipeline please contact us at <support@snowplowanalytics.com>.

## Configuration

- [schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/sql_query_enrichment_config/jsonschema/1-0-1)
- [example](https://github.com/snowplow/enrich/blob/master/config/enrichments/sql_query_enrichment_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Hypothetical example

Below you can see an example configuration using imaginary PostgreSQL database with CRM data, used to widen Snowplow event with context containing information about users.

The configuration JSON for this enrichment contains four sub-objects:

1. `inputs` specifies the datapoint(s) from the Snowplow event to use as values to substitute placeholders in `query` when performing SQL query
2. `database` defines how the enrichment can access your relational database
3. `query` defines prepared SQL statement to query your database
4. `output` lets you tune how you convert the returned row(s) into one or more self-describing JSONs ready to be attached to your Snowplow event
5. `cache` improves the enrichment’s performance by storing rows retrieved from your relational database

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/sql_query_enrichment_config/jsonschema/1-0-0",
  "data": {
    "name": "sql_query_enrichment_config",
    "vendor": "com.snowplowanalytics.snowplow.enrichments",
    "enabled": true,
    "parameters": {
      "inputs": [
        {
          "placeholder": 1,
          "pojo": {
            "field": "user_id"
          }
        },
        {
          "placeholder": 1,
          "json": {
            "field": "contexts",
            "schemaCriterion": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-*-*",
            "jsonPath": "$.userId"
          }
        },
        {
          "placeholder": 2,
          "pojo": {
            "field": "app_id"
          }
        }
      ],
      "database": {
        "postgresql": {
          "host": "rdms.intra.acme.com",
          "port": 5439,
          "sslMode": true,
          "username": "snowplow_enrich_ro",
          "password": "1asIkJed",
          "database": "crm"
        }
      },
      "query": {
        "sql": "SELECT username, email_address, date_of_birth FROM tbl_users WHERE user = ? AND application = ? LIMIT 1"
      },
      "output": {
        "expectedRows": "AT_MOST_ONE",
        "json": {
          "schema": "iglu:com.acme/user/jsonschema/1-0-0",
          "describes": "ALL_ROWS",
          "propertyNames": "CAMEL_CASE"
        }
      },
      "cache": {
        "size": 3000,
        "ttl": 60
      }
    }
  }
}
```

## Configuration

#### `inputs`

Specify an array of `inputs` to put instead of placeholders in prepared statement. Each input consists of a `placeholder` and a source: either `pojo` if the datapoint comes from the Snowplow enriched event POJO, or `json` if the datapoint comes from a self-describing JSON inside one of the three JSON fields. The `placeholder` is the index of the `?` SQL placeholder which this input will be used to populate. It must be an integer greater than or equal to 1. For the `pojo` source, the field name must be specified. A field name which is not recognized as part of the POJO will be ignored by the enrichment. For the `json` source, you must specify the field name as either `unstruct_event`, `contexts` or `derived_contexts`. You must then provide two additional fields:

- `schemaCriterion` lets you specify the self-describing JSON you are looking for in the given JSON field. You can specify only the SchemaVer MODEL (e.g. `1-*-*`), MODEL plus REVISION (e.g. `1-1-*`) or a full MODEL-REVISION-ADDITION version (e.g. `1-1-1`)
- `jsonPath` lets you provide the [JSON Path statement](https://github.com/gatling/jsonpath#jsonpath) to navigate to the field inside the JSON that you want to use as the input

The lookup algorithm is short-circuiting: the first match for a given key will be used.

#### `database`

The `database` section lets you configure how the enrichment should access your relational database. At the moment `postgresql` and `mysql` are supported, with the same fields available for both. Please see the **Database support** section above for notes on compatibility. Populate all properties in the `postgresql` or `mysql` configuration object, as follows:

- `host`, the hostname or IP address of the database server or cluster
- `port`, the port to connect to the database on
- `sslMode`, whether the database requires connections to use SSL
- `username`, the username to login to the database using
- `password`, the password for this username
- `database`, the name of the specific database to run the query against

**We strongly recommend that the username have minimal read-only permissions on just the entities required to execute the SQL query.** If your database server has additional authentication in place, such as IP whitelisting, you will need to configure this security to permit access from all of your servers running the Snowplow Enrichment process.

#### `query`

Consists of a single `sql` key with SQL query in a form of prepared statement to run on your relational database to return row(s) to attach to this event. Form of a prepared statement means it can has placeholders (`?`), which will be replaced with actual values, extracted from `input`s corresponding to their indexes. Some notes on the behavior:

- If a placeholder index required in the `sql` prepared statement is not found in any of the `inputs`, then the lookup will not proceed, but this will **not** be flagged as a failure
- A final `;` is optional
- Values will be inserted into prepared statement according to their type: strings will be quoted, number won’t
- **This enrichment makes no attempt to sanitize the SQL statement, nor to verify that the SQL statement does not have harmful side effects (such as SQL injection)**

#### `output`

The enrichment adds the returned row(s) into the `derived_contexts` field within a Snowplow enriched event. Because all JSONs in the `derived_contexts` field must be self-describing JSONs, use the `schema` field to specify the Iglu schema URI that you want to attach to the event. The `expectedRows` enum defines expected SQL output and can take the following values:

- `EXACTLY_ONE` – exactly one row is expected. 0 or 2+ rows will throw an error, causing the entire event to fail processing
- `AT_MOST_ONE` – either one or zero rows is expected. 2+ rows will throw an error
- `AT_LEAST_ZERO` – between 0 and N rows are expected – in JSON terms we are dealing with an array of results
- `AT_LEAST_ONE` – between 1 and N rows are expected – i.e. an array of results. 0 rows will throw an error

It is on user’s behalf to make sure SQL query returns correct amount of rows. The `describes` property dictates whether the `schema` is the self-describing schema for all rows returned by the query, or whether the `schema` should be attached to each of the returned rows:

- `ALL_ROWS` means that the `schema` should box all returned rows – i.e. one context will always be added to `derived_contexts`, regardless of how many rows that schema contains
- `EVERY_ROW` means that the `schema` should be attached to each returned row – so e.g. if 3 rows are returned, 3 contexts with this same schema will be added to `derived_contexts`

The `propertyNames` property supports reformatting of the returned columns to fit the JSON Schema’s conventions better. Supported options are:

- `AS_IS` – preserve the column names exactly as they are
- `CAMEL_CASE` – so `date_of_birth` becomes `dateOfBirth`
- `PASCAL_CASE` – so `date_of_birth` becomes `DateOfBirth`
- `SNAKE_CASE` – so `dateOfBirth` becomes `date_of_birth`
- `LOWER_CASE` – changes all characters to lowercase
- `UPPER_CASE` – changes all characters to uppercase

If these options aren’t bespoke enough, remember that you can use column aliasing in your SQL statement to tweak individual column names.

#### `cache`

A Snowplow enrichment can run many millions of time per hour, effectively launching a DoS attack on a data source if we are not careful. The `cache` configuration attempts to minimize the number of lookups performed. The cache is an LRU (least-recently used) cache, where less frequently accessed values are evicted to make space for new values. For the cache key, we use a complex object with an underlying Indexed HashMap, consisting of placeholder numbers as keys and extracted values as HashMap values. You can configure the `cache` as follows:

- `size` is the maximum number of entries to hold in the cache at any one time
- `ttl` is the number of seconds that an entry can stay in the cache before it is forcibly evicted. This is useful to prevent stale values from being retrieved in the case that your DB can return different values for the same key over time

#### `ignoreOnError`

When set to `true`, no failed event will be emitted if the SQL query fails and the enriched event will be emitted without the context added by this enrichment.

## Examples

### Single row

With this configuration:

```json
...
  "query": {
    "sql": "SELECT username, date_of_birth FROM tbl_users WHERE user = ?"
  },
  "output": {
    "expectedRows": "AT_MOST_ONE",
    "json": {
      "schema": "iglu:com.acme/user/jsonschema/1-0-0",
      "describes": "ALL_ROWS",
      "propertyNames": "CAMEL_CASE"
    }
  },
...
```

And this query result:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;
| username | date_of_birth |
| -------- | ------------- |
| karl     | 1980-06-12    |
```

This would be added to the `derived_contexts` array:

```json
{
  "schema": "iglu:com.acme/user/jsonschema/1-0-0",
  "data": {
    "username": "karl",
    "dateOfBirth": "1980-06-12T00:00:00"
  }
}
```

With this query result:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;
| username | date_of_birth |
| -------- | ------------- |
```

No context would be added to the `derived_contexts` array, but the event would continue processing. With this query result:

```sql
SELECT username, date_of_birth FROM tbl_users WHERE user = 123;
| username | date_of_birth |
| -------- | ------------- |
| karl     | 1980-06-12    |
| mary     | 1975-03-22    |
```

An error would be triggered, on account of the `AT_MOST_ONE` setting, and the event would be rejected.

### Multiple rows

With this configuration:

```json
...
  "query": {
    "sql": "SELECT * FROM product WHERE category = ?"
  },
  "output": {
    "expectedRows": "AT_LEAST_ZERO",
    "json": {
      "schema": "iglu:com.acme/products/jsonschema/1-0-0",
      "describes": "ALL_ROWS",
      "propertyNames": "AS_IS"
    }
  },
...
```

And this query result:

```sql
SELECT * FROM product WHERE category = 'homeware';
| SKU | prod_name |
| --- | --------- |
| 123 | iPad      |
| 456 | Ray-Bans  |
```

This single context would be added to the `derived_contexts` array:

```json
{
  "schema": "iglu:com.acme/products/jsonschema/1-0-0",
  "data": [
    {
      "SKU": "123",
      "prod_name": "iPad"
    },
    {
      "SKU": "456",
      "prod_name": "Ray-Bans"
    }
  ]
}
```

If we change the configuration to:

```json
...
  "output": {
    "expectedRows": "AT_LEAST_ZERO",
    "json": {
      "schema": "iglu:com.acme/product/jsonschema/1-0-0",
      "describes": "EVERY_ROW",
      "propertyNames": "LOWER_CASE"
    }
  }
...
```

Then two contexts would be added to the `derived_contexts` array:

```json
{
  "schema": "iglu:com.acme/product/jsonschema/1-0-0",
  "data": {
    "sku": "123",
    "prod_name": "iPad"
  }
}
{
  "schema": "iglu:com.acme/product/jsonschema/1-0-0",
  "data": {
    "sku": "456",
    "prod_name": "Ray-Bans"
  }
}
```

### Algorithm

This enrichment uses any relational database to fetch data in JSON format. Here are some clues on how this enrichment will handle some exceptional cases:

- if provided JSONPath is invalid – all events attempted to being enriched will be sent to `enriched/bad`
- if more than one context (derived or custom) matches `schemaCriterion` – first one will be picked, no matter if following have higher SchemaVer
- if input’s value found more than in one sources – last one will be picked, so try to put more precise input last (for example to get longitude/latitude pair use data from IP Lookup enrichment first and GPS-derived longitude/latitude second)
- if any of input key wasn’t found – SQL query won’t be performed and new context won’t be derived, but event will be processed as usual
- if DB query wasn’t successful for any reason or timed-out – event will be sent to `enriched/bad` bucket
- if DB connection was lost – event will be sent to `enriched/bad` bucket, but on next event, enrichment will try to reestablish connection
- all non-primitive values (objects, arrays) and `null`s will be interpreted as not-found values

### Data generated

This enrichment adds a new context to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0).

As during the SQL Query enrichment process the new context is added to `derived_contexts` of the enriched/good event, the data generated will end up in its own table determined by the custom `schema` key in `output` configuration sub-object.

---

# Event fingerprint enrichment
> Generate unique fingerprints for events using hash functions to enable deduplication and event matching.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/

This enrichment computes the fingerprint of an event using the query string parameters.

Both the key and the value of all query string parameters are used to compute the fingerprint, except for the fields specified in `excludeParameters`.

This is helpful when de-duplicating events.

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/event_fingerprint_config/jsonschema/1-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/event_fingerprint_enrichment.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

`hashAlgorithm` determines the algorithm that should be used to calculate the hash. Supported hashing algorithms are:

- [MD5](https://en.wikipedia.org/wiki/MD5)
- [SHA1](https://en.wikipedia.org/wiki/SHA-1)
- [SHA256](https://en.wikipedia.org/wiki/SHA-2)
- [SHA384](https://en.wikipedia.org/wiki/SHA-2)
- [SHA512](https://en.wikipedia.org/wiki/SHA-2)

The example configuration below would use all the parameters to compute the hash **except** the event ID (`eid`) and the device sent timestamp (`stm`).

```json
    "parameters": {
      "excludeParameters": [
        "eid",
        "stm"
      ],
      "hashAlgorithm": "MD5"
    }
```

Removing `stm` can be a good idea because in the scenario where tracker doesn't receive an acknowledgement after sending an event once and retries, the two copies of the event will have different `stm`s, whereas they are the same event.

Similarly, not much is gained by including the event ID in the hash given that this field is field is already used for de-duplication.

## Input

Query string parameters.

## Output

This enrichment will populate the field `event_fingerprint` of the atomic event.

---

# HTTP header extractor enrichment
> Extract HTTP request headers using regex patterns and attach them to events as derived contexts.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/

This enrichment can extract name/value pairs from the HTTP headers and attach them to the event as derived contexts.

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/http_header_extractor_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/http_header_extractor_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

All the headers that will match the regex defined in `headersPattern` parameter will be attached to the event.

In the example configuration, all the headers of the request would be attached to the event.

If we are only interested in the host for instance, we could have:

```json
      "headersPattern": "Host"
```

## Input

This enrichment uses the HTTP headers.

## Output

For **each** header matching the regex defined in `headersPattern`, a context with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/org.ietf/http_header/jsonschema/1-0-0) is added to the enriched event. Each context contains only one name/value pair.

---

# IAB bot detection enrichment
> Identify bot and spider traffic using the IAB/ABC International Spiders and Bots List based on IP and user agent.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/iab-enrichment/

The IAB Spiders and Robots enrichment uses the [IAB/ABC International Spiders and Bots List](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/) to determine whether an event was produced by a user or a robot/spider based on its IP address and user agent.

Spiders and bots are sometimes considered a necessary evil of the web. We want search engine crawlers to find our site, but we also don't want a lot of non-human traffic clouding our reporting.

The Interactive Advertising Bureau (IAB) is an advertising business organization that develops industry standards, conducts research, and provides legal support for the online advertising industry.

Their internationally recognized list of spiders and bots is regularly maintained to try and identify the IP addresses of known bots and spiders.

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/iab_spiders_and_robots_enrichment/jsonschema/1-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/iab_spiders_and_robots_enrichment.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

There are three fields that can be added to the `parameters` section of the enrichment configuration JSON:

- `ipFile`
- `excludeUseragentFile`
- `includeUseragentFile`

They correspond to one of the IAB/ABC database files, and need to have two inner fields:

- The `database` field containing the name of the database file.
- The `uri` field containing the URI of the bucket in which the database file is found. This field supports `http`, `https`, `gs` and `s3` schemes.

The table below describes the three types of database fields:

| **Field name**         | **Database description**                                         | **Database filename**           |
| ---------------------- | ---------------------------------------------------------------- | ------------------------------- |
| `ipFile`               | Denylist of IP addresses considered to be robots or spiders      | `"ip_exclude_current_cidr.txt"` |
| `excludeUseragentFile` | Denylist of useragent strings considered to be robots or spiders | `"exclude_current.txt"`         |
| `includeUseragentFile` | Allowlist of useragent strings considered to be browsers         | `"include_current.txt"`         |

All three of these fields **must** be added to the enrichment JSON, as the IAB lookup process uses all three databases in order to detect robots and spiders. Note that the database files are commercial and proprietary and should not be stored publicly – for instance, on unprotected HTTPS or in a public S3 bucket.

### Custom user agent lists

> **Note:** This feature is available since version 6.8.0 of Enrich.

In addition to the IAB database files, you can provide custom lists of user agent strings to supplement the detection. Two optional fields can be added to the `parameters` section:

| **Field name**      | **Description**                                                                                                      |
| ------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `excludeUseragents` | A list of user agent strings to be treated as robots or spiders (in addition to the `excludeUseragentFile` database) |
| `includeUseragents` | A list of user agent strings to be treated as browsers (in addition to the `includeUseragentFile` database)          |

Both fields accept a JSON array of strings. They are optional and default to empty lists if omitted.

A user agent matching `excludeUseragents` produces the following output values:

| Field           | Value               |
| --------------- | ------------------- |
| `spiderOrRobot` | `true`              |
| `category`      | `SPIDER_OR_ROBOT`   |
| `reason`        | `FAILED_UA_EXCLUDE` |
| `primaryImpact` | `UNKNOWN`           |

A user agent matching `includeUseragents` produces the following output values:

| Field           | Value        |
| --------------- | ------------ |
| `spiderOrRobot` | `false`      |
| `category`      | `BROWSER`    |
| `reason`        | `PASSED_ALL` |
| `primaryImpact` | `NONE`       |

Example:

```json
"excludeUseragents": ["my-custom-bot/1.0", "internal-crawler"],
"includeUseragents": ["my-legitimate-app/2.0"]
```

This is useful when you need to flag or allowlist specific user agents without modifying the IAB database files themselves.

> **Info:** These lists will take precedence over the IAB files. For example, a user agent added to `includeUseragents` will not be considered a robot even if it’s also in the `excludeUseragentFile` databaes.

## Input

This enrichment uses the following fields of a Snowplow event:

- `useragent` to determine an event's user agent, which will be validated against the databases described in `excludeUseragentFile` and `includeUseragentFile`.
- `user_ipaddress` to determine an event's IP address, which will be validated against the database described in `ipFile`.
- `derived_tstamp` to determine an event's datetime. Some entries in the Spiders and Robots List can be considered "stale", and will be given a `category` of `INACTIVE_SPIDER_OR_ROBOT` rather than `ACTIVE_SPIDER_OR_ROBOT` based on their age.

## Output

This enrichment adds a new context to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0).

Example:

```json
{
    "schema": "iglu:com.iab.snowplow/spiders_and_robots/jsonschema/1-0-0",
    "data": {
        "spiderOrRobot": false,
        "category": "BROWSER",
        "reason": "PASSED_ALL",
        "primaryImpact": "NONE"
    }
}
```

---

# Available enrichments
> Browse the complete catalog of Snowplow enrichments that add entities and enhance event data during processing.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/

Snowplow offers a large number of enrichments that can be used to enhance your event data. Most enrichments attach data to the event as [entities](/docs/fundamentals/entities/), although some enrichments fill or modify event fields directly.

The order of enrichments is hard-coded and cannot be changed, the below table lists available enrichments in order they are executed.

> **Note:** It’s not possible to configure more than one instance of the same enrichment except JavaScript enrichment starting with Enrich 4.1.0. It is possible to have multiple JavaScript enrichments starting with Enrich 4.1.0. For all other enrichments, you can still only configure one of each including SQL Query and API Request enrichments.

| Enrichment                                                                                                  | Description                                                                                                                                                                                                             |
| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/)                                     | Detect robots/spiders via the IAB/ABC International Spiders and Bots List.                                                                                                                                              |
| User Agent utils                                                                                            | This enrichment is deprecated. Please consider switching to [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/).                                                                                |
| [UA Parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)                         | Parse the user agent string and attach an entity with detailed user agent information. _(YAUAA enrichment is preferred.)_                                                                                               |
| [Currency Conversion](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/)     | Convert the values of all transactions to a specified base currency using Open Exchange Rates API. _(Please note the [limitations](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/).)_ |
| [Referrer Parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)             | Extract attribution data from referrer URLs.                                                                                                                                                                            |
| [Campaign Attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)   | Choose how query string parameters map to marketing campaign fields, e.g. `mkt_medium`. _If you do not enable the campaign attribution enrichment, those fields will not be populated._                                 |
| [Cross Navigation](/docs/pipeline/enrichments/available-enrichments/cross-navigation-enrichment/)           | Parse the extended cross navigation format in `_sp` query string parameter and attach the `cross_navigation` entity to an event. Available since Enrich version 4.1.0.                                                  |
| [Event Fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)         | Generate a fingerprint for the event by hashing a set of fields. Helpful for deduplicating events.                                                                                                                      |
| [Cookie Extractor](/docs/pipeline/enrichments/available-enrichments/cookie-extractor-enrichment/)           | Extract values of specific cookies into extra entities.                                                                                                                                                                 |
| [HTTP Header Extractor](/docs/pipeline/enrichments/available-enrichments/http-header-extractor-enrichment/) | Extract values of specific HTTP headers into extra entities.                                                                                                                                                            |
| [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)                                 | Parse the user agent string and attach an entity with detailed user agent information.                                                                                                                                  |
| [IP Lookup](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)                         | Look up useful data on the IP address in the MaxMind database.                                                                                                                                                          |
| [ASN Lookup](/docs/pipeline/enrichments/available-enrichments/asn-lookup-enrichment/)                       | Flag bot traffic by checking ASNs against known bad ASN lists.                                                                                                                                                          |
| [Bot Detection](/docs/pipeline/enrichments/available-enrichments/bot-detection-enrichment/)                 | Consolidate bot signals from YAUAA, IAB, and ASN lookup into a single entity.                                                                                                                                           |
| [JavaScript](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/)                | Write a JavaScript function which is executed for each event.                                                                                                                                                           |
| [SQL Query](/docs/pipeline/enrichments/available-enrichments/custom-sql-enrichment/)                        | Query an external relational database and attach the response as an extra entity.                                                                                                                                       |
| [API Request](/docs/pipeline/enrichments/available-enrichments/custom-api-request-enrichment/)              | Query an external HTTP(s) API and attach the response as an extra entity.                                                                                                                                               |
| [IP Anonymization](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/)           | Anonymize the IP addresses found in the `user_ipaddress` field by replacing some octets with `x`.                                                                                                                       |
| [PII Pseudonymization](/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/)   | Pseudoanonymize specific fields to comply with data privacy regulation.                                                                                                                                                 |

---

# IP anonymization enrichment
> Anonymize IP addresses by masking octets or segments to protect user privacy and comply with regulations.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/

This enrichment replaces the end of the user's IP address with "x"s, on a configurable length. For instance `13.54.45.87` could become `13.54.x.x`.

Both IPv4 and IPv6 are supported.

This enrichment runs after [IP lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/).

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/anon_ip/jsonschema/1-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/anon_ip.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

The number of octets (IPv4) to anonymize is specified with `anonOctets` and the number of segments (IPv6) to anonymize is specified with `anonSegments`.

For example, anonymizing one octet would change an IPv4 address of `255.255.255.255` to `255.255.255.x`, and anonymizing three octets would change it to `255.x.x.x`.

## Input

This enrichment uses the IP of the user, that can be found in `user_ipaddress` field of the atomic event.

## Output

The anonymized value of the IP address is updated in-place in `user_ipaddress` field, before ever being stored.

---

# IP lookup enrichment
> Resolve IP addresses to geographic locations and ISP information using MaxMind databases.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/

This enrichment uses [MaxMind](https://www.maxmind.com/en/geoip2-databases) databases to look up useful data based on the IP address collected by your Snowplow tracker(s).

When a user browses your site or app their IP address is collected. MaxMind maintains databases of additional points of information like geographic location, second level domain names (acme.com), Internet Service Provider, organization name and several other data points publicly associated with a given IP address.

The IP lookup enrichment uses MaxMind databases in order to take the IP address collected and add additional data points to every event generated by the user with a given IP address.

Some of the databases MaxMind maintains require a commercial subscription with MaxMind.

## Setting up this Enrichment

### 1. Decide which databases you’d like to use and download them

MaxMind offers a free tier and a paid tier of databases, which can be used with Snowplow.

From the free tier you can provide two databases to Snowplow:

- [GeoLite2 City Database](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data/), which contains geographic information (e.g. country) by IP address
- [GeoLite2 ASN Database](https://dev.maxmind.com/geoip/docs/databases/asn/) (since enrich version 6.7.0), which contains autonomous system numbers by IP address

From the paid tier you can provide four databases to Snowplow:

- [GeoIP2 City](https://www.maxmind.com/en/geoip2-city?rld=snowplow), which also contains geographic information, but with more precision and coverage than the GeoLite2 City Database
- [GeoIP2 ISP](https://www.maxmind.com/en/geoip2-isp-database?rld=snowplow), which contains information about the ISP serving that IP, and a more complete ASN mapping compared to the GeoLite2 ASN Database
- [GeoIP2 Domain](https://www.maxmind.com/en/geoip2-domain-name-database?rld=snowplow), which contains information about the domain at that IP address
- [GeoIP2 Connection Type](https://www.maxmind.com/en/geoip2-connection-type-database?rld=snowplow), which contains information about the connection type at that IP address.

You need to decide which of the different Maxmind databases listed above you wish to enrich your data with, download the .mmdb files and then setup the enrichment configuration accordingly.

### 2. Upload the databases to a location on your cloud

Once downloaded, take the .mmdb file(s) and upload them to a location on your cloud:

- Amazon S3 (if running Snowplow on AWS) e.g. s3://my-private-bucket/third-party/maxmind
- Google Cloud Storage (if running Snowplow on GCS) e.g. gs\://my-private-bucket/third-party/maxmind

When the database(s) need updating in future you can simply download the latest version and overwrite this file in your storage.

MaxMind also offer a method to [download and update their databases programmatically](https://dev.maxmind.com/geoip/geoipupdate/).

### 3. Configure the enrichment for your pipeline

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).
>
> Note that to test this enrichment, you will need events with realistic IP addresses (not local ones like `192.168.0.42`).
>
> This is not an issue if running Micro through Console, but will be trickier if running locally.
>
> If you are using a web browser to test your site or app, you can spoof a specific IP address using a browser plugin that sets an `X-Forwarded-For` header. For example, here are plugins for [Chrome](https://chrome.google.com/webstore/detail/x-forwarded-for-header/hkghghbnihliadkabmlcmcgmffllglin/) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/x-forwarded-for-injector/). Install the plugin and set the IP address to your liking.
>
> Alternatively, you can [set up Micro to receive external IP addresses](/docs/testing/snowplow-micro/local/remote-usage/#exposing-micro-via-a-public-domain-name).

There are five possible fields you can add to the “parameters” section of the enrichment configuration JSON: “geo”, “isp”, “domain”, “connectionType”, and “asn”:

- The `database` field contains the name of the MaxMind database file.
- The `uri` field contains the URI of the bucket in which the database file is found. This can have either `http:` or `s3:` or `gs:` as the scheme and must not end with a trailing slash.

It is important to note that accepted database filenames are the strings which are allowed in the `database` subfield. If the file name you provide is not one of these, the enrichment JSON will fail validation.

| ENRICHMENT PARAMETER | VALID DATABASE NAMES                                  |
| -------------------- | ----------------------------------------------------- |
| `geo`                | "GeoLite2-City.mmdb" (free) "GeoIP2-City.mmdb" (paid) |
| `isp`                | "GeoIP2-ISP.mmdb"                                     |
| `domain`             | "GeoIP2-Domain.mmdb"                                  |
| `connectionType`     | "GeoIP2-Connection-Type.mmdb"                         |
| `asn`                | "GeoLite2-ASN.mmdb"                                   |

### Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/ip_lookups.json)

#### Example minimal configuration

##### On AWS

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1",
    "data": {
        "name": "ip_lookups",
        "vendor": "com.snowplowanalytics.snowplow",
        "enabled": true,
        "parameters": {
            "geo": {
                "database": "GeoLite2-City.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            }
        }
    }
}
```

##### On GCS

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1",
    "data": {
        "name": "ip_lookups",
        "vendor": "com.snowplowanalytics.snowplow",
        "enabled": true,
        "parameters": {
            "geo": {
                "database": "GeoLite2-City.mmdb",
                "uri": "gs://my-private-bucket/third-party/maxmind"
            }
        }
    }
}
```

In the configurations above, we are enabling this enrichment to take all IP addresses from each event and do a lookup against the GeoLite2-City.mmdb.

The parameters to set start with the type of MaxMind database we are accessing (in this case the “geo” type). Then we specify the name of the database file, and the URI it’s available at.

When configuring the enrichment you will replace the following string `my-private-bucket/third-party/maxmind` with the path to your hosted database.

#### Example full configuration

To extend this enrichment for the additional databases offered by Maxmind we would simply repeat the process for the other databases.

Here is an example configuration using all relevant databases on MaxMind's paid tier:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1",
    "data": {
        "name": "ip_lookups",
        "vendor": "com.snowplowanalytics.snowplow",
        "enabled": true,
        "parameters": {
            "geo": {
                "database": "GeoIP2-City.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            },
            "isp": {
                "database": "GeoIP2-ISP.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            },
            "domain": {
                "database": "GeoIP2-Domain.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            },
            "connectionType": {
                "database": "GeoIP2-Connection-Type.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            }
        }
    }
}
```

Here is an example configuration using all relevant databases on MaxMind's free tier:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/ip_lookups/jsonschema/2-0-1",
    "data": {
        "name": "ip_lookups",
        "vendor": "com.snowplowanalytics.snowplow",
        "enabled": true,
        "parameters": {
            "geo": {
                "database": "GeoLite2-City.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            },
            "asn": {
                "database": "GeoLite2-ASN.mmdb",
                "uri": "s3://my-private-bucket/third-party/maxmind"
            }
        }
    }
}
```

## Output

This enrichment populates atomic table fields prefixed with "geo\_" and "ip\_" [seen here](https://github.com/snowplow/iglu-central/blob/8ff48b2485b3c95447e38a9bb925ef3f5266112c/schemas/com.snowplowanalytics.snowplow/atomic/jsonschema/1-0-0#L82).

| COLUMN NAME       | SAMPLE DATA    | PURPOSE                                                      | SOURCE DATABASE  |
| ----------------- | -------------- | ------------------------------------------------------------ | ---------------- |
| `geo_country`     | GB             | Country of IP origin                                         | `geo`            |
| `geo_region`      | ENG            | Region of IP origin                                          | `geo`            |
| `geo_city`        | London         | City of IP origin                                            | `geo`            |
| `geo_zipcode`     | EC2A           | Zip (postal) code of IP origin                               | `geo`            |
| `geo_latitude`    | 51.5237        | An approximate latitude (coordinates)                        | `geo`            |
| `geo_longitude`   | -0.089         | An approximate longitude (coordinates)                       | `geo`            |
| `geo_region_name` | England        | Region of IP origin                                          | `geo`            |
| `geo_timezone`    | Europe/London  | Timezone of IP origin                                        | `geo`            |
| `ip_isp`          | AT\&T Services | ISP name                                                     | `isp`            |
| `ip_organization` | AT\&T Services | Organization name for larger networks                        | `isp`            |
| `ip_domain`       | att.net        | Second level domain name                                     | `domain`         |
| `ip_netspeed`     | Cellular       | Indication of connection type (dial-up, cellular, cable/DSL) | `connectionType` |

Starting with Enrich 6.7.0, this enrichment supports ASN information, which is useful for detecting bot traffic coming from cloud computing providers.

If ASN data is available for a given IP address through one of the supplied databases (either ISP or the free ASN database), the enrichment adds a derived entity to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/asn/jsonschema/1-0-0).

Here is an example of a derived entity attached by this enrichment:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/asn/jsonschema/1-0-0",
    "data": {
        "number": 16509,
        "organization": "Amazon.com, Inc."
    }
}
```

---

# PII pseudonymization enrichment
> Pseudonymize personally identifiable information in events using hashing for privacy compliance.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/pii-pseudonymization-enrichment/

PII (personally identifiable information) pseudonymization enrichment runs after all the other enrichments and pseudonymizes the fields that are configured as PIIs.

It enables the users of Snowplow to better protect the privacy rights of data subjects, therefore aiding in compliance for regulatory measures.

In Europe the obligations regarding Personal Data handling have been outlined on the [GDPR EU website](https://www.gdpreu.org/the-regulation/key-concepts/personal-data/).

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/pii_enrichment_config/jsonschema/2-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/pii_enrichment_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

Two types of fields can be configured to be hashed:

- `pojo`: field that is effectively a scalar field in the enriched event (full list of fields that can be pseudonymized [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/pii_enrichment_config/jsonschema/2-0-1#L43-L60))
- `json`: field contained inside a self-describing JSON (e.g. in `unstruct_event`)

With the configuration example, the fields `user_id` and `user_ipaddress` of the enriched event would be hashed, as well as the fields `email` and `ip_opt` of the unstructured event in case its schema matches _iglu:com.mailchimp/subscribe/jsonschema/1-\*-\*_.

At the moment only `"pseudonymize"` strategy is available and the available hashing algorithms can be found below:

- `_MD2_`: the 128-bit algorithm [MD2](https://en.wikipedia.org/wiki/MD2_\(cryptography\)#MD2_hashes) (not-recommended due to performance reasons see [RFC6149](https://tools.ietf.org/html/rfc6149))
- `_MD5_`: the 128-bit algorithm [MD5](https://en.wikipedia.org/wiki/MD5#MD5_hashes)
- `_SHA-1_:` the 160-bit algorithm [SHA-1](https://en.wikipedia.org/wiki/SHA-1#Example_hashes)
- `_SHA-256_`: 256-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm
- `_SHA-384_`: 384-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm
- `_SHA-512_`: 512-bit variant of the [SHA-2](https://en.wikipedia.org/wiki/SHA-2#Comparison_of_SHA_functions) algorithm

It's **important** to keep these things in mind when using this enrichment:

- Hashing a field can change its format (e.g. email) and its length, thus making a whole valid original event invalid if its schema is not compatible with the hashing.
- When updating the `salt` after it has already been used, same original values hashed with previous and new salt will have different hashes, thus making a join impossible and/or creating duplicate values.

### `anonymousOnly` mode

Enrich 5.3.0 introduced the `anonymousOnly` mode. When [anonymousOnly](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/pii_enrichment_config/jsonschema/2-0-1#L155) is set to true, PII fields are masked only in events tracked in anonymous mode (i.e. the `SP-Anonymous` header is present).

This is useful for compliance with regulation such as GDPR, where you would start with [anonymous tracking](/docs/sources/web-trackers/anonymous-tracking/) by default (all identifiers are masked) and switch to non-anonymous tracking when the user consents to data collection (all identifiers are kept).

By default, `anonymousOnly` is `false`, i.e. PII fields are always masked.

## Input

[These fields](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/pii_enrichment_config/jsonschema/2-0-1#L43-L60) of the enriched event and any string or array of strings field of a self-describing event or entity can be hashed.

## Output

The fields are updated in-place in the enriched event.

If `emitEvent` is set to _true_ in the configuration, for each enriched event, an unstructured event wrapping the list of updates that happened with the fields is also emitted to the configured PII stream. Its schema can be found [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/pii_transformation/jsonschema/1-0-0).

---

# Referrer parser enrichment
> Extract attribution data from referrer URLs to identify traffic sources, search terms, and marketing channels.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/

This enrichment uses [snowplow referer-parser](https://github.com/snowplow/referer-parser) library to extract attribution data from referer URLs.

Knowing which sites refer users to our website is very much a staple of analytics in order to help understand traffic patterns. This enrichment takes the value of the referring URL and matches it against the company/site it belongs to.

This is particularly useful when looking for specific traffic from search engine providers or social networks for instance. Rather than scouring a full referrer URL list this enrichment adds an additional field so that it's possible to look at reports that combine sub-domains from some of the bigger referrers.

## Configuration example

- [Enrichment schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/referer_parser/jsonschema/2-0-1)
- [Example schema](https://github.com/snowplow/enrich/blob/master/config/enrichments/referer_parser.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### Internal domains

Snowplow has several subdomains like _community.snowplow\.io_ and _docs.snowplow\.io_. As users move from these subdomains to our main _snowplow\.io_ domain, we would like to capture that traffic as being referred internally. Therefore we would set the configuration in the example schema as such:

```json
"internalDomains": [
    "community.snowplow.io",
    "docs.snowplow.io"
],
```

Enabling this enrichment with the above configuration would fill the `refr_medium` column in our data warehouse with _"Internal"_ (rather then _"Unknown"_) when the referring URL to a page matches the subdomains above.

> **Note:** The enrichment will also classify `refr_medium` as `Internal` when an event's `page_urlhost` matches it's `refr_urlhost`, regardless of the configured `internalDomains`. This behavior is not configurable, and may require handling in data models or a [JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/) to change.

### Custom referrer mappings

> **Note:** This feature is available since version 6.9.0 of Enrich.

By default, the enrichment classifies referrers using a [hosted database](https://github.com/snowplow/referer-parser) of known sources. You can add your own referrer-to-category mappings directly in the enrichment configuration using the optional `referrers` parameter. This is useful when you need to classify new traffic sources (such as internal tools, niche search engines, or AI chatbots) without waiting for changes to the upstream database.

Custom mappings take precedence over the default database. If a domain appears in both your custom mappings and the default database, the custom mapping is used.

The `referrers` parameter is a nested object structured as follows:

```json
"referrers": {
  "<medium>": {
    "<source name>": {
      "domains": ["<domain1>", "<domain2>"],
      "parameters": ["<param1>"]
    }
  }
}
```

| Field           | Description                                                                                                       |
| --------------- | ----------------------------------------------------------------------------------------------------------------- |
| `<medium>`      | The referrer category (e.g., `search`, `social`, `email`). This value populates `refr_medium`.                    |
| `<source name>` | A human-readable name for the source (e.g., `"Google"`, `"Internal Search"`). This value populates `refr_source`. |
| `domains`       | An array of hostnames to match against the referrer URL. At least one domain is required.                         |
| `parameters`    | An optional array of URL query parameter names to extract search terms from. Matched values populate `refr_term`. |

For example, to classify a custom search engine and a social network:

```json
"referrers": {
  "search": {
    "Corporate Search": {
      "domains": ["search.example.com"],
      "parameters": ["q", "query"]
    }
  },
  "social": {
    "Internal Forum": {
      "domains": ["forum.example.com"]
    }
  }
}
```

With this configuration, a referrer URL of `https://search.example.com/?q=snowplow` would produce the following:

| Field         | Value              |
| ------------- | ------------------ |
| `refr_medium` | `search`           |
| `refr_source` | `Corporate Search` |
| `refr_term`   | `snowplow`         |

> **Tip:** You can use custom referrer mappings to immediately test new categorizations in your pipeline. Once validated, consider contributing your mappings back to the [upstream referer-parser database](https://github.com/snowplow/referer-parser) via a pull request.

## Output

This enrichment populates the following fields of the atomic event :

| Field         | Purpose                                                              |
| ------------- | -------------------------------------------------------------------- |
| `refr_medium` | Type of referer. Examples : Search, Internal, Unknown, Social, Email |
| `refr_source` | Name of referer if recognised. Examples: Google, Facebook            |
| `refr_term`   | Keywords if source is a search engine                                |

With this information in the data warehouse it's possible to get such insights:

| refr\_medium | number of sessions |
| ------------ | ------------------ |
| Search       | 272,699            |
| Internal     | 142,555            |
| Unknown      | 127,335            |
| Social       | 14,525             |
| Email        | 5,345              |

---

# UA parser enrichment
> Parse user agent strings to extract browser, operating system, and device information.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/

This enrichment uses the [ua-parser library](https://github.com/ua-parser/uap-core/) to parse the user agent and provide information about the user's device.

Collecting and separating out all the valuable device information to be found in the user agent is a great way to get insight into the types of devices and operating systems and versions your users are using. This helps with deciding which to support as well as in decisions around functionality releases.

With the device family field which is part of the context you can also benefit from the lookup done with our database to sort the multitude of different devices into higher level device types like desktop, mobile, tablet, other, etc.

> **Note:** The [YAUAA Enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/) supersedes this enrichment and is preferable in most cases as it will generate a richer dataset.
>
> You can continue to use the UA Parser Enrichment. It is handy if you have memory constraints running the Enrich application, as it uses significantly less memory than the YAUAA enrichment. Additionally, the UA Parser enrichment does further parsing of the Operating System and Browser Version numbers.
>
> Both enrichments will produce an additional entity attached to an event, based on the information available in the `User-Agent` header of the request from a user’s browser.
>
> **UA Parser vs YAUAA Output**
>
> | UA Parser Field    | YAUAA Field                     | Example (Firefox 106 on Mac OS X) |
> | ------------------ | ------------------------------- | --------------------------------- |
> | device\_family     | deviceClass                     | Desktop                           |
> | ❌                  | deviceName                      | Apple Macintosh                   |
> | ❌                  | deviceBrand                     | Apple                             |
> | ❌                  | deviceCpu                       | Intel                             |
> | ❌                  | deviceCpuBits                   | 32                                |
> | ❌                  | deviceFirmwareVersion           |                                   |
> | ❌                  | deviceVersion                   |                                   |
> | ❌                  | operatingSystemClass            | Desktop                           |
> | os\_family         | operatingSystemName             | Mac OS X                          |
> | os\_version        | operatingSystemVersion          | 10.15                             |
> | os\_major          | operatingSystemVersionMajor     | 10                                |
> | os\_minor          | ❌                               | 15                                |
> | os\_patch          | ❌                               |                                   |
> | os\_patch\_minor   | operatingSystemVersionBuild     |                                   |
> | ❌                  | operatingSystemNameVersion      | Mac OS X 10.15                    |
> | ❌                  | operatingSystemNameVersionMajor | Mac OS X 10                       |
> | ❌                  | layoutEngineClass               | Browser                           |
> | ❌                  | layoutEngineName                | Gecko                             |
> | ❌                  | layoutEngineVersion             | Gecko 106.0                       |
> | ❌                  | layoutEngineVersionMajor        | Gecko 106                         |
> | ❌                  | layoutEngineNameVersion         | 106.0                             |
> | ❌                  | layoutEngineNameVersionMajor    | 106                               |
> | ❌                  | layoutEngineBuild               | 20100101                          |
> | ❌                  | agentClass                      | Browser                           |
> | useragent\_family  | agentName                       | Firefox                           |
> | useragent\_version | agentVersion                    | 106.0                             |
> | useragent\_major   | agentVersionMajor               | 106                               |
> | useragent\_minor   | ❌                               | 0                                 |
> | useragent\_patch   | ❌                               |                                   |
> | ❌                  | agentNameVersion                | Firefox 106.0                     |
> | ❌                  | agentNameVersionMajor           | Firefox 106                       |
> | ❌                  | agentBuild                      |                                   |
> | ❌                  | agentLanguage                   |                                   |
> | ❌                  | agentLanguageCode               |                                   |
> | ❌                  | agentInformationEmail           |                                   |
> | ❌                  | agentInformationUrl             |                                   |
> | ❌                  | agentSecurity                   |                                   |
> | ❌                  | agentUuid                       |                                   |
> | ❌                  | webviewAppName                  |                                   |
> | ❌                  | webviewAppVersion               |                                   |
> | ❌                  | webviewAppVersionMajor          |                                   |
> | ❌                  | webviewAppNameVersionMajor      |                                   |
> | ❌                  | facebookCarrier                 |                                   |
> | ❌                  | facebookDeviceClass             |                                   |
> | ❌                  | facebookDeviceName              |                                   |
> | ❌                  | facebookDeviceVersion           |                                   |
> | ❌                  | facebookFBOP                    |                                   |
> | ❌                  | facebookFBSS                    |                                   |
> | ❌                  | facebookOperatingSystemName     |                                   |
> | ❌                  | facebookOperatingSystemVersion  |                                   |
> | ❌                  | anonymized                      |                                   |
> | ❌                  | hackerAttackVector              |                                   |
> | ❌                  | hackerToolkit                   |                                   |
> | ❌                  | koboAffiliate                   |                                   |
> | ❌                  | koboPlatformId                  |                                   |
> | ❌                  | iECompatibilityVersion          |                                   |
> | ❌                  | iECompatibilityVersionMajor     |                                   |
> | ❌                  | iECompatibilityNameVersion      |                                   |
> | ❌                  | iECompatibilityNameVersionMajor |                                   |
> | ❌                  | carrier                         |                                   |
> | ❌                  | gSAInstallationID               |                                   |
> | ❌                  | networkType                     |                                   |

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ua_parser_config/jsonschema/1-0-1)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/ua_parser_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

## Input

This enrichment uses the field `useragent`.

## Output

This enrichment adds a new context to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0).

| Field               | Description                          |
| ------------------- | ------------------------------------ |
| `useragent_family`  | Useragent family (browser) name      |
| `useragent_major`   | Useragent major version              |
| `useragent_minor`   | Useragent minor version              |
| `useragent_patch`   | Useragent patch version              |
| `useragent_version` | Full version of the useragent        |
| `os_family`         | Operation system name                |
| `os_major`          | Operation system major version       |
| `os_minor`          | Operation system minor version       |
| `os_patch`          | Operation system patch version       |
| `os_patch_minor`    | Operation system patch minor version |
| `os_version`        | Operation system full version        |
| `device_family`     | Device type                          |

As an example, the `User-Agent` string

```text
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
```

would be parsed with the following result:

| PARAMETER         | VALUE     |
| ----------------- | --------- |
| useragent\_family | Chrome    |
| useragent\_major  | 48        |
| useragent\_minor  | -         |
| useragent\_patch  | 2564      |
| os\_family        | Windows 7 |
| os\_major         | -         |
| os\_minor         | -         |
| os\_patch\_minor  | -         |
| os\_version       | Windows 7 |
| device\_family    | Computer  |

_\*empty values denoted by “-“_

---

# YAUAA enrichment
> Parse user agent strings with advanced detection using Yet Another UserAgent Analyzer for detailed device insights.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/

YAUAA (Yet Another User Agent Analyzer) enrichment is a powerful user agent parser and analyzer.

It uses [YAUAA API](https://yauaa.basjes.nl/) to parse and analyze all user agent information of an HTTP request and extract as many relevant information as possible about the user's device and browser, like for instance the device class (Phone, Tablet, etc.).

> **Warning:** YAUAA parsing relies entirely on in-memory _HashMaps_ and requires roughly 400 MB of RAM (see [here](https://yauaa.basjes.nl/README-MemoryUsage.html)). Additional memory is also needed if caching is enabled (by default).

There is no interaction with an external system.

## Configure your website to send client hints

The current trend in web browsers is to reduce the amount of information sent to servers in the `User-Agent` HTTP header, to help with user privacy. [Client Hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Client_hints) are a way for sites to opt in to sending the extra information which previously would have been part of the user agent header.

The YAUAA enrichment does not require you to make any changes to your website, if you are happy with the level of detail you get from the reduced user agent. However, it is possible to improve the accuracy of the user agent data produced by YAUAA, if you are able to configure your website to send [high entropy client hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Client_hints) to the collector by following these instructions.

There are two ways for your website to opt-in to sending client hints to the collector. In the first method, you must configure your webserver to set both a [`Accept-CH` HTTP header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-CH) and a [`Permissions-Policy` HTTP header](https://www.w3.org/TR/permissions-policy-1/) when serving the your site's main HTML pages:

```text
Accept-CH: sec-ch-ua, sec-ch-ua-full-version-list, sec-ch-ua-full-version, sec-ch-ua-mobile, sec-ch-ua-platform, sec-ch-ua-platform-version, sec-ch-ua-arch, sec-ch-ua-bitness, sec-ch-ua-model, sec-ch-ua-wow64
Permissions-Policy: ch-ua=("https://<YOUR COLLECTOR>"), ch-ua-full-version-list=("https://<YOUR COLLECTOR>"), ch-ua-full-version=("https://<YOUR COLLECTOR>"), ch-ua-mobile=("https://<YOUR COLLECTOR>"), ch-ua-platform=("https://<YOUR COLLECTOR>"), ch-ua-platform-version=("https://<YOUR COLLECTOR>"), ch-ua-arch=("https://<YOUR COLLECTOR>"), ch-ua-bitness=("https://<YOUR COLLECTOR>"), ch-ua-model=("https://<YOUR COLLECTOR>"), ch-ua-wow64=("https://<YOUR COLLECTOR>"),
```

Alternatively, in the second method, you can put a `meta` tag in the header secion of your site's HTML:

```text
<meta http-equiv="delegate-ch" content="sec-ch-ua https://<YOUR COLLECTOR>; sec-ch-ua-full-version-list https://<YOUR COLLECTOR>; sec-ch-ua-full-version https://<YOUR COLLECTOR>; sec-ch-ua-mobile https://<YOUR COLLECTOR>; sec-ch-ua-platform https://<YOUR COLLECTOR>; sec-ch-ua-platform-version https://<YOUR COLLECTOR>; sec-ch-ua-arch https://<YOUR COLLECTOR>; sec-ch-ua-bitness https://<YOUR COLLECTOR>; sec-ch-ua-model https://<YOUR COLLECTOR>; sec-ch-ua-wow64 https://<YOUR COLLECTOR>;">
```

## Configuration

- [Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.enrichments/yauaa_enrichment_config/jsonschema/1-0-0)
- [Example](https://github.com/snowplow/enrich/blob/master/config/enrichments/yauaa_enrichment_config.json)

> **Tip:** Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

Only one parameter can be set in the configuration : `cacheSize`. This field determines the number of already parsed user agents that are kept in memory for faster processing. If set to 0, caching is disabled. If not set, a default size is used for the cache (10000).

## Input

This enrichment uses the following inputs:

- The `useragent` field from the Snowplow event. Typically this field is taken from the `User-Agent` HTTP header. But Snowplow trackers can also override the user agent by setting the `ua` field [in the tracker payload](/docs/events/).
- Client hint HTTP headers. These are [a set of standard headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Client_hints) such as `Sec-CH-UA` which provide extra detail about the user agent.

## Output

This enrichment adds a new derived context to the enriched event with [this schema](https://github.com/snowplow/iglu-central/blob/master/schemas/nl.basjes/yauaa_context/jsonschema/1-0-1) (since enrich 1.4.0).

If a field can't be figured out by the algorithm, it won't be in the output. But some fields can have value _UNKNOWN_.

The only field that will always be present is `deviceClass`.

Here is an example of a derived context attached by this enrichment for a page visited with a Samsung Galaxy S9:

```json
{
  "schema":"iglu:nl.basjes/yauaa_context/jsonschema/1-0-1",
    "data": {
        "deviceClass":"Phone",
        "deviceName":"Samsung SM-G960F",
        "deviceBrand":"Samsung",
        "operatingSystemClass":"Mobile",
        "operatingSystemName":"Android",
        "operatingSystemVersion":"8.0.0",
        "operatingSystemNameVersion":"Android 8.0.0",
        "operatingSystemVersionBuild":"R16NW",
        "layoutEngineClass":"Browser",
        "layoutEngineName":"Blink",
        "layoutEngineVersion":"62.0",
        "layoutEngineVersionMajor":"62",
        "layoutEngineNameVersion":"Blink 62.0",
        "layoutEngineNameVersionMajor":"Blink 62",
        "agentClass":"Browser",
        "agentName":"Chrome",
        "agentVersion":"62.0.3202.84",
        "agentVersionMajor":"62",
        "agentNameVersion":"Chrome 62.0.3202.84",
        "agentNameVersionMajor":"Chrome 62"
   }
}
```

The full output possiblities generated by the YAUAA algorithm can be found [here](https://yauaa.basjes.nl/expect/fieldvalues/).

---

# Filter bot and spam events from pipeline
> Configure filters to automatically discard bot traffic and spam events from your Snowplow pipeline.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/filtering-bot-events/

Sometimes you don’t want the event to appear in your data warehouse or lake, e.g. because you suspect it comes from a bot and not a real user.

Starting with Enrich 5.3.0, it is possible to drop an event by calling `event.drop()` in JavaScript enrichment code:

```js
const botPattern = /.*Googlebot.*/;

function process(event) {
  const useragent = event.getUseragent();

  if (useragent !== null && botPattern.test(useragent)) {
    event.drop();
  }
}
```

This mechanism can be used to drop not only good events, but also invalid events. The dropped events will not be sent to any stream or destination, thus lowering the infrastructure costs.

> **Warning:** There is no way to recover dropped events therefore use it with caution.

Another way to discard events is throwing an exception in your JavaScript code, which will send the event to [failed events](/docs/fundamentals/failed-events/):

```js
const botPattern = /.*Googlebot.*/;

function process(event) {
  const useragent = event.getUseragent();

  if (useragent !== null && botPattern.test(useragent)) {
    throw "Filtered event produced by Googlebot";
  }
}
```

> **Warning:** This will create an “enrichment failure” failed event, which may be tricky to distinguish from genuine failures in your enrichment code, e.g. due to a mistake.

---

# Introduction to enrichments
> Configure enrichments to add extra properties and dimensions to your Snowplow events during the enrichment process.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/

During Enrichment your events have extra properties and values attached to them, also known as dimension widening.

Snowplow enrichments can be categorized into three brackets:

- Hardcoded enrichments loading `atomic.events` (legacy)
- Configurable enrichments loading `atomic.events` (legacy)
- Configurable enrichments adding new contexts to the `derived_contexts` JSON array

_Legacy enrichments_ are those which populate `atomic.events` table as opposed to dedicated enrichment tables. The hardcoded legacy enrichments normally take place as part of common enrichment process and they precede configurable enrichments. During the common enrichment process the data received from collector(s) is mapped according to our [Canonical Event Model](/docs/fundamentals/canonical-event/).

_Configurable enrichments_ often depend on the data produced by the common enrichment process.

### Hardcoded enrichments

The following fields are populated depending on whether the tracker provided the corresponding value or not.

| Raw Parameter | Enriched Parameter | Purpose                                                                                                                                                 |
| ------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eid`         | `event_id`         | The unique event identifier (UUID). Assigned during enrichment if not provided with `eid`                                                               |
| `cv`          | `v_collector`      | Collector type/version                                                                                                                                  |
| `tnuid`       | `network_userid`   | User ID set by Snowplow using 3rd party cookie. Overwriten with tracker-set `tnuid`.                                                                    |
| `ip`          | `user_ipaddress`   | Snowplow collectors log IP address as standard. However, you can override the value derived from the collector by populating this value in the tracker. |
| `ua`          | `useragent`        | Raw useragent (browser string). Could be overwritten with `ua`.                                                                                         |

The following fields are populated depending on the collector and ETL (Extract, Transform, Load) utilized in the pipeline.

| Added Parameter    | Purpose                                            |
| ------------------ | -------------------------------------------------- |
| `v_etl`            | Host ETL version                                   |
| `etl_tstamp`       | Timestamp event began ETL                          |
| `collector_tstamp` | Time stamp for the event recorded by the collector |

The raw parameter `res` (if present) representing the screen/monitor resolution and coming in as a combination of width and height (ex. `1280x1024`) is broken up into separate entities.

| Added Parameter     | Purpose                 |
| ------------------- | ----------------------- |
| `dvce_screenwidth`  | Screen / monitor width  |
| `dvce_screenheight` | Screen / monitor height |

The `url` parameter provides the value for `page_url` in `atomic.events`, which represents the current page's URL. The following parts are extracted and populate separate fields as outlined below.

| Added Parameter    | Purpose                                                                            |
| ------------------ | ---------------------------------------------------------------------------------- |
| `page_urlscheme`   | Scheme (protocol), ex. "http"                                                      |
| `page_urlhost`     | Host (domain), ex. "[www.snowplowanalytics.com](http://www.snowplowanalytics.com)" |
| `page_urlport`     | Port if specified, 80 if not                                                       |
| `page_urlpath`     | Path to page, ex. "/product/index.html"                                            |
| `page_urlquery`    | Querystring, ex. "id=GTM-DLRG"                                                     |
| `page_urlfragment` | Fragment (anchor), ex. "4-conclusion"                                              |

Similarly, `page_referrer` gets the value from `refr`, which represents the referer’s URL, and the following parts are extracted and populate separate fields as shown below.

| Added Parameter    | Purpose                      |
| ------------------ | ---------------------------- |
| `refr_urlscheme`   | Scheme (protocol)            |
| `refr_urlhost`     | Host (domain)                |
| `refr_urlport`     | Port if specified, 80 if not |
| `refr_urlpath`     | Path to page                 |
| `refr_urlquery`    | Querystring                  |
| `refr_urlfragment` | Fragment (anchor)            |

Additionally the derived timestamp is calculated, `derived_tstamp`. See [Timestamps](/docs/events/timestamps/) for more details.

Finally, contexts, unstructured events and the relevant configurable enrichments (if enabled) are validated against their corresponding JSON schemas and the array of the derived contexts is assembled.

### Configurable enrichment

All configurable enrichments are listed on the [Available Enrichments](/docs/pipeline/enrichments/available-enrichments/) page.

The following configurable enrichments write data into `atomic.events` table (legacy enrichments):

- [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/)
- [IP lookups enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/)
- [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)
- [Currency conversion enrichment](/docs/pipeline/enrichments/available-enrichments/currency-conversion-enrichment/)
- [referer-parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)
- [Event fingerprint enrichment](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)

All other configurable enrichments create a separate context and thus are loaded into their own dedicated tables.

---

# Manage enrichments in Console
> Enable, configure, and deploy enrichments for your pipelines using Snowplow Console UI.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/managing-enrichments/

Snowplow [Console](https://console.snowplowanalytics.com) enables you to manage enrichments that run on each of your pipelines and development environments.

## Recommended workflow

You can enable and edit enrichments on any pipeline, but we recommend the following workflow to test the configuration before deploying it:

- Navigate to a [development environment](/docs/testing/snowplow-micro/console/) (based on Snowplow Micro)
- Enable and configure the enrichment as required
- Send some events to the environment's endpoint and validate the fields appear as expected in the event payload
- Deploy the configuration to your pipeline(s)

Alternatively, you can use [Snowplow Micro](/docs/testing/snowplow-micro/local/) to test your changes locally.

> **Warning:** Invalid enrichment changes can halt data processing in the pipeline. For example, an enrichment might be pointing to a non-existent database file, or the JavaScript code in the JavaScript enrichment might have typos. This is why we recommend testing in a low-risk environment first.

## Configure an enrichment

Within [Console](https://console.snowplowanalytics.com), use the **Pipelines** section to navigate to the pipeline or development environment where you would like to change the enrichment. Open the **Enrichments** tab. You will see a listing of all enrichments and their current status (enabled or disabled).

Click on the enrichment you want to change. You will see the current configuration if the enrichment is enabled, and the default one if it's not. For each enrichment, the reference on the configuration format is linked in the top navigation bar.

- To enable or edit the enrichment, click on **Edit configuration**. Adjust the configuration and click **Publish** to deploy the changes. This will enable the enrichment if it was disabled.
- To disable the enrichment, click on the **⋮** menu and select **Disable**.

## Deploy an enrichment from a development environment to a pipeline

If you followed the recommended workflow to test an enrichment in a development environment, you will need to copy the same configuration into a pipeline.

To do that, navigate to the development environment under **Pipelines**, select the **Enrichments** tab and click on the enrichment in question. Then click **Deploy**, select the desired pipeline and click **Deploy configuration**.

---

# Manage enrichments in Snowplow Self-Hosted
> Enable and configure enrichments in Snowplow Self-Hosted deployments using Terraform on AWS, GCP, or Azure.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/managing-enrichments/terraform/

If you have installed Snowplow via [Quick Start](/docs/get-started/self-hosted/), you will have the following enrichments enabled by default:

- [UA parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/)
- [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/)
- [Campaign Attribution](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/)
- [Event fingerprint](/docs/pipeline/enrichments/available-enrichments/event-fingerprint-enrichment/)
- [Referer parser](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/)

To enable an extra enrichment, such as the [IP anonymisation enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/):

- Follow the instructions for [AWS](https://github.com/snowplow-devops/terraform-aws-enrich-kinesis-ec2#inserting-custom-enrichments), [GCP](https://github.com/snowplow-devops/terraform-google-enrich-pubsub-ce#inserting-custom-enrichments) or [Azure](https://github.com/snowplow-devops/terraform-azurerm-enrich-event-hub-vmss#inserting-custom-enrichments), updating the `anonOctets` and `anonSegments` according to the number of octets/ segments that you would like to be anonymised
- Run `terraform apply`
- Now when you query your events you should find that the `user_ipaddress` has been anonymised

> **Note:** The IAB and IP Lookups enrichments require a 3rd party database to function.

To disable any enrichment, you can follow the instructions for [AWS](https://github.com/snowplow-devops/terraform-aws-enrich-kinesis-ec2#disabling-default-enrichments), [GCP](https://github.com/snowplow-devops/terraform-google-enrich-pubsub-ce#disabling-default-enrichments) or [Azure](https://github.com/snowplow-devops/terraform-azurerm-enrich-event-hub-vmss#disabling-default-enrichments).

If you are using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing, you can configure enrichments following the [usage guide](/docs/testing/snowplow-micro/local/enrichments/).

---

# Configure your Snowplow pipeline
> Configure collector settings, manage enrichments, and secure your Snowplow data pipeline infrastructure.
> Source: https://docs.snowplow.io/docs/pipeline/

![architecture diagram](/assets/images/architecture-16e4cd694721410bb8a1ff943a4c5255.png)

This section explains how to validate and update your [collector configuration](/docs/pipeline/collector/), manage [enrichments](/docs/pipeline/enrichments/), and has information on [security](/docs/pipeline/security/).

---

# Configure AWS KMS customer managed keys
> Set up AWS KMS custom encryption keys for enhanced pipeline security with full control over data encryption.
> Source: https://docs.snowplow.io/docs/pipeline/security/customer-managed-keys/

This guide provides step-by-step instructions for setting up AWS KMS (Key Management Service) custom keys for your Snowplow pipeline. Custom KMS keys provide enhanced security by allowing you to maintain full control over encryption keys used by your data infrastructure.

> **Note:** This is a bolt-on security feature available for enterprise customers using CDI Private Managed Cloud or CDI Cloud deployments.

AWS Custom KMS Keys ensure that all data stored at-rest in AWS services (Kinesis, SQS, and S3) are encrypted with a key that you own and control. This provides:

- Enhanced security: full control over encryption keys
- Compliance: meet regulatory requirements for data encryption
- Audit control: ability to revoke access to encrypted data
- Key management: centralized control over data encryption

## Prerequisites

To use custom KMS keys, you will need:

- AWS account with appropriate permissions to create and manage KMS keys
- Access to your AWS Console in the same region as your Snowplow pipeline
- Snowplow pipeline account ID (provided by your Customer Success Manager)
- Your security team approval for cross-account key sharing

This feature has some limitations:

- EMR cluster: if you're using the RDB Transformer component (only required for batch loading to Redshift and Databricks), it will need additional permissions adding (`kms:Decrypt`, `kms:GenerateDataKey*`) to enable access to customer S3 buckets with customer-managed KMS keys.
- Single key recommendation: while it's technically possible to use different keys for different services, we recommend using a single key for all pipeline resources for simplicity.
- Regional restrictions: keys must be created in the same region as your pipeline infrastructure.

## Step 1: Create your custom KMS key

Go to the AWS Console in the same region as your Snowplow pipeline and navigate to **KMS** → **Customer managed keys**. The URL will look similar to `https://us-east-1.console.aws.amazon.com/kms/home?region=us-east-1#/kms/keys`.

Click **Create key** and configure with the following settings:

| Setting             | Value                                            |
| ------------------- | ------------------------------------------------ |
| Key type            | Symmetric                                        |
| Key usage           | Encrypt and decrypt                              |
| Key material origin | KMS                                              |
| Regionality         | Single region                                    |
| Alias               | `snowplow-pipeline-key` (or your preferred name) |

When setting key administrators:

- Key administrators: select the roles/users from your security team that need administrative access to the key
- Other AWS Accounts: add your Snowplow pipeline account ID here

> **Note:** The "Other AWS Accounts" step is essential. Without adding the Snowplow pipeline account ID, the encryption will not work.

## Step 2: Key policy configuration

The key creation process will generate a policy similar to the one below. We strongly recommend allowing root access for the Snowplow pipeline account:

**Details**

```json
{
  "Id": "key-consolepolicy-3",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow access for Key Administrators",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourSecurityRole",
          "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourAdminRole"
        ]
      },
      "Action": [
        "kms:Create*",
        "kms:Describe*",
        "kms:Enable*",
        "kms:List*",
        "kms:Put*",
        "kms:Update*",
        "kms:Revoke*",
        "kms:Disable*",
        "kms:Get*",
        "kms:Delete*",
        "kms:TagResource",
        "kms:UntagResource",
        "kms:ScheduleKeyDeletion",
        "kms:CancelKeyDeletion",
        "kms:RotateKeyOnDemand"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow use of the key",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:root"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow attachment of persistent resources",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:root"
      },
      "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}
```

Replace the placeholder values in this policy:

| Placeholder                    | Replace With                   | Example              |
| ------------------------------ | ------------------------------ | -------------------- |
| `YOUR_ACCOUNT_ID`              | Your AWS account ID            | `123456789012`       |
| `YourSecurityRole`             | Your security team's role name | `SecurityTeamRole`   |
| `YourAdminRole`                | Your administrative role name  | `AdminRole`          |
| `SNOWPLOW_PIPELINE_ACCOUNT_ID` | Snowplow pipeline account ID   | Provided by Snowplow |

If your security policies don't allow root access (`arn:aws:iam::<account_id>:root`), you can use these specific role patterns instead:

```json
{
  "Sid": "Allow use of the key - Specific Roles",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/SnowplowOperator",
      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/sp-*",
      "arn:aws:iam::SNOWPLOW_PIPELINE_ACCOUNT_ID:role/snowplow-*"
    ]
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey",
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*"
}
```

Using non-root policies requires coordination with Snowplow Support to ensure all necessary roles are included. We strongly recommend the root access approach for simplicity.

## Step 3: Share key information with Snowplow

After creating your key, collect the following information:

- Key ARN: found in the key details page
  - Format: `arn:aws:kms:REGION:YOUR_ACCOUNT_ID:key/KEY_ID`
- Key ID: the unique identifier for your key
- Region: the AWS region where the key was created

Share this information with your Snowplow Customer Success Manager or through a support ticket:

- Key ARN
- Confirmation that cross-account access is configured
- Any specific requirements or restrictions

## Step 4: Snowplow configuration

Once you've shared the key information, Snowplow will configure the following pipeline components to use your custom KMS key.

All **pipeline S3 buckets** will be configured with your custom KMS key:

- Batch archive bucket
- Batch output bucket (used in RDB Transformer/Loader process for Redshift and Databricks)
- Batch processing bucket (used in RDB Transformer/Loader process for Redshift and Databricks)
- S3 bad events bucket
- S3 enriched events bucket

Pipeline **Kinesis streams** will be encrypted with your key:

- Good events stream
- Bad events stream
- Enriched events stream
- Incomplete events stream

If surge protection is enabled, **SQS queues** will use your custom key:

- Good events queue
- Bad events queue

**Other pipeline components** like Snowbridge will be granted access to decrypt/encrypt data using your key.

## Step 5: Post-configuration

After Snowplow applies your custom KMS key:

- Immediate effect: encryption settings change immediately on resources
- Temporary disruption: services may briefly fail until proper access is configured
- Drift application: Snowplow will apply necessary IAM policy updates to resolve access issues

You can verify the configuration by:

- Checking S3 bucket encryption settings in your AWS Console
- Verifying Kinesis stream encryption status
- Confirming SQS queue encryption (if applicable)
- Monitoring pipeline health through Snowplow dashboards

## Security best practices

For recommended settings:

- Enable key rotation: turn on automatic key rotation for enhanced security
- Monitor usage: use AWS CloudTrail to track key usage
- Regular audits: periodically review key policies and access patterns
- Documentation: maintain internal documentation of key management procedures

For key management:

- Backup procedures: ensure key policies are backed up and version controlled
- Access reviews: regular review of who has administrative access to keys
- Incident response: plan for key compromise or rotation scenarios

## Troubleshooting

If you encounter access denied errors:

- Verify the key policy includes the correct Snowplow pipeline account ID
- Ensure the key is in the same region as your pipeline
- Check that **Other AWS Accounts** was properly configured during key creation

Pipeline failures after KMS implementation are expected temporarily while Snowplow applies the necessary IAM policy updates. Contact Support if issues persist beyond the expected configuration window.

To validate your key policy:

```bash
# Check current key policy
aws kms get-key-policy \
  --key-id YOUR_KEY_ID \
  --policy-name default
```

---

# Pipeline security
> Security features and best practices for protecting your Snowplow pipeline infrastructure and data.
> Source: https://docs.snowplow.io/docs/pipeline/security/

This section contains information relating to pipeline security.

## [📄️ Customer managed keys](/docs/pipeline/security/customer-managed-keys/)

[Set up AWS KMS custom encryption keys for enhanced pipeline security with full control over data encryption.](/docs/pipeline/security/customer-managed-keys/)

---

# Define attribute groups in Console
> Define attribute groups to calculate behavioral data from real-time event streams, batch warehouse tables, or external sources. Configure attributes, attribute keys, TTL lifetimes, and test definitions before publishing.
> Source: https://docs.snowplow.io/docs/signals/attributes/attribute-groups/

Define the behavior you want to capture in [attribute groups](/docs/signals/concepts/#attribute-groups). Choose whether to calculate attributes from your event stream or warehouse.

To create an attribute group, go to **Signals** > **Attribute groups** in Snowplow Console and follow the instructions.

![Create attribute group form with name, description, data source, and owner fields](/assets/images/attribute-group-create-05c9671a86bc1133c6625497bb434064.png)

The first step is to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer
- Which data source you want to use

## Data source

> **Note:** A warehouse connection is required to create `Batch` and `External Batch` attributes.

There are three [sources](/docs/signals/concepts/#data-sources) to choose from:

- **Stream**: real-time Snowplow event stream
- **Batch**: a new warehouse table created by Signals, calculated from your `atomic` events table
- **External batch**: pre-calculated values in a warehouse table that you can sync to the Profiles Store

Attribute groups are configured differently based on the data source.

### Stream

By default, Signals will calculate attributes from events in your real-time stream: the stream source. Check out the [quick start tutorial](/tutorials/signals-quickstart/start/) for a step-by-step guide.

You'll need to define the [attributes](/docs/signals/attributes/attributes/) you want to calculate from your event stream.

### Batch

Attribute groups with a batch source use dbt to calculate the defined attributes as a new table. Signals will sync the calculated attributes to the Profiles Store.

First, define the [attributes](/docs/signals/attributes/attributes/) you want to calculate from your `atomic` events table.

Once you've created and published the group, create and configure the dbt models. Follow the instructions shown, or check out the [batch engine tutorial](/tutorials/signals-batch-engine/start/) for a step-by-step guide.

![Batch configuration instructions showing dbt setup steps](/assets/images/attribute-group-batch-instructions-2f460bdece8ea63f01100c34e9a813e2.png)

### External batch

Attribute groups with an external batch source don't require attribute definition, as no calculation will be performed. This source type allows you to sync existing warehouse values with Signals so they're available in your Profiles Store.

Provide the warehouse and table details, and select which fields you want to send to Signals.

![External batch source configuration showing warehouse table and field mapping options](/assets/images/attribute-group-external-batch-fields-645921d8d99b1a8c4cd7aec3ac157b2a.png)

To minimize latency, select a timestamp field which Signals will use to determine which rows have changed since the last sync. The sync engine will only send the new rows to the Profiles Store.

## Attribute keys

All attribute groups need an [attribute key](/docs/signals/concepts/#attribute-keys).

Signals includes four built-in attribute keys, based on commonly used identifiers from the atomic [user-related fields](/docs/fundamentals/canonical-event/#user-fields) in all Snowplow events.

To create a custom attribute key, navigate to **Signals** > **Attribute keys** within Console. Click the **Create attribute key** button.

![Create attribute key form with name, description, and atomic property selection](/assets/images/attribute-key-create-7771bedf4e1b02a0915a80001741cef5.png)

You will need to provide:

- A unique name
- An optional description
- An optional email address for the primary owner or maintainer
- Which [atomic](/docs/fundamentals/canonical-event/#common-fields) property you want to calculate attributes against

To edit or delete a custom attribute key, go to the key details page and click the **Edit** button, or the `⋮` button followed by **Delete**.

## Attribute lifetimes

We recommend setting a Time to live (TTL) value for each attribute group. Some attributes will only be relevant for a certain amount of time, and eventually stop being updated. To avoid stale attributes staying in your Profiles Store forever, configure a TTL for the attribute group.

The suggested default is 7 days for stream attribute groups, and 365 days for batch attribute groups.

When none of the attributes for an attribute group have been updated for the defined lifespan, the attribute group expires. Any attribute values for this group will be deleted: fetching them will return `None` values.

If Signals then processes a new event that calculates the attribute again, or materializes the attribute from the warehouse again, the expiration timer is reset.

## Testing the attribute definitions

> **Note:** A warehouse connection is required to test attribute definitions.

After defining one or more [attributes](/docs/signals/attributes/attributes/) for groups with a stream or batch source, you can test out the configuration with the **Run preview** button.

This will output a table of attributes calculated from your `atomic` events table, using a random subset of events from the last hour.

## Publishing the attribute group

Once you're happy with your attribute group configuration, click **Create attribute group** to save it. It will be saved as a draft, and not yet available to Signals.

![Draft attribute group page showing Edit and Publish buttons](/assets/images/attribute-group-draft-c1d87d4475afae4157d9f077bd4b0d77.png)

Click the **Edit** button if you want to make changes to the attribute group.

To send the attribute group configuration to your Signals infrastructure, click the **Publish** button. This will allow Signals to start calculating attributes or syncing tables, and populating the Profiles Store.

![Published attribute group page showing active status and management options](/assets/images/attribute-group-published-2cf1af60f705a14cbe0ba0d61c5390f6.png)

> **Note:** If the attribute group has a batch source, Signals won't be able to do anything until you've completed the dbt configuration steps. Complete the dbt steps before publishing the group.

### Versioning

Attribute groups are versioned. This allows you to iterate on the definitions without breaking downstream processes. You'll select specific attribute group versions when you define [services](/docs/signals/attributes/services/).

All attribute groups start as `v1`. If you make changes to the definition, the version will be automatically incremented.

## Deleting an attribute group

To unpublish or delete an attribute group, click the `⋮` button on the group details page.

![Attribute group management menu showing Edit, Unpublish, and Delete options](/assets/images/attribute-group-edit-delete-cdff6c24c74fe31d67ad224c2c79f5cd.png)

Unpublishing is version specific. It will stop Signals from calculating attributes for this version of this group. Existing attribute values will remain in your Profiles Store, but they won't be updated. You can republish it later if needed.

Choose **Delete** to permanently delete all versions of the attribute group, along with attribute values in your Profiles Store for this group.

If the attribute group version is used by a [service](/docs/signals/concepts/#services), you'll need to update the service definition before unpublishing or deleting.

If the attribute group version is used by a published [intervention](/docs/signals/concepts/#interventions), deleting or unpublishing it will unpublish the intervention.

---

# Define attributes in Console
> Define individual attributes within attribute groups by selecting event schemas or event specifications, properties, and aggregation types. Use time periods and criteria filters to control when attributes update.
> Source: https://docs.snowplow.io/docs/signals/attributes/attributes/

[Attributes](/docs/signals/concepts/#attribute-groups) are defined as part of attribute groups. To create an attribute, you'll need to set:

- A name, ideally one that describes the attribute
- Which event schema or event specification to calculate it from
- What property in the schema to consider for the calculation
- What kind of aggregation you want to calculate over time, e.g. `mean` or `last`

![Attribute configuration interface showing name, event filter, and aggregation options](/assets/images/attribute-group-attributes-112be3abdd020aa9bf10977a7b68fb1c.png)

## Event selection

Use the event filter to choose which event type to calculate the attribute from.

![Event filter dropdown showing available event specifications, Snowplow, and custom event schemas](/assets/images/attribute-event-specs-92440853627cf27a7b4f3ee6295900e9.png)

Click the dropdown to see the available events, listed by name and vendor:

- **Event Specifications**: select any [event specification](/docs/event-studio/tracking-plans/event-specifications/) from an existing [tracking plan](/docs/event-studio/tracking-plans/).

- **Snowplow events**: select any built-in Snowplow or [Iglu Central](https://iglucentral.com) schema. For legacy reasons, to calculate an attribute from [structured](/docs/events/custom-events/#structured-events) events find `event (com.google.analytics.measurement-protocol)`.

- **Custom events**: select any schema or data structure that's available within your pipeline.

> **Note:** The search finds direct matches only, so use the exact name of the event, schema, or vendor.

Once you've selected an event and version, click **Confirm** to add the attribute to your attribute group.

## Aggregation options

Signals supports a number of different aggregation types.

| Aggregation           | Description                                                                                                                                          | Required property type in schema      |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| Counter               | Count events                                                                                                                                         | No property used for this aggregation |
| Sum                   | Sum of property values                                                                                                                               | Numeric                               |
| Min                   | Minimum property value                                                                                                                               | Numeric                               |
| Max                   | Maximum property value                                                                                                                               | Numeric                               |
| Mean                  | Average of property values                                                                                                                           | Numeric                               |
| First                 | First property value seen                                                                                                                            | String, Numeric, Boolean              |
| Last                  | Last property value seen                                                                                                                             | String, Numeric, Boolean              |
| Most Frequent         | Most frequent property value seen                                                                                                                    | String, Numeric, Boolean              |
| Least Frequent        | Least frequent property value seen                                                                                                                   | String, Numeric, Boolean              |
| Approx Count Distinct | Approximate distinct count as calculated by the [HyperLogLog algorithm](https://redis.io/docs/latest/develop/data-types/probabilistic/hyperloglogs/) | String, Numeric, Boolean              |
| Category Count        | Dictionary of unique values and their counts                                                                                                         | String, Numeric, Boolean              |
| Unique List           | List of unique property values                                                                                                                       | String, Numeric, Boolean              |

A property isn't used for `counter` aggregation. To only count events with a specific property value, use a criteria filter.

## Property selection

You can calculate attributes based on properties in any part of your events:

- [Atomic](/docs/fundamentals/canonical-event/) properties: these are available for all events
- Event schema properties: choose properties within your chosen event
- Entity properties: choose properties from schemas that you are tracking as entities with your chosen event

![Property selection interface with atomic, event schema, and entity property tabs](/assets/images/attribute-property-selector-d2cb6ca2f9acc7207cf1a5eaa5c35f9c.png)

Click **Confirm** to specify the property for this attribute.

## Time period

Add an optional time period to the attribute to aggregate it over a rolling window. Signals won't include events older than the specified time period in the calculation.

Find the time period option within **More options**. Click **Done** to save it.

![Time period configuration dialog for setting rolling window attributes](/assets/images/attribute-set-period-1245a531daa2acb8b7019decfda6c451.png)

## Filtering with criteria

Use criteria to filter the events used to calculate an attribute. They allow you to be specific about which subsets of events should trigger attribute updates. For example, instead of counting all page views in a user's session, you may wish to calculate only views for the homepage, or a login page.

Find the criteria option within **More options**.

Defining criteria has three steps:

1. Select which property to filter on, similarly to the property selection for the attribute
2. Choose which logical operator to use
3. Enter the value to filter on

If you enter multiple criteria, you will have the option to require `all` or `any` of them to be met for the attribute to update.

![Criteria filter configuration with property selection, operator, and value fields](/assets/images/attribute-criteria-8db2a31c1ee6c759463b7dc0176fb9fb.png)

Click **Done** to save the criteria when you're finished.

---

# Define and retrieve attributes in Signals
> Define attributes to calculate behavioral data from real-time streams or your warehouse, then retrieve calculated values using services or attribute groups.
> Source: https://docs.snowplow.io/docs/signals/attributes/

[Attributes](/docs/signals/concepts/#attribute-groups) are the behavioral facts you want Signals to calculate, such as a user's page view count or lifetime value. You define attributes within [attribute groups](/docs/signals/attributes/attribute-groups/), then retrieve calculated values through [services](/docs/signals/attributes/services/) or directly from individual groups.

## Define attributes

There are three methods for defining attributes in Signals:

- Snowplow Console UI
- [Signals Python SDK](/docs/signals/attributes/using-python-sdk/)
- [Signals API](/docs/signals/connection/#signals-api)

### Snowplow Console

To use the UI to manage Signals, log in to [Console](https://console.snowplowanalytics.com) and navigate to the **Signals** section.

Use the configuration interface to define [attribute groups](/docs/signals/attributes/attribute-groups/) and [services](/docs/signals/attributes/services/).

![Console Signals navigation options](/assets/images/console-navbar-11b65d884e2ac3450f8248e862b354c9.png)

## Retrieve attributes

Your calculated attributes are stored in the Profiles Store, and retrieved using [services](/docs/signals/concepts/#services).

To use attributes to take action in your application, you'll want to retrieve only the relevant values. This would usually be the attributes for the current user.

For example, use the current user's unique `domain_userid` identifier to retrieve attributes defined against the `domain_userid` attribute key.

You have three options for consuming attributes, depending on your use case or application:

- [Signals Node.js SDK](https://www.npmjs.com/package/@snowplow/signals-node) (TypeScript)
- [Signals Python SDK](https://pypi.org/project/snowplow-signals/)
- [Signals API](/docs/signals/connection/#signals-api)

Start by [connecting to Signals](/docs/signals/connection/).

### Using a service

The preferred way to retrieve attributes is by using a [service](/docs/signals/concepts/#services). This allows you to retrieve attributes in bulk, from multiple attribute groups.

**Python:**

Use `get_service_attributes()` to retrieve attributes from a service. Signals will return the attributes as a dictionary.

Here's an example:

```python
# The Signals connection object has been created as sp_signals

calculated_values = sp_signals.get_service_attributes(
    name="my_service",
    attribute_key="domain_userid",
    identifier="218e8926-3858-431d-b2ed-66da03a1cbe5",
)
```

The table below lists all available arguments for `get_service_attributes()`

| Argument        | Description                                  | Type     | Required? |
| --------------- | -------------------------------------------- | -------- | --------- |
| `name`          | The name of the service                      | `string` | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string` | ✅         |
| `identifier`    | The specific attribute key value             | `string` | ✅         |

**Node.js:**

Use `getServiceAttributes()` to retrieve attributes for a single identifier from a specific service. Signals will return the attributes as a JavaScript object.

Here's an example:

```typescript
// The Signals connection object has been created as signals

const calculatedValues = await signals.getServiceAttributes({
  name: "my_service",
  attribute_key: "domain_userid",
  identifier: "218e8926-3858-431d-b2ed-66da03a1cbe5",
});
```

The table below lists all available arguments for `getServiceAttributes()`

| Argument        | Description                                  | Type     | Required? |
| --------------- | -------------------------------------------- | -------- | --------- |
| `name`          | The name of the service                      | `string` | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string` | ✅         |
| `identifier`    | The specific attribute key value             | `string` | ✅         |

Use `getBatchServiceAttributes()` to retrieve attributes for multiple identifiers from a service in a single API call. This is more efficient than calling `getServiceAttributes()` multiple times.

```typescript
// Retrieve cart data for multiple users
const batchResults = await signals.getBatchServiceAttributes({
  name: "shopping_cart_service",
  attribute_key: "domain_userid",
  identifiers: [
    "218e8926-3858-431d-b2ed-66da03a1cbe5",
    "f47ac10b-58cc-4372-a567-0e02b2c3d479",
    "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
  ]
});
```

The table below lists all available arguments for `getBatchServiceAttributes()`

| Argument        | Description                                  | Type       | Required? |
| --------------- | -------------------------------------------- | ---------- | --------- |
| `name`          | The name of the service                      | `string`   | ✅         |
| `attribute_key` | The attribute key to retrieve attributes for | `string`   | ✅         |
| `identifiers`   | Array of attribute key values to look up     | `string[]` | ✅         |

***

### Retrieve individual attributes

You can also retrieve attributes directly from a specific [attribute group](/docs/signals/concepts/#attribute-groups). This is useful when:

- You want to retrieve only a small subset of attributes
- You haven't defined a service yet

**Python:**

Use `get_group_attribtues()` to retrieve specific attributes. Signals will return the attributes as a dictionary.

Here's an example:

```python
# The Signals connection object has been created as sp_signals

calculated_values = sp_signals.get_group_attribtues(
    name="my_attribute_group",
    version=1,
    attributes=["page_view_count"],
    attribute_key="domain_userid",
    identifier="218e8926-3858-431d-b2ed-66da03a1cbe5",
)
```

The table below lists all available arguments for `get_group_attribtues()`

| Argument        | Description                             | Type                         | Required? |
| --------------- | --------------------------------------- | ---------------------------- | --------- |
| `name`          | The name of the attribute group         | `string`                     | ✅         |
| `version`       | The attribute group version             | `int`                        | ✅         |
| `attributes`    | The names of the attributes to retrieve | `string` or list of `string` | ✅         |
| `attribute_key` | The attribute key name                  | `string`                     | ✅         |
| `identifier`    | The specific attribute key value        | `string`                     | ✅         |

**Node.js:**

Use `getGroupAttributes()` to retrieve specific attributes from an attribute group. Signals will return the attributes as a JavaScript object.

Here's an example:

```typescript
// The Signals connection object has been created as signals

const calculatedValues = await signals.getGroupAttributes({
  name: "my_attribute_group",
  version: 1,
  attributes: ["page_view_count"],
  attribute_key: "domain_userid",
  identifier: "218e8926-3858-431d-b2ed-66da03a1cbe5",
});
```

The table below lists all available arguments for `getGroupAttributes()`

| Argument        | Description                             | Type       | Required? |
| --------------- | --------------------------------------- | ---------- | --------- |
| `name`          | The name of the attribute group         | `string`   | ✅         |
| `version`       | The attribute group version             | `number`   | ✅         |
| `attributes`    | The names of the attributes to retrieve | `string[]` | ✅         |
| `attribute_key` | The attribute key name                  | `string`   | ✅         |
| `identifier`    | The specific attribute key value        | `string`   | ✅         |

***

---

# Define services in Console
> Services group multiple versioned attribute groups into stable consumption interfaces for applications. Create and manage services to control which attribute versions your applications use.
> Source: https://docs.snowplow.io/docs/signals/attributes/services/

[Services](/docs/signals/concepts/#services) group attribute groups together for serving to your applications.

To create a service, go to **Signals** > **Services** in Snowplow Console and follow the instructions.

![Create service form with name, description, owner, and attribute group selection](/assets/images/service-create-85585d669c129ebb7cb30d74209dfc12.png)

To configure a service, you'll need to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer
- Which attribute groups to include

## Versioning

When choosing which attribute groups to include, you'll select a specific version of each attribute group.

Services themselves are not versioned. You can update them to use different attribute groups, or different attribute group versions, at any time.

![Service configuration showing selected attribute groups with version numbers](/assets/images/service-groups-adfabeee05def063578d69fda35203b2.png)

## Managing services

Services are automatically published as soon as they're created. They're a wrapper for consuming calculated attributes, and don't cause any additional processing themselves.

To edit a service definition, go to the service details page and click the **Edit** button.

To delete a service, go to the service details page and click the `⋮` button, then choose **Delete**.

![Service management menu with Edit and Delete options](/assets/images/service-edit-delete-7e8bf6d7c7571a961344c7d77b47263b.png)

---

# Define attribute keys with the Signals Python SDK
> Create custom attribute keys based on atomic properties to specify the analytical context for attribute calculations. Use built-in keys like domain_userid or define your own.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/attribute-groups/attribute-keys/

An [attribute key](/docs/signals/concepts/#attribute-keys) is the identifier that attributes are calculated against.

Signals includes a number of out-of-the-box attribute keys based on commonly used identifiers from the out-of-the-box atomic [user-related fields](/docs/fundamentals/canonical-event/#user-fields) in all Snowplow events.

Import them into your notebook like this:

```python
from snowplow_signals import (
    domain_userid,
    domain_sessionid,
    user_id,
    network_userid,
)
```

## Custom attribute keys

You can also define custom attribute keys, which allows you to calculate attributes on any other Snowplow atomic property you want. Atomic properties are those that are defined in the [atomic fields](/docs/fundamentals/canonical-event/#common-fields) of the core Snowplow event, not properties tracked as part of an entity.

For example, an attribute key that groups by `app_id` can be defined as:

```python
from snowplow_signals import AttributeKey

app_id_attribute_key = AttributeKey(
    name="app_id_attribute_key",
    description="The id for the app",
    key="app_id",
)
```

## Options

The table below lists all available arguments for a custom attribute key:

| Argument      | Description                                                                              | Type        | Required? |
| ------------- | ---------------------------------------------------------------------------------------- | ----------- | --------- |
| `name`        | The name of the attribute key                                                            | `string`    | ✅         |
| `description` | A description of the attribute key                                                       | `string`    | ❌         |
| `key`         | The key used to join this attribute key to an attribute table                            | `string`    | ❌         |
| `owner`       | The owner of the attribute key, typically the email of the primary maintainer            | `string`    | ❌         |
| `ttl`         | The amount of time that attributes for the attribute key will live in the Profiles Store | `timedelta` | ❌         |

If a `key` isn't specified, the `name` will be used.

Here's an extended example using all possible arguments, based on the atomic `platform` property:

```python
from datetime import timedelta
from snowplow_signals import AttributeKey

platform_attribute_key = AttributeKey(
    name="platform_tracking_attribute_key",
    description="Attribute key for analyzing user behavior patterns across different platforms (web, mobile, server-side) to understand cross-platform engagement and optimize user experience",
    key="platform",
    owner="analytics-team@snowplow.com",
    ttl=timedelta(days=365),
)
```

Custom attribute keys are added to attribute groups in the same way as any of the out-of-the-box attribute keys.

---

# Define attributes with the Signals Python SDK
> Define Attribute objects programmatically by specifying event schemas, aggregation types, properties, and filtering criteria. Configure time periods and default values using the Python SDK.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/attribute-groups/attributes/

[Attributes](/docs/signals/concepts/#attribute-groups) are defined as part of attribute groups. To create an attribute, you'll need to set:

- A name, ideally one that describes the attribute
- Which event schema to calculate it from
- What property in the schema to consider for the calculation
- What kind of aggregation you want to calculate over time, e.g. `mean` or `last`
- What type of value you want the attribute to hold, e.g. `double` or `string`

Attribute calculation starts when the definitions are published, and aren't backdated.

## Minimal example

This is the minimum configuration needed to create an attribute:

```python
from snowplow_signals import Attribute, Event

my_attribute = Attribute(
    name="button_click_counter",
    type="int32",
    events=[
        Event(
            vendor="com.snowplowanalytics.snowplow",
            name="button_click",
            version="1-0-0",
        )
    ],
    aggregation="counter"
)
```

Once applied and active, this attribute definition will trigger every time Signals processes an event with the schema [`iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0). Starting from 0, the stored attribute will be an integer value that increases by 1 with every `button_click` event.

## Options

The table below lists all available arguments for an `Attribute`:

| Argument        | Description                                                                                                                                  | Type                                                                                                                                                                                                                       | Required? |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `name`          | The name of the attribute                                                                                                                    | `string`                                                                                                                                                                                                                   | ✅         |
| `description`   | The description of the attribute                                                                                                             | `string`                                                                                                                                                                                                                   | ❌         |
| `events`        | List of Snowplow `Event`s that the attribute is calculated on                                                                                | List of `Event` type                                                                                                                                                                                                       | ✅         |
| `aggregation`   | The calculation to be performed                                                                                                              | One of: `counter`, `sum`, `min`, `max`, `mean`, `first`, `last`, `most_frequent`, `least_frequent`, `approx_count_distinct`, `category_count`, `unique_list`                                                               | ✅         |
| `type`          | The type of the aggregation result                                                                                                           | One of: `bytes`, `string`, `int32`, `int64`, `double`, `float`, `bool`, `dict`, `unix_timestamp`, `bytes_list`, `string_list`, `int32_list`, `int64_list`, `double_list`, `float_list`, `bool_list`, `unix_timestamp_list` | ✅         |
| `criteria`      | List of `Criteria` to filter the events                                                                                                      | List of `Criteria` type                                                                                                                                                                                                    | ❌         |
| `property`      | The property of the event or entity you wish to use in the aggregation                                                                       | `string`                                                                                                                                                                                                                   | ❌         |
| `period`        | The time period window over which the aggregation should be calculated                                                                       | Python `timedelta`                                                                                                                                                                                                         | ❌         |
| `default_value` | The default value to use if the aggregation returns no results. If not set, the default value is automatically assigned based on the `type`. |                                                                                                                                                                                                                            | ❌         |

### Specifying events

The `events` list describes the types of events that the attribute is calculated from. They're references to Snowplow events that exist in your Snowplow account, based on event schemas.

An `Event` accepts the following parameters:

| Argument  | Description                                     | Type     |
| --------- | ----------------------------------------------- | -------- |
| `name`    | `event_name` column in `atomic.events` table    | `string` |
| `vendor`  | `event_vendor` column in `atomic.events` table  | `string` |
| `version` | `event_version` column in `atomic.events` table | `string` |

Use the following details for Snowplow page view, page ping, or structured events:

```python
# Page view event
sp_page_view=Event(
    name="page_view",
    vendor="com.snowplowanalytics.snowplow",
    version="1-0-0"
)

# Page ping event
sp_page_ping=Event(
    name="page_ping",
    vendor="com.snowplowanalytics.snowplow",
    version="1-0-0"
)

# Structured event
sp_structured=Event(
    name="event",
    vendor="com.google.analytics",
    version="1-0-0"
)
```

All of these parameters are optional, and work like wildcards.

To calculate an attribute for version 2-0-2 only of an event data structure with the schema `iglu:com.snowplowanalytics.snowplow.media/destination/jsonschema/2-0-2`, the `Event` would be:

```python
event=Event(
    name="destination",
    vendor="com.snowplowanalytics.snowplow.media",
    version="2-0-2"
)
```

To calculate an attribute for all versions of that event:

```python
event=Event(
    name="destination",
    vendor="com.snowplowanalytics.snowplow.media",
)
```

To calculate an attribute for all events for a specific vendor:

```python
event=Event(
    vendor="com.snowplowanalytics.snowplow.media",
)
```

### Filtering events by specific values

The `criteria` list contains the filters used to calculate an attribute from the specified event.

It allows you to be specific about which subsets of events should trigger attribute updates. For example, instead of counting all page views in a user's session, you may wish to calculate only views for an FAQs page, or a "contact us" page.

The `criteria` list takes a `Criteria` type, with possible arguments:

| Argument | Description                                          | Type                |
| -------- | ---------------------------------------------------- | ------------------- |
| `all`    | All criterion conditions must be met                 | list of `Criterion` |
| `any`    | At least one of the criterion conditions must be met | list of `Criterion` |

A `Criterion` specifies a filtering rule on the specified event.

When creating a `Criterion`, you can use its operator-like methods to define the filtering for a property of the event payload. The available methods are:

- `.eq` checks equality similar to using the `=` operator
- `.neq` checks non-equality similar to using the `!=` operator
- `.gt` checks if a property is greater than a value
- `.gte` checks if a property is greater than or equal to a value
- `.lt` checks if a property is less than a value
- `.lte` checks if a property is less than or equal to a value
- `.like` checks if a property matches a value using a `LIKE` operator
- `.in_list` checks if a property is in the list of values using an `IN` operator
-

These methods accept the property as a first argument and then the value to filter against.

| Argument   | Description                                                             | Type                                                   |
| ---------- | ----------------------------------------------------------------------- | ------------------------------------------------------ |
| `property` | The property of the event payload you want the criterion to run against | `AtomicProperty`, `EventProperty`, or `EntityProperty` |
| `value`    | The value to compare the property to                                    | Type-checked based on the operator                     |

The `AtomicProperty`, `EventProperty` and `EntityProperty` are classes to help you target a property of an event to be filtered. Use:

- `AtomicProperty` to target atomic properties in the event payload
- `EventProperty` to target properties in the event data structure in the event payload
- `EntityProperty` to target properties in the entity data structures in the event payload

Some examples:

```python
# Targets the app_id atomic property in the event payload
AtomicProperty(name="app_id")

# Targets the action property of the com.example/test_event/jsonschema/1-*-*
EventProperty(
    vendor="com.example",
    name="test_event",
    major_version=1,
    path="action"
)

# Targets the age property of the first com.example/user_context/jsonschema/1-*-* entity
EntityProperty(
    vendor="com.example",
    name="user_context",
    major_version=1,
    path="age"
)
```

For a more complete example, if you want to calculate an attribute for page views of either the FAQs or "contact us" page, the `Criteria` could be:

```python
criteria=Criteria(
    any=[
        Criterion.like(
            AtomicProperty(name="page_url"),
            "%/faq%"
        ),
        Criterion.like(
            AtomicProperty(name="page_url"),
            "%/contact-us%"
        )
    ]
)
```

The `page_url` property is from the built-in [atomic event properties](/docs/fundamentals/canonical-event/) in all Snowplow events.

## Extended examples

These examples use all the available configuration options.

### Example 1

This example extends the previous minimal example. Now the attribute is only calculated for `button_click` events where the `id` property is equal to `generate_emoji_btn`.

```python
from snowplow_signals import Attribute, Event, Criteria, Criterion, EventProperty
from datetime import timedelta

button_click_counter_attribute = Attribute(
    name="emoji_button_click_counter",
    description="The number of clicks for the 'generate emoji' button",
    type="int32",
    events=[
        Event(
            vendor="com.snowplowanalytics.snowplow",
            name="button_click",
            version="1-0-0",
        )
    ],
    aggregation="counter",
    criteria=Criteria(
        all=[
            Criterion.eq(
                EventProperty(
                    vendor="com.snowplowanalytics.snowplow",
                    name="button_click",
                    major_version=1,
                    path="id"
                ),
                "generate_emoji_btn"
            )
        ]
    ),
    period=timedelta(minutes=10),
    default_value=0
)
```

The attribute will be calculated over the last 10 minutes as a rolling window.

### Example 2

Here's a new example showing how to use the `property` option to access one of the [atomic event properties](/docs/fundamentals/canonical-event/), in this case `mkt_medium`.

In this example the attribute is calculated from either a page view or a custom event:

```python
from snowplow_signals import Attribute, Event

referrer_source_attribute = Attribute(
    name="referrer_source",
    description="Referrer",
    type="string",
    events=[
        Event(
            name="page_view",
            vendor="com.snowplowanalytics.snowplow",
            version="1-0-0"
        ),
        Event(
            name="login_landing",
            vendor="com.business.example",
            version="1-0-0"
        )
    ],
    aggregation="last",
    criteria=None,
    property=AtomicProperty(name="mkt_medium"),
    period=None,
    default_value=None
)
```

This attribute will be updated to the most recent referrer URL every time a page view or `login_landing` event is processed.

### Example 3

This example shows how to use the `property` option to access values in any part of a tracked event.

Tthe attribute is based on product prices, tracked within the product entity in an ecommerce transaction event:

```python
from snowplow_signals import Attribute, Event, Criteria, Criterion, EventProperty

my_new_attribute = Attribute(
    name="products_total_purchase_value",
    description="Total purchase value for all products",
    type="int64",
    events=[
        Event(
            vendor="com.snowplowanalytics.snowplow.ecommerce",
            name="snowplow_ecommerce_action",
            version="1-0-2",
        )
    ],
    aggregation="sum",
    criteria=Criteria(
        all=[
            Criterion.eq(
                EntityProperty(
                        vendor="com.snowplowanalytics.snowplow.ecommerce",
                        name="product",
                        major_version=1,
                        index=[0],
                        path="price"
                ),
                "transaction"
            )
        ]
    ),
    property=EntityProperty(
        vendor="com.snowplowanalytics.snowplow.ecommerce",
        name="product",
        major_version=1,
        path="price",
    ),
    period=None,
    default_value=0
)
```

This attribute will be calculated for Snowplow ecommerce events with schema [`iglu:com.snowplowanalytics.snowplow.ecommerce/snowplow_ecommerce_action/jsonschema/1-0-2`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.ecommerce/snowplow_ecommerce_action/jsonschema/1-0-2) and the `type` property `transaction`. The products in a transaction are stored as [product entities](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.ecommerce/product/jsonschema/1-0-0) in the `contexts_com_snowplowanalytics_snowplow_ecommerce_product_1` array. The `price` property of the first product is used to calculate the attribute.

---

# Define attribute groups with the Signals Python SDK
> Define StreamAttributeGroup, BatchAttributeGroup, or ExternalBatchAttributeGroup objects programmatically. Configure attributes, data sources, TTL, and versioning using the Python SDK.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/attribute-groups/

Define the behavior you want to capture in [attribute groups](/docs/signals/concepts/#attribute-groups).

## Attribute groups by data source

Each of the three available [data sources](/docs/signals/concepts/#data-sources) has its own attribute group class. Choose which one to use depending on how you want to calculate and sync the attributes.

**StreamAttributeGroup:**

Use a `StreamAttributeGroup` to calculate attributes from the real-time event stream.

```python
from snowplow_signals import StreamAttributeGroup, domain_sessionid

my_stream_attribute_group = StreamAttributeGroup(
    name="my_stream_attribute_group",
    version=1,
    attribute_key=domain_sessionid,
    owner="user@company.com",
    attributes=[
        # Previously defined attributes
        page_view_count,
        products_added_to_cart_feature,
    ],
)
```

**BatchAttributeGroup:**

Use a `BatchAttributeGroup` to calculate attributes from the atomic events table. Read more about this in the [batch calculations](/docs/signals/attributes/using-python-sdk/batch-calculations/) section.

```python
from snowplow_signals import BatchAttributeGroup, domain_sessionid

my_batch_attribute_group = BatchAttributeGroup(
    name="my_batch_attribute_group",
    version=1,
    attribute_key=domain_sessionid,
    owner="user@company.com",
    attributes=[
        # Previously defined attributes
        page_view_count,
        products_added_to_cart_feature,
    ],
)
```

**ExternalBatchAttributeGroup:**

Use an `ExternalBatchAttributeGroup` to sync attributes from an existing warehouse table. Read more about this in the [batch calculations](/docs/signals/attributes/using-python-sdk/batch-calculations/) section.

In this case, the attributes are already defined elsewhere and pre-calculated in the warehouse. Instead of `attributes`, this attribute group has `fields`.

```python
from snowplow_signals import ExternalBatchAttributeGroup, domain_sessionid, Field

my_external_batch_attribute_group = ExternalBatchAttributeGroup(
    name="my_external_batch_attribute_group",
    version=1,
    attribute_key=domain_sessionid,
    owner="user@company.com",
    batch_source=data_source,   # Assuming this object has been previously defined
    fields=[
        Field(
            name="TOTAL_TRANSACTIONS",
            type="int32",
        ),
        Field(
            name="TOTAL_REVENUE",
            type="int32",
        ),
        Field(
            name="AVG_TRANSACTION_REVENUE",
            type="int32",
        ),
    ],
)
```

***

## Attribute group options

The tables below list all available arguments for each type of attribute group:

**StreamAttributeGroup:**

| Argument        | Description                                           | Type                | Default | Required? |
| --------------- | ----------------------------------------------------- | ------------------- | ------- | --------- |
| `name`          | The name of the attribute group                       | `string`            |         | ✅         |
| `version`       | The version of the attribute group                    | `int`               | 1       | ❌         |
| `attribute_key` | The attribute key associated with the attribute group | `AttributeKey`      |         | ✅         |
| `owner`         | The owner of the attribute group                      | `Email`             |         | ✅         |
| `description`   | A description of the attribute group                  | `string`            |         | ❌         |
| `ttl`           | Time-to-live for attributes in the Profile Store      | `timedelta`         |         | ❌         |
| `attributes`    | List of attributes to calculate                       | list of `Attribute` |         | ✅         |
| `online`        | Calculate attributes (`True`) or not (`False`)        | `bool`              | `True`  | ❌         |

**BatchAttributeGroup:**

| Argument        | Description                                           | Type                | Default | Required? |
| --------------- | ----------------------------------------------------- | ------------------- | ------- | --------- |
| `name`          | The name of the attribute group                       | `string`            |         | ✅         |
| `version`       | The version of the attribute group                    | `int`               | 1       | ❌         |
| `attribute_key` | The attribute key associated with the attribute group | `AttributeKey`      |         | ✅         |
| `owner`         | The owner of the attribute group                      | `Email`             |         | ✅         |
| `description`   | A description of the attribute group                  | `string`            |         | ❌         |
| `ttl`           | Time-to-live for attributes in the Profile Store      | `timedelta`         |         | ❌         |
| `attributes`    | List of attributes to calculate                       | list of `Attribute` |         | ✅         |
| `batch_source`  | The batch data source for the attribute group         | `BatchSource`       |         | ❌         |
| `online`        | Calculate attributes (`True`) or not (`False`)        | `bool`              | `True`  | ❌         |

For `BatchAttributeGroup` groups, it's a good idea to publish initially with `online=False`. This is because Signals will be unable to calculate the attributes until the dbt project has been configured to create the new table. Once you've configured dbt, republish the attribute group with `online=True` to start calculating the attributes.

**ExternalBatchAttributeGroup:**

| Argument        | Description                                           | Type            | Default | Required? |
| --------------- | ----------------------------------------------------- | --------------- | ------- | --------- |
| `name`          | The name of the attribute group                       | `string`        |         | ✅         |
| `version`       | The version of the attribute group                    | `int`           | 1       | ❌         |
| `attribute_key` | The attribute key associated with the attribute group | `AttributeKey`  |         | ✅         |
| `owner`         | The owner of the attribute group                      | `Email`         |         | ✅         |
| `description`   | A description of the attribute group                  | `string`        |         | ❌         |
| `ttl`           | Time-to-live for attributes in the Profile Store      | `timedelta`     |         | ❌         |
| `batch_source`  | The batch data source for the attribute group         | `BatchSource`   |         | ✅         |
| `fields`        | Table columns for syncing                             | list of `Field` |         | ✅         |
| `online`        | Calculate attributes (`True`) or not (`False`)        | `bool`          | `True`  | ❌         |

***

If no `ttl` is set, the attribute key's `ttl` will be used. If the attribute key also has no `ttl`, there will be no time limit for attributes. We highly recommend setting a TTL. Our suggested default is 7 days for stream attribute groups, and 365 days for batch attribute groups.

Use the `online` property to control whether or not Signals should actively compute the attributes, or just register the configuration.

### Versioning

Use `version=1` for the first version of an attribute group. After publishing, if you want to change the definition in any way, iterate the version number.

## Testing attribute groups

To understand what the output of an attribute group will look like, use the Signals `test` method. This will output a table of attributes calculated from your `atomic` events table.

```python
from snowplow_signals import Signals

# Connect to Signals
sp_signals = Signals(
        {{ config }}
    )

# Run the test
test_data = sp_signals.test(
    attribute_group=my_attribute_group,
    app_ids=["website"] # The app_id in your Snowplow events
)
```

> **Note:** While you can filter on specific app\_ids during testing, both the streaming and batch engines may be configured to process only a subset of relevant app\_ids to avoid unnecessary compute. As a result, testing with an arbitrary app\_id may not yield expected data if it isn’t included in the configured subset.

To see which attributes an attribute group has, use `get_attribute_group()`. Here's an example:

```python
attribute_definitions = sp_signals.get_attribute_group(
    name="my_attribute_group",
    version=1,
)

print(attribute_definitions)
```

The table below lists all available arguments for `get_attribute_group()`

| Argument  | Description                     | Type     | Required? |
| --------- | ------------------------------- | -------- | --------- |
| `name`    | The name of the attribute group | `string` | ✅         |
| `version` | The attribute group version     | `int`    | ❌         |

If you don't specify a version, Signals will retrieve the latest version.

## Publishing attribute groups

Use the [`publish()` method](/docs/signals/connection/#publishing-and-deleting) to register attribute groups with Signals. This makes them available for real-time calculation and retrieval.

```python
from snowplow_signals import Signals

# Connect to Signals
sp_signals = Signals(
        {{ config }}
    )

# Publish attribute groups
sp_signals.publish([
        my_attribute_group,
        my_other_attribute_group
    ])
```

If you only want to publish the attribute group definitions, and don't want to calculate the attribute values now, set the `online=False` option.

## Attribute groups can be set to expire

Some attributes will only be relevant for a certain amount of time, and eventually stop being updated.

To avoid stale attributes staying in your Profiles Store forever, you can configure TTL lifetimes for attribute keys and attribute groups. When none of the attributes for an attribute key or attribute group have been updated for the defined lifespan, the attribute key or attribute group expires. Any attribute values for this attribute key or attribute group will be deleted: fetching them will return `None` values.

If Signals then processes a new event that calculates the attribute again, or syncs the attribute from the warehouse again, the expiration timer is reset.

## Extended stream attribute group example

This example shows all the available configuration options for a stream attribute group.

This attribute group groups attributes for a user attribute key, to be calculated in real-time.

```python
from snowplow_signals import StreamAttributeGroup, user_id

stream_attribute_group = StreamAttributeGroup(
    name="comprehensive_stream_attribute_group",
    version=2,
    attribute_key=user_id,
    owner="data-team@company.com",
    attributes=[
        page_view_count,
        session_duration,
        conversion_rate,
    ],
    description="User engagement attributes in real-time",
    ttl=timedelta(days=90),  # Attributes live in the Profiles Store for 90 days
)
```

Signals will start calculating attributes as soon as this attribute group configuration is applied.

---

# Connect to existing warehouse tables
> Connect Signals to existing warehouse tables using ExternalBatchAttributeGroup and BatchSource objects. Define table details, timestamp fields, and field mappings to sync pre-calculated attributes.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/attribute-groups/warehouse-config/

To sync existing, pre-calculated attributes to Signals, use an `ExternalBatchAttributeGroup` attribute group to define which source table to use, and which fields (columns) to use from the table. Using existing warehouse tables doesn't require any additional modeling.

Configure which table to sync by specifying a `BatchSource` object for the group.

> **Info:** A `BatchSource` isn't required for `BatchAttributeGroup` objects, only `ExternalBatchAttributeGroup`.

## Provide source table details

The `BatchSource` defines how to connect to the table of interest in your warehouse. Here's an example:

```python
from snowplow_signals import BatchSource

data_source = BatchSource(
    name="ecommerce_transaction_interactions_source",
    database="SNOWPLOW_DEV1",
    schema="SIGNALS",
    table="SNOWPLOW_ECOMMERCE_TRANSACTION_INTERACTIONS_FEATURES",
    timestamp_field="UPDATED_AT",
    owner="user@company.com",
)
```

The table below lists all available arguments for a `BatchSource`:

| Argument          | Description                                                            | Type     | Required? |
| ----------------- | ---------------------------------------------------------------------- | -------- | --------- |
| `name`            | The name of the source                                                 | `string` | ✅         |
| `description`     | A description of the source                                            | `string` | ❌         |
| `database`        | The database where the attributes are stored                           | `string` | ✅         |
| `schema`          | The schema for the table of interest                                   | `string` | ✅         |
| `table`           | The table where the attributes are stored                              | `string` | ✅         |
| `timestamp_field` | Primary timestamp of the attribute value                               | `string` | ❌         |
| `owner`           | The owner of the source, typically the email of the primary maintainer | `string` | ❌         |

The sync engine only sends rows with a newer timestamp to the Profiles Store, based on the `timestamp_field`. For each attribute key, make sure there is only one row per timestamp — otherwise, one value may be discarded arbitrarily.

## Define which fields to sync

Instead of `attributes`, this attribute group class has `fields`.

Here's an example:

```python
from snowplow_signals import ExternalBatchAttributeGroup, domain_userid, Field

attribute_group = ExternalBatchAttributeGroup(
    name="ecommerce_transaction_interactions_attributes",
    version=1,
    attribute_key=domain_userid,
    owner="user@company.com",
    batch_source=data_source,
    fields=[
        Field(
            name="TOTAL_TRANSACTIONS",
            type="int32",
        ),
        Field(
            name="TOTAL_REVENUE",
            type="int32",
        ),
        Field(
            name="AVG_TRANSACTION_REVENUE",
            type="int32",
        ),
    ],
)
```

The table below lists all available arguments for a `Field`:

| Argument      | Description                | Type                                                                                                                                                                                                                | Required? |
| ------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `name`        | The name of the field      | `string`                                                                                                                                                                                                            | ✅         |
| `description` | A description of the field | `string`                                                                                                                                                                                                            | ❌         |
| `type`        | The type of the field      | One of: `bytes`, `string`, `int32`, `int64`, `double`, `float`, `bool`, `unix_timestamp`, `bytes_list`, `string_list`, `int32_list`, `int64_list`, `double_list`, `float_list`, `bool_list`, `unix_timestamp_list`, | ✅         |

---

# Calculate and sync warehouse attributes to Signals
> Calculate new batch attributes with dbt models or sync existing warehouse tables to the Profiles Store. Use the batch engine CLI to generate dbt projects and register tables.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/batch-calculations/

You can use existing attributes that are already in your warehouse, or use the Signals batch engine to calculate new attributes in a new table.

To use historical, warehouse attributes in your real-time use cases, you will need to sync the data to the Profiles Store. Signals includes a sync engine to do this.

> **Note:** Only Snowflake and BigQuery are supported currently. However, you can also use Signals without the warehouse functionality.

Signals is configured slightly differently depending if you're using existing tables or creating new ones.

- `BatchAttributeGroup`: create a new warehouse table with new attributes, calculated from your `atomic` events table
- `ExternalBatchAttributeGroup`: sync pre-calculated values to Signals from an existing warehouse table

## Sync engine

The sync engine is a cron job that sends warehouse attributes to the Profiles Store.

The engine will be enabled when you either:

- Apply an `ExternalBatchAttributeGroup` for an existing table
- Run the batch engine `sync` command after creating new attribute tables

Once enabled, syncs begin at a fixed interval. By default, this is every 1 hour. Only the records that have changed since the last sync are sent to the Profiles Store.

## Using existing attributes

Using existing tables in your warehouse is the more straight-forward approach, as it doesn't require any additional modeling. You'll need to [define an `ExternalBatchAttributeGroup`](/docs/signals/attributes/using-python-sdk/attribute-groups/warehouse-config/), including a `BatchSource` warehouse configuration object.

To start syncing existing tables, [publish](/docs/signals/connection/#publishing-and-deleting) your `ExternalBatchAttributeGroup` group to Signals.

```python
sp_signals.publish([attribute_group])
```

The sync will begin: the sync engine will look for new records at a given interval, based on the `timestamp_field` and the last time it ran. The default time interval is 1 hour. If you use dbt, consider creating a dbt snapshot over your attributes table to capture changes if it is refreshed in a drop and recompute manner. This way you can use the `dbt_valid_from` as the `timestamp_field` for optimal incremental syncs.

## Creating new attribute tables

To create new batch attributes, you'll need to [define a `BatchAttributeGroup`](/docs/signals/attributes/using-python-sdk/attribute-groups/), and publish it to Signals. However, further steps are necessary to create the required dbt models and tables in your warehouse, and register them with Signals.

The included batch engine CLI tool will help you with this process. Check out the [batch engine tutorial](/tutorials/signals-batch-engine/start/) for a step-by-step guide.

### Installing the CLI tool

The [Signals Python SDK](https://pypi.org/project/snowplow-signals/) includes an optional CLI tool called the batch engine. It will help you create the required dbt models and tables in your warehouse, and register them with Signals.

The batch engine is installed separately from the main Python SDK.

Choose where your new Signals dbt projects will live. Install the CLI tool there with:

```bash
pip install 'snowplow-signals[batch-engine]'
```

This adds the `snowplow-batch-engine` tool to your environment.

### CLI commands

The available options are:

```text
  init              # Initialize dbt project structure and base configuration
  generate          # Generate dbt project assets
  sync       # Registers the attribute table snapshot as a data source with Signals and publishes the Attribute Group so that syncing can begin
  test_connection   # Test the connection to the authentication and API services
```

A `--verbose` flag is available for every command.

Here's an example of using the CLI:

```bash
snowplow-batch-engine init --verbose
```

### Creating and registering tables

Check out the [batch engine tutorial](/tutorials/signals-batch-engine/start/) for a walkthrough of the required steps.

The dbt models generated by the batch engine process events incrementally. This avoids unnecessary reprocessing, and along with the pre-aggregation logic, minimizes computational costs.

### Model variables

The model created for each attribute group has configurable variables. The most important one to update before running is the `snowplow__start_date` variable. This will depend on how far back you have data, and how much of it you want to process.

You will need to update the variables for each attribute group individually, by editing the `dbt_project.yml` files. The table below lists the configurable variables for each model:

| Variable                                   | Description                                                                                           | Default Value                                                                                                   |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `snowplow__start_date`                     | Date from where the model starts looking for events, based on both `load_tstamp` and `derived_tstamp` | `'2025-01-01'`                                                                                                  |
| `snowplow__app_id`                         | Filter the data on specific `app_id`s                                                                 | `[]`                                                                                                            |
| `snowplow__backfill_limit_days`            | Limit backfill increments for the `filtered_events_table`                                             | `1`                                                                                                             |
| `snowplow__late_event_lookback_days`       | The number of days to allow for late arriving data to be reprocessed during daily aggregation         | `5`                                                                                                             |
| `snowplow__min_late_events_to_process`     | The threshold number of skipped daily events to process during daily aggregation                      | `1`                                                                                                             |
| `snowplow__atomic_schema`                  | Change this if you aren't using `atomic` schema for Snowplow event data                               | `'atomic'`                                                                                                      |
| `snowplow__database`                       | Change this if you aren't using `target.database` for Snowplow event data                             |                                                                                                                 |
| `snowplow__events_table`                   | Change this if you aren't using `events` table for Snowplow event data                                | `'events'`                                                                                                      |
| `snowplow__include_current_day_in_windows` | false                                                                                                 | If set to true, the `current_day` with incomplete data is also taken into account for `last_x_day` type windows |
| `snowplow__databricks_catalog`             | `hive_metastore`                                                                                      | Catalog used for the atomic events table (Databricks only)                                                      |

---

# Define attributes programmatically with the Signals Python SDK
> Programmatically define attribute groups and services using the Signals Python SDK. Connect to Signals and publish configurations via code instead of using the Console UI.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/

The pages in this section describe how to use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) to define attribute groups and services. See the [interventions section](/docs/signals/interventions/) to learn how to [define interventions](/docs/signals/interventions/using-python-sdk/) using the SDK.

Start by [connecting to Signals](/docs/signals/connection/) to create a `Signals` object. You'll need this connection to publish configurations to Signals.

Check out the [attribute groups](/docs/signals/attributes/using-python-sdk/attribute-groups/) and [services](/docs/signals/attributes/using-python-sdk/services/) pages to learn how to configure them programmatically.

---

# Define services programmatically with the Signals Python SDK
> Define Service objects that group multiple attribute groups with version control. Use the Python SDK to publish services for stable attribute consumption.
> Source: https://docs.snowplow.io/docs/signals/attributes/using-python-sdk/services/

[Services](/docs/signals/concepts/#services) group attribute groups together for serving to your applications.

There are two ways to define the attribute groups in your service:

- `AttributeGroup` objects: use the objects directly if you have them available in your code
- Dictionaries: refer to the attribute group by name and version, in the format `{"name": "group_name", "version": 1}`

This example code shows both options:

```python
from snowplow_signals import Service

# Refers to the attribute groups by name
my_service = Service(
    name='my_service',
    description='A collection of attribute groups',
    owner="user@company.com",
    attribute_groups=[
        # Specify exact versions using dictionaries
        {"name": "user_attributes", "version": 2},
        {"name": "session_attributes", "version": 1},
    ],
)

# Uses the objects directly
my_service = Service(
    name='my_service',
    description='A collection of attribute groups',
    owner="user@company.com",
    attribute_groups=[
        # Previously defined attribute groups
        my_attribute_group,
        another_attribute_group
    ],
)
```

The table below lists all available arguments for a `Service`

| Argument           | Description                                                             | Type                             | Required? |
| ------------------ | ----------------------------------------------------------------------- | -------------------------------- | --------- |
| `name`             | The name of the service                                                 | `string`                         | ✅         |
| `description`      | A description of the service                                            | `string`                         | ❌         |
| `owner`            | The owner of the service, typically the email of the primary maintainer | `string`                         | ✅         |
| `attribute_groups` | List of attribute groups with optional version specification            | list of `AttributeGroup` or dict | ❌         |

## Publishing services

Use the [`publish()` method](/docs/signals/connection/#publishing-and-deleting) to register services with Signals. This makes them available for use.

```python
from snowplow_signals import Signals

# Connect to Signals
sp_signals = Signals(
        {{ config }}
    )

# Publish services
sp_signals.publish([
        my_service,
        my_other_service
    ])
```

---

# Core Signals components and concepts
> Signals introduces attribute groups for defining behavioral data, services for consuming attributes, and interventions for triggering actions. Learn about data sources, attribute keys, and the Profiles Store.
> Source: https://docs.snowplow.io/docs/signals/concepts/

Signals introduces a new set of data governance concepts to Snowplow. As with schemas for Snowplow event data, Signals components are strictly defined, structured, and versioned.

Signals has three main configurable components:

- **Attribute groups**, for defining and calculating attributes
- **Services**, for consuming calculated attributes in your applications
- **Interventions**, for consuming calculated attributes and triggering actions in your applications

**Attribute groups** are where you define the behavioral data you want to calculate. Each attribute group contains multiple **attributes** - the specific facts about user behavior you want to measure or take action on - along with the configuration that defines how to calculate them, and from what data. Attributes can only be defined within attribute groups; they are effectively properties of the attribute group.

To use attributes to trigger actions such as in-app messages, discounts, or personalized journeys, use services or interventions.

**Services** provide a stable interface layer between your calculated attributes and your applications. Each service can contain multiple attribute groups, pinned to specific versions. You'd build the logic within your application for how to use the retrieved attributes. **Interventions** are a separate abstraction for defining when to trigger actions in your application.

![Detailed Signals architecture showing how attribute groups, services, and interventions connect to the Profiles Store](/assets/images/overview-detailed-eca044c19e5f9b3860254f7b03342c12.png)

## Attribute groups

Attribute groups are where you define the data you want to calculate. Each attribute group is a versioned collection that specifies:

- The **attributes** to calculate - the specific behavioral facts about users
- The **data source** - whether to calculate from the real-time stream, or in batch from the warehouse
- The **attribute key** that provides the analytical context
- Other metadata such as description or owner

### Types of attribute

Attributes describe what kind of calculation to perform, and what event data to evaluate. They can only exist within attribute groups.

Attributes can be categorized into four main types, depending on the type of user behavior you want to understand:

| Type          | Description                                            | Example                              |
| ------------- | ------------------------------------------------------ | ------------------------------------ |
| Time windowed | Actions that happened within the last X period of time | `products_added_to_cart_last_10_min` |
| Lifetime      | Calculated over all the available data                 | `total_product_price_clv`            |
| First touch   | The first event or property that happened              | `first_mkt_source`                   |
| Last touch    | The most recent event or property that happened        | `last_device_class`                  |

Signals includes a range of different aggregations for calculating attributes, including `mean`, `counter`, or `unique_list`. See the full list in the [attribute configuration](/docs/signals/attributes/attributes/) page.

### Attribute keys

An attribute key is an identifier that provides the analytical context for all attribute calculations within a group. The identifier can be any atomic field of a Snowplow event, such as `domain_userid`.

To demonstrate the necessity of attribute keys, consider the attribute `num_views_in_last_10_min`. It represents a count of page view events, with a 10 minute time limit. This table lists some possible meanings of the attribute, based on the attribute key configured for its attribute group:

| Attribute                  | Attribute key      | Description                                                                         |
| -------------------------- | ------------------ | ----------------------------------------------------------------------------------- |
| `num_views_in_last_10_min` | User               | How many pages a user has viewed within the last 10 minutes                         |
| `num_views_in_last_10_min` | Page               | How many page views a page has received within the last 10 minutes                  |
| `num_views_in_last_10_min` | Product            | How many times a product has been viewed within the last 10 minutes                 |
| `num_views_in_last_10_min` | App                | How many page views occurred within an app in the last 10 minutes                   |
| `num_views_in_last_10_min` | Device             | How many page views came from a specific device in the last 10 minutes              |
| `num_views_in_last_10_min` | Marketing campaign | How many page views were generated by a campaign in the last 10 minutes             |
| `num_views_in_last_10_min` | Geographic region  | How many page views came from users in one region within the last 10 minutes        |
| `num_views_in_last_10_min` | Customer segment   | How many page views were generated by users in a segment within the last 10 minutes |

Each of these is likely to have a different calculated value.

You can [define your own attribute keys](/docs/signals/attributes/attribute-groups/#attribute-keys), or use the built-in ones. Signals comes with predefined attribute keys for user, device, and session. Their identifiers are from the out-of-the-box atomic [user-related fields](/docs/fundamentals/canonical-event/#user-fields) in all Snowplow events.

| Attribute key      | Type     |
| ------------------ | -------- |
| `user_id`          | `string` |
| `domain_userid`    | `uuid`   |
| `network_userid`   | `uuid`   |
| `domain_sessionid` | `uuid`   |

## Data sources

Whether to compute attributes in real-time from the event stream or in batch from the warehouse is an important decision. Broadly, you might use:

- **Stream** for real-time use cases, such as tracking the latest product a user viewed, or the number of page views in a session
- **Batch** sources (warehouse tables) for historical analysis, such as calculating a user's purchase history or average session length

This table summarizes the options for different types of processing:

| Feature                            | Supported in real-time stream                                     | Supported in batch            |
| ---------------------------------- | ----------------------------------------------------------------- | ----------------------------- |
| Real-time calculation              | ✅                                                                 | ❌                             |
| Time windowing operations          | ✅                                                                 | ✅                             |
| Computing user lifetime attributes | ✅ from the point at which the attribute was defined               | ✅                             |
| Reprocessing data                  | ❌ attributes are only calculated from the moment they are defined | ✅                             |
| Non-Snowplow data                  | ❌                                                                 | ✅ using external batch source |

### Stream source

When Signals is deployed in your pipeline, the event stream is read by the streaming engine. All tracked events are inspected.

Real-time stream flow:

1. Behavioral data event is received by [Collector](/docs/pipeline/collector/)

2. Event is enriched by [Enrich](/docs/pipeline/enrichments/)

3. The Signals stream engine reads from the events stream

4. The stream engine checks if attributes are defined for the event

5. Are there any attributes to calculate?

   - No: nothing happens, and the process ends
   - Yes: processing continues

6. Signals evaluates and computes the attributes

7. Updated attributes are pushed to the Profiles Store

### Batch source

The batch data source uses dbt to generate and calculate new tables of attributes from your Snowplow atomic events table. Signals then syncs them to the Profiles Store periodically using the sync engine.

Batch flow:

1. Behavioral data events arrive in the warehouse

2. Signals compares timestamps to check for new rows in the atomic events table

3. Are there any new rows?

   - No: nothing happens
   - Yes: processing continues

4. Signals runs dbt models to update the attribute tables

5. Updated tables are synced to the Profiles Store

### External batch source

Use an external batch source to sync tables of existing, pre-calculated values to the Profiles Store. The external batch tables can be any data. For example, you may want to include transactional data in your Signals use case.

## Services

Services are where you define how to use the calculated attributes in your applications.

Rather than connecting applications directly to attribute groups, services allow you to consume specific attribute group versions. This provides a consistent set of consumable attributes. We strongly recommend using services in production applications.

By using services you can:

- Iterate on attribute definitions without worrying about breaking downstream processes
- Migrate to new attribute group versions by updating the service definition, without having to update the application code

## Interventions

Interventions are opportunities to take actions to improve user outcomes. They're automated triggers fired by changes in attribute values, or by your own applications.

They can be thought of as "if-then" rules that run in real time. For example:

- `If` a user has viewed five products in the last ten minutes
- `Then` automatically send that user a personalized offer

In practice, this rule could work like this:

1. User views product pages
2. The application tracks the user behavior and sends events
3. Signals evaluates the events and updates attribute values
4. Signals checks whether the new attribute values meet any intervention conditions
5. The attribute `product_view_count_last_10_min` has just been updated to `5`, matching the intervention's trigger criteria
6. The application receives the subscribed intervention payload
7. The application sends a personalized offer to the user

Interventions are usually relevant only within a certain moment. Therefore, an intervention is sent only the first time the criteria are met.

Standard rule-based interventions can have multiple conditions, or trigger criteria, based on the values of different attributes.

### Targeting

Interventions are targeted based on attribute keys, which determine both the scope and specificity of when they're delivered.

> **Note:** Interventions can be defined against any attribute key, as long as its values are non-enumerable. For example, the built-in Snowplow attribute keys `domain_userid`, `domain_sessionid`, and `network_userid` are suitable targets since their values are canonically formatted UUIDs, e.g. `8c9104e3-c300-4b20-82f2-93b7fa0b8feb`.

For individual-level targeting, use user-specific attribute keys. For example, use `domain_userid` to target individual users, or `domain_sessionid` to target users during specific sessions, when session-level conditions are met.

For broadcast-level targeting, use attribute keys related to the application context. For example, you could use `campaign_id` to target all users who arrived from a specific marketing campaign.

Interventions can have multiple attribute keys. By default, the intervention will target the attribute keys associated with their criteria attributes.

Custom targeting can be useful when you have broad criteria but want a more narrow target. For example, if your criteria use a broad attribute like `page_id`, you might not want to send the intervention to everyone on that page. If you're also checking user-specific criteria, target `domain_userid` to reach only the specific user who meets all conditions.

Another use for custom targeting is when you want to target an attribute key that's available in the triggering event, but isn't part of your intervention logic. For example, to send an intervention to a `domain_userid` when they do a certain number of things in a single `pageview_id`. By targeting the user rather than the page, you can ensure that the intervention is received even if they've gone onto a new page by the time the triggering event is processed, or if the application is not subscribed to the `pageview_id`.

### Subscribing

To receive and take action on interventions, you'll need to:

- [Subscribe](/docs/signals/interventions/subscribe/) to them within your application
- Define the logic of how the application should react

> **Note:** Subscription is by attribute key, not by intervention.
>
> A subscription using a specific attribute key ID, for example a `domain_userid` ID for the current user, will receive all triggered interventions for that ID.

Once subscribed, all triggered interventions will be streamed to the consumer application.

For back-end applications using the [Signals Python SDK](https://pypi.org/project/snowplow-signals/), [Signals Node.js SDK](https://www.npmjs.com/package/@snowplow/signals-node), or [Signals API](/docs/signals/connection/#signals-api), subscribe within the application code by passing in the relevant attribute key IDs. For web applications, use the [Signals browser plugin](/docs/signals/interventions/subscribe/) for the JavaScript tracker to subscribe automatically to relevant interventions.

### Targeting example

As an example, consider these two interventions:

- `intervention1` targets the `domain_userid` and `domain_sessionid` attribute keys

  - It has two criteria rules, where `any` must be met
  - The first rule is: `attribute_group1:page_view_count == 10`
  - The second rule is: `attribute_group2:ad_count is not null`

- `intervention2` targets the `domain_userid` attribute key
  - It has one criteria rule, `attribute_group3:button_click_count > 20`

To receive interventions for the current user and session, subscribe to their interventions by providing the `domain_userid` and `domain_sessionid` ID values.

Interventions will be triggered when:

- The user views 10 pages, **or** the session has an ad event

  - `intervention1` will be delivered to one of the subscribed targets.
  - Signals will initially create a payload for each target. However, they'll have the same ID as they're triggered by a single event, and will be deduped.

- The user exceeds 20 button clicks for the first time
  - `intervention2` will be delivered to the subscribed user.

Assuming it's the user's first session, the flow looks like this:

- First page view

- First ad event
  - `intervention1` is triggered and delivered to one of the targets

- More page views, button clicks, and ad events

- Tenth page view
  - Nothing happens: because `intervention1` sent already on seeing the ad event in this session, it's not triggered again

- Twenty-first button click

  - The attribute value changes from 20 to 21, crossing the required threshold
  - `intervention2` is triggered and delivered to the user target

- Twenty-second button click
  - Nothing happens: the threshold has already passed

This user has already seen more than 10 pages, so `intervention1` can never be triggered by that rule for them. However, if their current session expires, and the application subscribes to their new `domain_sessionid` ID, `intervention1` can be triggered again by the first ad event of the session.

Likewise, the user has already clicked a button more than 20 times, so `intervention2` will never be sent to them again.

### Types of intervention

Signals has two types of interventions: rule-based or direct.

**Rule-based** interventions are the default, and are:

- Based on attribute values
- Defined in advance via Snowplow Console, Python SDK, or API
- Triggered automatically when their criteria are met

Use cases include:

- Real-time personalization
- Agentic chatbots
- Dynamic pricing

**Direct** interventions are:

- Not based on attribute values or rules
- Configurable only via the Python SDK or API
- Sent directly on pushing an intervention configuration to Signals

Use cases for direct interventions include:

- Emergency notifications, e.g. system outages
- Real-time business decisions, e.g. breaking news
- Manual campaign targeting
- Sensitive communications requiring authentication or authorization

> **Info:** Direct interventions use the [Signals API](/docs/signals/connection/#signals-api) `api/v1/registry/interventions` endpoints under the hood. These endpoints check that the submitted bearer token in the API call is valid and authorized.
>
> Conversely, subscription to automated rule-based interventions uses the `api/v1/interventions` endpoint. This endpoint is public: it doesn't perform authentication or authorization, just checks for the requested attribute keys.

## Profiles Store

The Profiles Store is a database where Signals saves all your calculated attribute values. When Signals calculates attributes from your events or warehouse data, or syncs pre-calculated data, it stores them here organized by attribute group. Your applications retrieve these stored values using the Signals SDKs or API.

The Profiles Store keeps track of current attribute values, and automatically removes old data based on the TTL settings you configure for each attribute group. It acts as the central source of truth for your Signals deployment.

---

# Connect to Snowplow Signals
> Set up a Signals connection through Snowplow Console or Sandbox to start configuring attributes and interventions. Connect using the Python SDK, Node.js SDK, or REST API with your deployment credentials.
> Source: https://docs.snowplow.io/docs/signals/connection/

If you're new to Signals, you'll need to set up a Signals connection in one of two ways:

- Log in to [Snowplow Console](https://console.snowplowanalytics.com) and navigate to the **Signals** section (for Snowplow customers with Signals).
- Set up a Signals instance with [Signals Sandbox](https://try-signals.snowplow.io/dashboard) to explore the product's features and capabilities.

## Snowplow Console

Click **Enable** to start setting up a Signals connection.

![Snowplow Console Signals section showing Enable button to set up connection](/assets/images/console-no-connection-152c34507c46b982e913d47be8f7a082.png)

> **Note:** Signals can also be deployed without connecting to a warehouse.

You'll need to:

- Select which warehouse to use
- Specify your warehouse account details
- Specify your Snowplow atomic events table
- Run the provided script

Click **Test and create connection** to trigger the Signals deployment. You'll be able to start using Signals as soon as the infrastructure is ready.

To use the UI to manage Signals, navigate to the **Signals** section.

Use the configuration interface to define [attribute groups](/docs/signals/attributes/attribute-groups/), [services](/docs/signals/attributes/services/), and [interventions](/docs/signals/interventions/).

![Console Signals landing page with navigation for attribute groups, services, and interventions](/assets/images/console-landing-c09cd5b000aef369b1e855ab7f9294da.png)

## Signals Sandbox

Alternatively, you can use the Signals Sandbox to test out the functionality and features of Signals. When you log in to the Signals Sandbox dashboard, a temporary Signals instance will be deployed.

![Signals Sandbox landing page with pipeline deployment in progress](/assets/images/sandbox-deployment-76fbca5be29cc360e32c7e324bb2a169.png)

## Connection credentials

You will need different connection values depending on your deployment type.

**Console:**

To connect to Signals using the Signals SDKs or API, you will need four values. Use the Console **Overview** page to access them.

| Value           | Description                             | Where to get it                                          | Format                                             |
| --------------- | --------------------------------------- | -------------------------------------------------------- | -------------------------------------------------- |
| Signals API URL | The API URL for your Signals deployment | Console > **Signals** > **Overview**                     | `https://{{123abc}}.signals.snowplowanalytics.com` |
| API key         | A Snowplow API key                      | [Generated by you](/docs/account-management/) in Console | UUID                                               |
| API key ID      | A Snowplow API key ID                   | [Generated by you](/docs/account-management/) in Console | UUID                                               |
| Organization ID | Your Snowplow organization ID           | Console > **Signals** > **Overview**                     | UUID                                               |

Add these four tokens to your environment secrets.

**Signals Sandbox:**

To connect to Signals using the Signals SDKs or API, you will need two values. These are found at the bottom of the Signals Sandbox dashboard under **Configuration details**.

| Value           | Description                                  | Where to get it  | Format                              |
| --------------- | -------------------------------------------- | ---------------- | ----------------------------------- |
| Signals API URL | The API URL for your Signals deployment      | Profiles API URL | `https://{{123abc}}.svc.snplow.net` |
| Sandbox Token   | An authorization token to connect to Signals | Sandbox Token    | UUID                                |

Add these values to your environment secrets.

***

## Signals Python SDK

Use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) to define attribute groups, services, and interventions via code. You can also retrieve calculated attributes and subscribe to interventions using this SDK.

To use the Python SDK, first choose where to store your Signals configurations. We recommend creating a new repository. The easiest way to use the SDK is within a Jupyter notebook.

Install the SDK into your environment:

```bash
pip install snowplow-signals
```

Once installed, you can start to define configuration components. To test or publish your configuration, or retrieve calculated attributes or interventions from the Profiles Store, you'll need to connect to your Signals deployment.

Create a `Signals` object by passing in the required values depending on your deployment type.

**Console:**

```python
from snowplow_signals import Signals

sp_signals = Signals(
    api_url=SIGNALS_DEPLOYED_URL,
    api_key=CONSOLE_API_KEY,
    api_key_id=CONSOLE_API_KEY_ID,
    org_id=ORG_ID,
)
```

**Signals Sandbox:**

```python
from snowplow_signals import SignalsSandbox

sp_signals = SignalsSandbox(
    api_url=SIGNALS_DEPLOYED_URL,
    sandbox_token=SIGNALS_SANDBOX_TOKEN,
)
```

***

The created `Signals` object has the following methods:

| Method                   | Description                                                                 |
| ------------------------ | --------------------------------------------------------------------------- |
| `publish`                | Registers the provided objects with Signals                                 |
| `unpublish`              | Unpublishes objects from Signals                                            |
| `delete`                 | Fully deletes objects from Signals (must unpublish first)                   |
| `test`                   | Tests an attribute group against the atomic events table                    |
| `get_service_attributes` | Retrieves attributes for a specific service from the Profiles Store         |
| `get_group_attributes`   | Retrieves attributes for a specific attribute group from the Profiles Store |
| `get_attribute_group`    | Retrieves an attribute group from the Profiles Store                        |
| `push_intervention`      | Push an intervention to subscribers for a set of attribute keys             |
| `pull_interventions`     | Open a streaming subscription of interventions for a set of attribute keys  |

Check out the [attribute groups](/docs/signals/attributes/using-python-sdk/attribute-groups/), [services](/docs/signals/attributes/using-python-sdk/services/), and [interventions](/docs/signals/attributes/using-python-sdk/) pages to learn how to configure them programmatically.

Read more about retrieving calculated attributes [here](/docs/signals/attributes/), and about interventions [here](/docs/signals/interventions/subscribe/).

### Publishing and deleting

Use the same object management methods for attribute groups, services, attribute keys, and interventions:

- Use `publish()` to register objects with Signals. This makes them available for real-time calculation and retrieval.
- Use `unpublish()` to stop active calculation without losing the object definitions.
- Use `delete()` to permanently remove objects from Signals. Objects must be unpublished before deletion. If you delete an attribute group, the calculated attributes in the Profiles Store will also be deleted.

```python
from snowplow_signals import StreamAttributeGroup, Service, RuleIntervention

# Define your objects (assuming these are already created)
objects_to_manage = [my_attribute_group, my_service, my_intervention]

# 1. Publish objects
published_objects = sp_signals.publish(objects_to_manage)

# 2. Unpublish objects
unpublished_objects = sp_signals.unpublish(objects_to_manage)

# 3. Delete objects permanently - must unpublish first
sp_signals.delete(objects_to_manage)
```

## Signals Node.js SDK

Use the [Node.js SDK](https://www.npmjs.com/package/@snowplow/signals-node) to retrieve calculated attributes.

Install the SDK into your application:

```bash
npm i @snowplow/signals-node
# or
yarn add @snowplow/signals-node
# or
pnpm i @snowplow/signals-node
```

Create a `Signals` object by passing in the required values.

**Console:**

```typescript
import { Signals } from '@snowplow/signals-node';

const signals = new Signals({
  baseUrl: SIGNALS_DEPLOYED_URL,
  apiKey: CONSOLE_API_KEY,
  apiKeyId: CONSOLE_API_KEY_ID,
  organizationId: ORG_ID,
});
```

**Signals Sandbox:**

```typescript
import { Signals } from '@snowplow/signals-node';

const signals = new Signals({
  baseUrl: SIGNALS_DEPLOYED_URL,
  sandboxToken: SIGNALS_SANDBOX_TOKEN,
});
```

***

The created `Signals` object has the following methods:

| Method                      | Description                                                                 |
| --------------------------- | --------------------------------------------------------------------------- |
| `getServiceAttributes`      | Retrieves attributes for a specific service from the Profiles Store         |
| `getGroupAttributes`        | Retrieves attributes for a specific attribute group from the Profiles Store |
| `getBatchServiceAttributes` | Retrieves attributes for multiple identifiers from a service                |

Read more about retrieving calculated attributes [here](/docs/signals/attributes/), and about interventions [here](/docs/signals/interventions/subscribe/).

## Signals API

The Signals API allows you to directly configure and retrieve attributes and interventions. To access the full Swagger API documentation for your Signals deployment, use your Signals API URL followed by `/docs/`:

```bash
{{API_URL}}/docs/
```

Your API documentation is linked in Console on the **Overview** page, under **Configuration details**.

---

# Get started with Snowplow Signals
> Learn how to use Snowplow Signals to compute user attributes from event data, retrieve them in real time for personalization and agentic applications, and trigger automated actions based on user behavior.
> Source: https://docs.snowplow.io/docs/signals/get-started/

Snowplow Signals computes user attributes from your data. You can retrieve these attributes in your applications to take real-time actions. Signals can also send automated triggers to your applications based on calculated attributes.

Use Signals to:

- Provide enriched user context to chatbots and other agentic applications, in near real time
- Deliver personalized recommendations, dynamic pricing, and adaptive UIs based on current behavior
- Trigger actions automatically when users meet specific criteria

By default, Signals calculates attributes from your real-time Snowplow event stream, but you can also configure it to calculate from warehouse data.

## Try Signals for free

Signals Sandbox is a [free, lightweight sandbox environment](https://try-signals.snowplow.io/) where you can try out Signals without needing a Snowplow account or pipeline. Log in with your GitHub account to get started.

The Sandbox provides you with:

- Signals infrastructure and Snowplow pipeline deployed in a dedicated environment
- [Event Collector](/docs/fundamentals/) endpoint
- Signals Profiles API endpoint
- Sandbox Token for authentication

Check out the [real-time interventions tutorial](/tutorials/signals-interventions/start/) for a hands-on introduction to using the Sandbox. The tutorial uses a demo ecommerce website to generate events, but you could also send your own events to the Sandbox Collector using any [Snowplow tracker](/docs/sources/).

Use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) or [Signals API](/docs/signals/connection/#signals-api) to start experimenting with [attributes](/docs/signals/concepts/#attribute-groups) and [interventions](/docs/signals/concepts/#interventions).

## Architecture

You will need a [Snowplow pipeline](/docs/get-started/) to use Signals. Your Signals infrastructure is deployed into the same cloud as your pipeline.

> **Note:** Only Snowflake and BigQuery are supported currently. However, you can also use Signals without the warehouse functionality.

Signals consists of several new infrastructure components. When running Signals, your Snowplow pipeline will continue to process events as usual.

The core Signals components are:

- **Profiles Store**: stores calculated attributes and configurations
- Signals **SDKs** and **API**: allow you to manage and fetch attributes and interventions
- **Streaming engine**: computes attributes from Snowplow events in stream, and sends them directly to the Profiles Store
- **Sync engine**: periodically updates the Profiles Store with batch attributes
- **Batch engine**: runs in your warehouse to compute attributes from warehouse tables

![Snowplow Signals architecture diagram showing core components including Profiles Store, SDKs, streaming engine, and batch engine](/assets/images/overview-incl-batch-engine-66e509478551fca3e794c55f8aeb9fbb.svg)

## Workflow

These are the high-level steps for using Signals:

1. Decide on the business logic
2. Apply the configurations to Signals
3. Use the attributes and interventions to take action in your application

Check out the [quick start tutorial](/tutorials/signals-quickstart/start/) for help getting started.

### 1. Decide on the business logic

Your first step is to decide what changes in user behavior you're aiming for. What systems or data will you need to achieve this? This planning will help you decide which attributes you want to calculate, and which interventions you're interested in defining.

You'll also need to decide whether to calculate attributes from your real-time event stream (default), or from warehouse data, or both.

Read more about attributes and interventions on the [concepts](/docs/signals/concepts/) page.

### 2. Apply the configuration

We recommend using [Console](https://console.snowplowanalytics.com) to define your attributes and interventions. You could also use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/), or even the [Signals API](/docs/signals/connection/#signals-api).

Once you've created your configurations, apply them to Signals by publishing them. It will start calculating attributes and populating the Profiles Store.

### 3. Take action in your application

Retrieve calculated attributes in your application using the [Node.js](https://www.npmjs.com/package/@snowplow/signals-node) or [Python](https://pypi.org/project/snowplow-signals/) Signals SDKs. You could also use the Signals API.

Use the attributes to update the user experience, or subscribe to [interventions](/docs/signals/concepts/) to automatically take action based on user behavior.

## Resources

Check out the Signals foundations tutorials:

- [Quick Start](/tutorials/signals-quickstart/start/) for defining stream attributes using the UI or the Python SDK
- [Set up the Signals batch engine](/tutorials/signals-batch-engine/start/) to calculate and sync attributes from warehouse data

Follow the [real-time prospect scoring with ML](/tutorials/signals-ml-prospect-scoring/intro/) solution accelerator to explore using Signals with a machine learning use case.

---

# Define and receive interventions to trigger real-time actions with Signals
> Create rule-based interventions with criteria and targeting to automatically trigger actions when attribute values change. Configure using Console UI, Python SDK, or API.
> Source: https://docs.snowplow.io/docs/signals/interventions/

[Interventions](/docs/signals/concepts/#interventions) are automated triggers that enable real-time actions based on user behavior. You'll need to define and [subscribe](/docs/signals/interventions/subscribe/) to them.

There are three methods for defining interventions in Signals:

- Snowplow Console UI
- [Signals Python SDK](/docs/signals/interventions/using-python-sdk/)
- [Signals API](/docs/signals/connection/#signals-api)

To create an intervention using the UI, go to **Signals** > **Interventions** in Snowplow Console and follow the instructions.

The first step is to specify:

- A unique name
- An optional description
- The email address of the primary owner or maintainer

![Create intervention form with name, description, and owner fields](/assets/images/intervention-create-92808b8ed9db1d48636405066ce3df83.png)

Next, configure when the intervention should trigger.

## Criteria

Criteria are the conditional rules that determine when an intervention should trigger.

> **Note:** An intervention is sent only the first time the criteria are met. Read an example of how this works on the [Concepts page](/docs/signals/concepts/#targeting-example).

Defining intervention criteria has three steps:

1. Select which attribute from a published attribute group to evaluate
2. Choose which logical operator to use
3. Enter the value to trigger on

![Intervention criteria configuration showing attribute selection, operator, and value fields](/assets/images/intervention-criteria-attribute-17a46f32fc8d0df79bcee451db3b0b80.png)

When adding more than one criteria, you can require all or any of them to be met.

![Multiple intervention criteria with \&quot;all\&quot; or \&quot;any\&quot; logic selection](/assets/images/intervention-criteria-all-d05e3e972c1f1ec916af4bbee3ac34e3.png)

These attributes are both from groups with the `domain_userid` attribute key. Therefore, this intervention is targeted to users. When a [subscribed](/docs/signals/interventions/subscribe/) user reaches 10 page views while using Chrome, the intervention will trigger.

## Target and payload

Define a custom intervention target scope by selecting [attribute keys](/docs/signals/concepts/#targeting). These are the attribute keys that will receive the information.

By default, the intervention will target the attribute keys of the attribute groups defined in the criteria. Specify attribute keys here if you want different targets.

You can also select one or more attribute groups to be sent with the intervention. When the intervention triggers, it will include the latest values for all attributes in the selected groups.

![Intervention delivery configuration showing attribute key targeting and attribute group payload selection](/assets/images/intervention-delivery-8b7a64861da887b3a6be2a28247f5d92.png)

## Publishing the intervention

Once you're happy with your intervention configuration, click **Create intervention** to save it. It will be saved as a draft, and not yet available to Signals.

![Draft intervention page showing Edit and Publish buttons](/assets/images/intervention-draft-cec4394a4e720ceae7fb0c2442f089f6.png)

Click the **Edit** button if you want to make changes to the intervention.

To send the intervention configuration to your Signals infrastructure, click the **Publish** button. This will allow Signals to start monitoring attribute value changes.

![Published intervention page with sample subscription code](/assets/images/intervention-published-937a3576030437ba8a7363b1e8e8e07e.png)

The intervention page also includes sample code to help you subscribe to it. Read more about this in the [receiving interventions](/docs/signals/interventions/subscribe/) page.

### Versioning

Interventions are versioned. This allows you to iterate on the definitions without breaking downstream processes. All interventions start as `v1`. If you make changes to the definition, the version will be automatically incremented.

Within criteria, the attributes are always evaluated based on the latest published version of the attribute group that contains the attribute. For example, if `attribute_group` v1, v2, and v3 all have the required `attribute`, v3 will be used. If you then publish `attribute_group` v4, which removes `attribute`, v3 will still be used.

## Testing

Signals provides two ways to test an intervention: a preview you can run before publishing, and a live test you can trigger once the intervention is published.

### Preview

While defining an intervention, you can preview how many users would have received it by running it against the current state of your warehouse data. Click **Run preview** to see how many attribute keys match the criteria you've defined.

![Intervention preview showing test results with match count against scanned warehouse records](/assets/images/intervention-preview-ca15ec102432a285849e6ffcac19d5fa.png)

### Send a test intervention

Once the intervention is published, you can send a test by specifying a particular attribute key identifier. Signals will dispatch an intervention to that identifier so you can verify your subscription is set up correctly and diagnose any connection errors before relying on it in production.

![Intervention connection test showing attribute key input field to send a test intervention](/assets/images/intervention-connection-test-a96363911003583adc095e425a80c5c4.png)

## Deleting an intervention

To unpublish or delete an intervention, click the `⋮` button on the details page.

![Intervention management menu showing Edit, Unpublish, and Delete options](/assets/images/intervention-edit-delete-ecd541c2c5c565cb3b87489261ced4fb.png)

Unpublishing is version specific. You can republish it later if needed. Choose **Delete** to permanently delete all versions of the intervention.

---

# Subscribe to and receive interventions in your applications
> Subscribe to interventions by attribute key ID to automatically respond to user behavior changes. Use the Python SDK, browser tracker plugin, or Signals API to receive intervention payloads.
> Source: https://docs.snowplow.io/docs/signals/interventions/subscribe/

Subscribe to interventions to automatically respond within your application. You have three options for receiving interventions, depending on your use case or application:

- [Signals Python SDK](https://pypi.org/project/snowplow-signals/)
- Plugin for the [browser tracker](/docs/sources/web-trackers/)
- [Signals API](/docs/signals/connection/#signals-api)

Subscription is by attribute key ID, not by individual intervention. Start by [connecting to Signals](/docs/signals/connection/).

> **Note:** An intervention is sent only the first time the criteria are met. Read an example of how this works on the [Concepts page](/docs/signals/concepts/#targeting-example).

## Using the Signals Python SDK

Subscribe by providing IDs for the attribute keys you're interested in receiving interventions for. The IDs must be in a non-enumerable format, such as UUIDs.

```python
from snowplow_signals import AttributeKeyIdentifiers, InterventionInstance

# Attribute keys to subscribe to
targets = AttributeKeyIdentifiers({
  "domain_sessionid": ["8c9104e3-c300-4b20-82f2-93b7fa0b8feb"],
  "domain_userid": ["218e8926-3858-431d-b2ed-66da03a1cbe5"],
})

# Define the subscription
subscription = sp_signals.pull_interventions(targets)

# Add a custom handler to be called with each intervention received
subscription.add_handler(print)

# Open the subscription request and begin fetching interventions as they are published
subscription.start()

# Block until an intervention is received
# Because of the `print` handler above, the intervention will be printed out
intervention_instance = subscription.get()

# Cancel the subscription and abort the connection/background thread
subscription.stop()
```

## Using the browser tracker plugin

For web applications using the Snowplow [browser tracker](/docs/sources/web-trackers/), you can subscribe to interventions using the [Signals browser plugin](https://www.npmjs.com/package/@snowplow/signals-browser-plugin).

The workflow is:

1. Create a Snowplow tracker with the plugin configured
2. Add custom handlers to react to the interventions
3. Subscribe to interventions

```typescript
import { newTracker } from '@snowplow/browser-tracker';
import {
  SignalsInterventionsPlugin,
  addInterventionHandlers,
  subscribeToInterventions,
} from '@snowplow/signals-browser-plugin';

// Install the Signals Intervention plugin
newTracker('sp1', '{{collector_url}}', {
  appId: 'my-app-id',
  plugins: [ SignalsInterventionsPlugin() ],
});

// Add custom handlers
addInterventionHandlers({
  myHandler(intervention) {
    console.log("intervention received!", intervention);
  },
});

// Subscribe to interventions using your Signals API endpoint
subscribeToInterventions({
  endpoint: "000000000000.signals.snowplowanalytics.com",
});
```

By default, the plugin will automatically subscribe to interventions for the `domain_userid` and `domain_sessionid` attribute keys.

You can optionally configure the plugin for additional attribute keys. There are two options:

- `attributeKeyTargets`: dynamically subscribe to attribute key IDs extracted from the tracker's generated events
- `attributeKeyIds`: subscribe to a specific attribute key ID

This example assumes you've defined a `pageview_id` attribute key, based on the `web_page` entity added by default to all web events.

```typescript
subscribeToInterventions({
  endpoint: "000000000000.signals.snowplowanalytics.com", // Signals API endpoint
  attributeKeyTargets: {
    pageview_id: "/co/com.snowplowanalytics.snowplow/web_page/id",
  },
  attributeKeyIds: {
    network_userid: "177234df-d421-412e-ad8d-8bf97515b2807",
  },
});
```

The plugin will automatically disconnect/reconnect whenever the attribute key IDs change. The browser will close the connection on navigation away from the page.

### Intervention events

The plugin generates Snowplow events to track interventions. The events include the intervention payload as a custom entity.

- [`iglu:com.snowplowanalytics.signals/intervention_receive/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_receive/jsonschema/1-0-0): tracked when the plugin receives an intervention
- [`iglu:com.snowplowanalytics.signals/intervention_handle/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_handle/jsonschema/1-0-0): tracked when a custom handler is passed the intervention payload and reports successful handling (does not return an error)
- [`iglu:com.snowplowanalytics.signals/intervention_handle_error/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.signals/intervention_handle_error/jsonschema/1-0-0): tracked when a custom handler is passed the intervention payload and reports failure (throws an error)

## Payload

When delivered, interventions contain the following information:

| Argument                    | Description                                                             | Type      |
| --------------------------- | ----------------------------------------------------------------------- | --------- |
| `intervention_id`           | A unique identifier for this triggered intervention                     | `string`  |
| `name`                      | The unique name/identifier of the intervention                          | `string`  |
| `version`                   | The intervention version                                                | `integer` |
| `attributes`                | An object containing the target's attributes on intervention triggering | `object`  |
| `target_attribute_key`      | An object containing the target attribute key information               | `object`  |
| `target_attribute_key.name` | The name of the target attribute key                                    | `string`  |
| `target_attribute_key.id`   | The value of the target attribute key                                   | `string`  |

Here's an example intervention payload:

```json
{
  "intervention_id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "high_value_cart_abandonment",
  "version": 2,
  "attributes": {
    "cart_value": 250.00,
    "items_in_cart": 3,
    "time_on_checkout_page": 120,
    "previous_purchases": 8,
    "last_activity_timestamp": "2025-09-23T14:30:15Z"
  },
  "target_attribute_key": {
    "name": "domain_userid",
    "id": "218e8926-3858-431d-b2ed-66da03a1cbe5"
  }
}
```

---

# Define interventions with the Signals Python SDK
> Create rule-based interventions with criteria or push direct interventions programmatically. Use the Python SDK to define intervention logic, targeting, and versioning.
> Source: https://docs.snowplow.io/docs/signals/interventions/using-python-sdk/

To use the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) to define interventions, start by [connecting to Signals](/docs/signals/connection/) to create a `Signals` object. You'll need this connection to publish interventions.

There are two ways to define an intervention using the SDK or [Signals API](/docs/signals/connection/#signals-api):

- Rule-based interventions (default)
- Direct interventions

## Rule-based interventions

Rule-based interventions are triggered automatically when their criteria are met. They use `RuleIntervention` objects, and are published to Signals using [`publish()`](/docs/signals/connection/#publishing-and-deleting), similar to other configuration objects.

> **Note:** An intervention is sent only the first time the criteria are met. Read an example of how this works on the [Concepts page](/docs/signals/concepts/#targeting-example).

This is the minimum configuration needed to create an intervention definition:

```python
from snowplow_signals import RuleIntervention, InterventionCriterion

hello_intervention = RuleIntervention(
    name="say_hello",
    owner="me@example.com",
    criteria=InterventionCriterion(
        attribute="example_group:test_attribute",
        operator="is not null",
    ),
)

sp_signals.publish([hello_intervention])
```

Once applied and active, this intervention will trigger the first time Signals processes an event that first sets the `example_group` attribute group's `test_attribute` attribute to a value that is not null (e.g. the first time it gets set).

### Options

The table below lists all available arguments for a `RuleIntervention`:

| Argument                | Description                                                                                                                                                                   | Type                                                                                                              | Required? |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------- |
| `name`                  | The unique name/identifier of the intervention                                                                                                                                | `string`                                                                                                          | ✅         |
| `owner`                 | Email address for the owner of this intervention definition                                                                                                                   | `string`                                                                                                          | ✅         |
| `description`           | A human-readable description of the intervention                                                                                                                              | `string`                                                                                                          | ❌         |
| `version`               | A numeric version for this definition                                                                                                                                         | `integer`                                                                                                         | ❌         |
| `target_attribute_keys` | List of attribute key names to publish this intervention to. If not defined, defaults to the attribute keys associated with any attribute groups you reference in `criteria`. | `string[]`                                                                                                        | ❌         |
| `criteria`              | Tree of `Criterion` expressions to evaluate against attribute key attributes                                                                                                  | One of: `InterventionCriterion`, `InterventionCriteriaAll`, `InterventionCriteriaAny`, `InterventionCriteriaNone` | ✅         |

### Evaluating attributes with criteria

The `criteria` tree defines the conditions that an attribute key's attributes should meet to be eligible for the intervention to trigger.

When a referenced attribute is updated, the updated and previous states are evaluated against the criteria. If the previous state did not meet the conditions but the newly updated state does, the trigger activates and the intervention gets published to the attribute keys that have a value and are defined in the `target_attributes_key` setting. Criteria always refer to the latest published version of the attribute group that contained that attribute.

The simplest `criteria` tree takes an `InterventionCriterion` instance, with possible arguments:

| Argument    | Description                                                              | Type                                                                                           |
| ----------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| `attribute` | The attribute group and attribute name to evaluate, separated by a colon | `string`                                                                                       |
| `operator`  | The operator used for evaluating the attribute                           | `str`, see table below                                                                         |
| `value`     | If required by the `operator`, the comparison value                      | `str`, `int`, `float`, `bool`, `list[str]`, `list[int]`, `list[float]`, `list[bool]` or `None` |

This table shows the available operators:

| Operator      | Description                 | Value is required? |
| ------------- | --------------------------- | ------------------ |
| `=`           | Equals                      | ✅                  |
| `!=`          | Not equals                  | ✅                  |
| `<`           | Less than                   | ✅                  |
| `>`           | Greater than                | ✅                  |
| `<=`          | Less than or equal          | ✅                  |
| `>=`          | Greater than or equal       | ✅                  |
| `like`        | Pattern matching (SQL LIKE) | ✅                  |
| `not like`    | Pattern not matching        | ✅                  |
| `rlike`       | Regex pattern matching      | ✅                  |
| `not rlike`   | Regex not matching          | ✅                  |
| `in`          | Value in list               | ✅ (list)           |
| `not in`      | Value not in list           | ✅ (list)           |
| `is null`     | Value is null/empty         | ❌                  |
| `is not null` | Value exists                | ❌                  |

For more complex conditions, `criteria` also accepts `InterventionCriteriaAll`, `InterventionCriteriaAny`, and `InterventionCriteriaNone`. You can use these classes to combine multiple criteria as lists with the following parameters:

| Argument                   | Description                                                                                                  | Parameters                        |
| -------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------- |
| `InterventionCriteriaAll`  | Logically AND all the inner criteria. Only trigger if all the nested criteria are also true/triggered.       | `all`: list of nested `criteria`  |
| `InterventionCriteriaAny`  | Logically OR all the inner criteria. Only trigger if at least one of the nested criteria are true/triggered. | `any`: list of nested `criteria`  |
| `InterventionCriteriaNone` | Logically NOR all the inner criteria. Only trigger if all of the nested criteria are false/untriggered.      | `none`: list of nested `criteria` |

These support nesting, so you can create complex conditions if required:

```python
from snowplow_signals import (
    RuleIntervention,
    InterventionCriterion,
    InterventionCriteriaAll,
    InterventionCriteriaAny,
    InterventionCriteriaNone,
)

criteria = InterventionCriteriaAll(all=[
    InterventionCriterion(attribute="mygroup:pageview_count", operator=">=", value=3),
    InterventionCriteriaAny(any=[...]),
    InterventionCriteriaNone(none=[...]),
])
```

### Extended example

This example shows how to use criteria:

```python
from snowplow_signals import (
    RuleIntervention,
    InterventionCriterion,
    InterventionCriteriaAll,
    LinkAttributeKey,
)

cart_abandonment = RuleIntervention(
    name="cart_abandonment_discount",
    owner="marketing@company.com",
    description="Offer discount when high-value cart is abandoned",
    criteria=InterventionCriteriaAll(all=[
        InterventionCriterion(
            attribute="shopping:cart_value",
            operator=">",
            value=75.00
        ),
        InterventionCriterion(
            attribute="shopping:minutes_since_last_activity",
            operator=">=",
            value=15
        ),
    ]),
    target_attribute_keys=[
        LinkAttributeKey(name="domain_userid")
    ]  # Target individual users
)

sp_signals.publish([cart_abandonment])
```

It will trigger based on attribute changes within the `shopping` attribute group, but only if all conditions are met.

## Direct interventions

Direct interventions have no criteria, and are not tied to attribute values. They use `InterventionInstance` objects, and are pushed to Signals using `push_intervention`, rather than being published like other configuration objects.

> **Note:** Direct interventions are only available using the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) or [Signals API](/docs/signals/connection/#signals-api).

You can directly push these interventions to any attribute keys. If the intervention is valid, Signals will immediately deliver it to any subscribers for the targeted attribute key IDs.

```python
from snowplow_signals import AttributeKeyIdentifiers, InterventionInstance, Signals

# The specific attribute key IDs to publish the intervention to
targets = AttributeKeyIdentifiers({
  "domain_sessionid": ["8c9104e3-c300-4b20-82f2-93b7fa0b8feb"],
})

sp_signals.push_intervention(
  targets,
  InterventionInstance(
    name="my_intervention",
    version=1,
  )
)
```

## Versioning

Use `version=1` for the first version of an intervention. After publishing, if you want to change the definition in any way, iterate the version number.

---

# Snowplow Signals Documentation
> Snowplow Signals is a real-time personalization engine that computes and acts on behavioral data from your pipeline. It enables in-session stream and historical user data access for personalized experiences, recommendations, and dynamic pricing.
> Source: https://docs.snowplow.io/docs/signals/introduction/

**Give your AI agents the context they need to act intelligently with Snowplow Signals**.

[Get started with Signals](https://try-signals.snowplow.io/)

[Try out Signals for free without a Snowplow account](https://try-signals.snowplow.io/)

[Snowplow for Composable Analytics](/docs/)

[Learn about the Customer Data Infrastructure that's the foundation for Signals](/docs/)

## Explore Signals

Attributes

Calculate user attributes in real time

[Learn about attributes](/docs/signals/concepts/#attribute-groups) [Define attributes](/docs/signals/attributes/attribute-groups/) [Retrieve attributes](/docs/signals/attributes/)

Interventions

Trigger actions automatically

[Learn about interventions](/docs/signals/concepts/#interventions) [Define interventions](/docs/signals/interventions/) [Subscribe to interventions](/docs/signals/interventions/subscribe/)

Signals Python SDK

Use Signals within a notebook

[Connect to Signals](/docs/signals/connection/#signals-python-sdk) [Define attributes](/docs/signals/attributes/using-python-sdk/) [Define interventions](/docs/signals/interventions/using-python-sdk/)

## Get hands on with Signals

> **Tip:** Visit our [Developer Hub](https://snowplow.io/developer-hub/) for demo videos, blog posts, workshops, and more.

[Build a personalized travel agent](/tutorials/signals-personalize-travel/intro/)

[Set up a demo travel site, and integrate with Snowplow Signals to personalize content and chatbot responses based on user behavior](/tutorials/signals-personalize-travel/intro/)

[Score prospects in real time using Signals and ML](/tutorials/signals-ml-prospect-scoring/intro/)

[Try out Signals for live prospect scoring with machine learning](/tutorials/signals-ml-prospect-scoring/intro/)

[Implement real-time interventions in an ecommerce app](/tutorials/signals-interventions/start/)

[Explore Signals interventions in a demo e-shop application](/tutorials/signals-interventions/start/)

[See all tutorials ->](/tutorials/)

---

# Adding extra data: the Subject class in C++ tracker
> Attach user and device information to tracked events using the Subject class. Set user ID, screen resolution, viewport, color depth, timezone, language, user-agent, and IP address for richer behavioral data.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/adding-extra-data-the-subject-class/

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

Create a subject like this and add it to your tracker using the `Snowplow::create_tracker` call:

```cpp
auto subject = make_shared<Subject>();
subject.set_user_id("a-user-id");

auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject);
```

You can also attach custom Subject information to individual events. In this way, you may track events describing different users or devices using the same tracker. Events can be assigned a shared C++ pointer to a Subject instance using the `set_subject` method. The following example shows how to attach a subject instance to a structured event (see [Tracking specific events](/docs/sources/c-tracker/tracking-specific-events/) for more information on tracking events):

```cpp
auto subject = std::make_shared<Subject>();
subject->set_user_id("another-user");

StructuredEvent se("category", "action");
se.set_subject(subject);
```

The Subject class has a set of `set...()` methods to attach extra data relating to the user to all tracked events. The below table maps the setter functions to the event properties that they affect.

| Setter method           | Resulting event property                   |
| ----------------------- | ------------------------------------------ |
| `set_user_id`           | `user_id`                                  |
| `set_screen_resolution` | `dvce_screenwidth` and `dvce_screenheight` |
| `set_viewport`          | `br_viewwidth` and `br_viewheight`         |
| `set_color_depth`       | `br_colordepth`                            |
| `set_timezone`          | `os_timezone`                              |
| `set_language`          | `br_lang`                                  |
| `set_useragent`         | `useragent`                                |
| `set_ip_address`        | `user_ipaddress`                           |

We will discuss each of these in turn below:

## Set user ID with "set\_user\_id"

You can set the user ID to any string:

```cpp
subject->set_user_id( "{{USER ID}}" );
```

Example:

```cpp
subject->set_user_id("alexd");
```

## Set screen resolution with "set\_screen\_resolution"

If your code has access to the device’s screen resolution, then you can pass this in to Snowplow too:

```cpp
subject->set_screen_resolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```cpp
subject->set_screen_resolution(1366, 768);
```

## Set viewport dimensions with "set\_viewport"

If your code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```cpp
subject->set_viewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```cpp
subject->set_viewport(300, 200);
```

## Set color depth with "set\_color\_depth"

If your code has access to the bit depth of the device’s color palette for displaying images, then you can pass this in to Snowplow too:

```cpp
subject->set_color_depth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```cpp
subject->set_color_depth(32);
```

## Set timezone with "set\_timezone"

This method lets you pass a user’s timezone in to Snowplow:

```cpp
subject->set_timezone( {{TIMEZONE}} );
```

The timezone should be a string:

```cpp
subject->set_timezone("Europe/London");
```

## Set the language with "set\_language"

This method lets you pass a user’s language in to Snowplow:

```cpp
subject->set_language( {{LANGUAGE}} );
```

The language should be a string:

```cpp
subject->set_language('en');
```

## Set custom user-agent with "set\_useragent"

To change the [user-agent string](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) sent along with events to identify the application and system, you may set custom useragent using this method:

```cpp
subject->set_useragent( {{USERAGENT}} );
```

The user-agent should be a string:

```cpp
subject->set_useragent("YourApp/5.0 (Macintosh; Intel Mac OS X 10_15_7)");
```

## Set user's IP address with "set\_ip\_address"

To set the user's IP address, you may use this method:

```cpp
subject->set_ip_address( {{IP_ADDRESS}} );
```

The IP address should be a string:

```cpp
subject->set_ip_address("169.254.0.2");
```

If the IP address is not set, the events will be assigned the IP address from the HTTP request by the Collector.

---

# track client sessions with the C++ tracker
> Enable client session tracking in the C++ tracker to track user sessions with foreground and background timeouts. Attach session context entities including session ID, index, and first event timestamp.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/client-sessions/

You can optionally decide to include Client Sessionization. This object will keep track of your users sessions and can be configured to timeout after a certain amount of inactivity.

Activity is determined by how often events are sent with the Tracker – so you will need to send events to keep the current session active. Sessions are updated when new events are tracked. There are two timeouts that are checked: foreground and background timeout. Depending on whether the app is in foreground or backgroud, the relevant timeout is used to compare the time difference since previous event and, in case it surpasses the timeout, a new session is started.

In [Initialisation](/docs/sources/c-tracker/initialisation/), we discussed how to create a tracker with optional session tracking enabled and configurable foreground and background timeouts. If you enabled session tracking when creating the tracker, you can access a `ClientSession` instance using the tracker:

```cpp
auto client_session = Snowplow::get_default_tracker()->get_client_session();
```

To set the background/foreground state you will need to detect this and then set this on the `ClientSession` object like so:

```cpp
client_session.set_is_background(true || false);
```

When client sessions are used, the [`client_session`](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) context entity is added to all tracked event. This entity consists of the following properties:

| Attribute             | Description                                                                                                   | Required? |
| --------------------- | ------------------------------------------------------------------------------------------------------------- | --------- |
| `userId`              | An identifier for the user of the session.                                                                    | Yes       |
| `sessionId`           | An identifier (UUID) for the session.                                                                         | Yes       |
| `sessionIndex`        | The index of the current session for this user.                                                               | Yes       |
| `eventIndex`          | Optional index of the current event in the session. Signifies the order of events in which they were tracked. | No        |
| `previousSessionId`   | The previous session identifier (UUID) for this user.                                                         | No        |
| `storageMechanism`    | The mechanism that the session information has been stored on the device.                                     | Yes       |
| `firstEventId`        | The optional identifier (UUID) of the first event id for this session.                                        | No        |
| `firstEventTimestamp` | Optional date-time timestamp of when the first event in the session was tracked.                              | No        |

## Session store

The session store is used to persist the currently active session.

The tracker provides the `SqliteStorage` class that can be used as the session store. However, you may also provide a custom session store implementation. To do so, define a class that inherits from the `SessionStore` struct:

```cpp
struct SessionStore {
  virtual unique_ptr<json> get_session() = 0;
  virtual void set_session(const json &session_data) = 0;
  virtual void delete_session() = 0;
};
```

The `SessionStore` struct defines functions to retrieve, set, and delete the current session. It represents sessions in their JSON format. The three operations have the following behavior:

| Function         | Description                                                                    |
| ---------------- | ------------------------------------------------------------------------------ |
| `get_session`    | Return a unique pointer to the current session or nullptr if it doesn't exist. |
| `set_session`    | Persist the current session.                                                   |
| `delete_session` | Remove and reset the current session.                                          |

---

# Configure emitters in the C++ tracker
> Configure emitters to send events to your collector with customizable batch sizes, byte limits, and retry behavior. Use SQLite event store or custom EventStore implementation with request callbacks.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/emitters/

Emitters are responsible for sending events to the collector. Each tracker is given a single emitter. Once the emitter receives an event from the Tracker a few things start to happen:

- The event is added to a local SQLite3 database or custom event store (blocking execution).
- A long running daemon thread is started which will continue to send events as long as they can be found in the database (asynchronous).
- The emitter loop will grab a range of events from the database up until the `batch_size` passed to it as configuration.
- The emitter will send all of these events as determined by the Request, Protocol and ByteLimits.
  - Each request is sent in its thread.
- Once sent, it will process the results of all the requests sent and will remove all successfully sent events from the database. If the request failed, the events will be retried after a retry delay (see below).

In [Initialisation](/docs/sources/c-tracker/initialisation/), we discussed how to create a tracker with an emitter configured using `EmitterConfiguration` or by instantiating an `Emitter` instance directly. Both of these options provide the same configuration functionality (e.g., storage options, byte limits, setting custom HTTP clients) that were discussed previously. This page will go into more detail on some of the configurable emitter properties.

## Event store

The event store is used to store an event queue with events scheduled to be sent. Events are added to the event store when they are tracked and removed when they are successfuly emitted or when emitting fails without any scheduled retries.

The tracker provides the `SqliteStorage` class that can be used as the event store. It uses SQLite to store the event queue. By default it will create the required files wherever the application is being run from.

You may also provide a custom event store implementation. To do so, define a class that inherits from the `EventStore` struct:

```cpp
struct EventStore {
  virtual void add_event(const Payload &payload) = 0;
  virtual void get_event_rows_batch(list<EventRow> *event_list, int number_to_get) = 0;
  virtual void delete_event_rows_with_ids(const list<int> &id_list) = 0;
};
```

The `EventStore` struct defines functions to insert, retrieve, and remove events from the queue. Events are represented using their `Payload` instance which is persisted in a `EventRow` wrapper that also assigns IDs to each stored event (these event row IDs are different from event IDs used in the event payloads). There are three supported operations:

| Function                     | Description                                                 |
| ---------------------------- | ----------------------------------------------------------- |
| `add_event`                  | Insert event payload into event queue.                      |
| `get_event_rows_batch`       | Retrieve event rows from event queue up to the given limit. |
| `delete_event_rows_with_ids` | Remove event rows with the given event row IDs.             |

## Emitter request callback

The emitter enables you to set a callback function to be called after events are attempted to be sent to the Collector. This callback is fired after HTTP requests are made and you can subscribe for specific emit statuses. The following statuses can be subscribed to:

- `EmitStatus::SUCCESS` – events were successfuly sent to the Collector.
- `EmitStatus::FAILED_WILL_RETRY` – events failed to be sent to the Collector but will be retried later.
- `EmitStatus::FAILED_WONT_RETRY` – events failed to be sent to the Collector and won't be retried later (they will be dropped).

The callback is given two arguments – list of event IDs, and their emit status. You can only set one callback at once but you can subscribe to multiple emit statuses using binary operations. The following example shows how to set a callback that is called for all three emit statuses and prints to standard output.

```cpp
emitter_configuration.set_request_callback(
    [](list<string> event_ids, EmitStatus emit_status) {
      switch (emit_status) {
      case EmitStatus::SUCCESS:
        printf("Successfuly sent %lu events.\n", event_ids.size());
        break;
      case EmitStatus::FAILED_WILL_RETRY:
        printf("Failed to send %lu events, but will retry.\n", event_ids.size());
        break;
      case EmitStatus::FAILED_WONT_RETRY:
        printf("Failed to send %lu events and won't retry.\n", event_ids.size());
        break;
      }
    },
    EmitStatus::SUCCESS | EmitStatus::FAILED_WILL_RETRY | EmitStatus::FAILED_WONT_RETRY);
```

The callback is executed in a new thread. The `set_request_callback` function can't be called when the Emitter is running.

## HTTP request retry behavior

The Emitter has a retry functionality that repeatedly sends the same events to the Collector in case HTTP requests fail to get through. Requests are retried in case the connection to the Collector fails to be estabilished. They are also retried for all 3xx and 5xx HTTP status codes in server response and most 4xx status codes with the following exceptions – 400, 401, 403, 410, and 422. These status codes signal a rejection of the events being sent to the Collector and, therefore, the events in the requests are dropped and not sent again.

The `Emitter` and `EmitterConfiguration` provide an option to set custom retry behavior for 3xx, 4xx and 5xx HTTP status codes. You can call the `set_retry_for_status_code` function on the `Emitter` or `EmitterConfiguration` instance to define whether to retry or not for a given HTTP status code. Here is an example that overrides the default behavior and enables retries on 422 status code:

```cpp
emitter_configuration.set_retry_for_status_code(422, true);
```

## Request retry delay (back-off)

Failed requests are retried with exponentially increasing delays between subsequent retry requests. This ensures that the Collector is not overwhelmed with too many requests and also saves resources on the client by reducing the number of requests during failures.

The retry delay calculation is an exponential function with multiplicative factor 2. To prevent spikes of traffic, small amount of randomness is added to the delay (at most 10% of the total value). Finally, it is limited to be no larger than around 2 minutes. The following is a sample sequence of the retry delays given 5 failed requests: 0.101s, 0.198s, 0.404s, 0.791s, 1.603s.

## Manual flushing

You may want to force an emitter to send all events in its buffer, even if the buffer is not full. The Tracker class has a `flush()` method which flushes its emitter.

Example:

```cpp
tracker->flush();
```

## Using a custom HTTP Client

You may optionally configure the HTTP client to be used to make HTTP requests to the collector. This is done by passing a unique pointer to a class inheriting from `HttpClient` that the Emitter will take ownership of. If not configured, the Emitter will use the built-in `HttpClientWindows` on Windows, `HttpClientApple` on Apple operating systems, and `HttpClientCurl` on other Unix systems.

---

# C++ tracker SDK
> The Snowplow C++ Tracker enables event tracking from C++ applications, games, and servers. Create subjects, emitters, and trackers to send behavioral events to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/

The [Snowplow C++ Tracker](https://github.com/snowplow/snowplow-cpp-tracker) allows you to track Snowplow events from your C++ apps, games and servers.

There are three basic types of object you will create when using the Snowplow C++ Tracker: subjects, emitters, and trackers.

- A subject represents a user whose events are tracked.
- A tracker constructs events and sends them to an emitter.
- The emitter then sends the event to the endpoint you configure; a valid Snowplow Collector.

---

# Initialize the C++ tracker
> Create a C++ tracker using the Snowplow interface with default or custom configuration. Configure tracker, network, emitter, and session settings using configuration objects or manage Tracker, Emitter, and ClientSession directly.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/initialisation/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

## Import the library

Import the C++ Tracker library and use the `snowplow` namespace like so:

```cpp
#include <snowplow/snowplow.hpp>

using namespace snowplow;
```

That’s it – you are now ready to initialize a tracker instance that you will use to track events. There are 3 ways to do that:

1. Create a tracker with default configuration using the `Snowplow` interface.
2. Create a tracker with custom configuration using the `Snowplow` interface.
3. Create and manage `Tracker`, `Emitter`, and `ClientSession` directly.

## Option 1: Creating a tracker with default configuration using the "Snowplow" interface

The `Snowplow` class provides static methods that let you easily create a new tracker. It can be as simple as:

```cpp
auto tracker = Snowplow::create_tracker(
    "ns", // tracker namespace used to identify the tracker
    "https://com.acme.collector", // Snowplow collector URL
    POST, // HTTP method used to send events to the collector
    "events.db" // Relative path to an SQLite database used for event queue and session tracking
);
```

Optionally, you may choose to attach a `Subject` instance with information about the user and device (see the next page on [Adding data](https://file+.vscode-resource.vscode-cdn.net/Users/matus/Projects/Snowplow/snowplow-cpp-tracker/docs/03-adding-data.md) to learn more about the `Subject`):

```cpp
auto subject = std::make_shared<Subject>() // initialize a C++ shared pointer for the Subject
subject->set_user_id("a-user-id");

auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject);
```

Finally, client session tracking will be automatically enabled and session information will be attached to all events (see documentation on [Client sessions](https://file+.vscode-resource.vscode-cdn.net/Users/matus/Projects/Snowplow/snowplow-cpp-tracker/docs/06-client-sessions.md) for more details). If you wish to disable session tracking, pass `false` as the last argument:

```cpp
auto tracker = Snowplow::create_tracker("ns", "https://com.acme.collector", POST, "events.db", subject, false);
```

After you create a tracker, you can access it from anywhere using it's namespace:

```cpp
auto tracker = Snowplow::get_tracker("ns");
```

You can also access the default tracker using `Snowplow::get_default_tracker()`. The default tracker is the first initialized tracker (unless it is removed). If you have multiple trackers, you can choose the default tracker by calling `Snowplow::set_default_tracker(tracker2)`.

To remove reference of previously initialized tracker from the Snowplow interface (the tracker will be deleted once all remaininig references to it are deleted):

```cpp
Snowplow::remove_tracker(tracker);
```

## Option 2: Creating a tracker with custom configuration using the "Snowplow" interface

There are a number of settings that you may want to customize when creating a tracker instance. You can make use of configuration objects to provide custom configuration for the tracker:

```cpp
TrackerConfiguration tracker_config(
    "namespace", // tracker namespace
    "app-id", // application ID
    mob // platform that the tracker runs on
);
tracker_config.set_desktop_context(false); // do not attach a desktop context entity to events (on by default)

NetworkConfiguration network_config("https://com.acme.collector", POST); // collector URL and method

EmitterConfiguration emitter_config("sp.db"); // event queue DB path if SQLite or custom `EventStore` instance
emitter_config.set_batch_size(500); // maximum number of events to send at a time

SessionConfiguration session_config(
    "sp.db", // session DB path if SQLite (usually same as event queue) or custom `SessionStore` instance
    5000, // foreground timeout in ms
    5000 // background timeout in ms
);

auto tracker = Snowplow::create_tracker(tracker_config, network_config, emitter_config, session_config);
```

### Tracker configuration using "TrackerConfiguration"

`TrackerConfiguration` contains settings to identify and configure the tracker. It's constructor takes 3 attributes:

| Constructor attribute | Description                                                                                          | Default |
| --------------------- | ---------------------------------------------------------------------------------------------------- | ------- |
| `name_space`          | Tracker namespace to identify the tracker and also attach as a property to tracked events.           | None    |
| `app_id`              | Application ID.                                                                                      | ""      |
| `platform`            | Enum of the platform the Tracker is running on, can be one of: web, mob, pc, app, srv, tv, cnsl, iot | srv     |

It further provides 2 optional setter functions:

| Setter                | Description                                                                                                             | Default |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------- |
| `set_use_base64`      | Whether to use base64 encoding in events.                                                                               | true    |
| `set_desktop_context` | Whether to add a desktop\_context, which gathers information about the device the tracker is running on, to each event. | true    |

### Network configuration using "NetworkConfiguration"

`NetworkConfiguration` has only two properties set in it's constructor to configure the Snowplow collector:

| Constructor attribute | Description                                                                                                                                                                   | Default                                                                                   |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `collector_url`       | Full URL of the Snowplow collector including the protocol (or defaults to HTTPS if protocol not present).                                                                     | None                                                                                      |
| `method`              | HTTP method to use when sending events to collector – GET or POST.                                                                                                            | POST                                                                                      |
| `curl_cookie_file`    | Path to a file where to store cookies in case http\_client is nullptr and the CURL HTTP client is used – only relevant under Linux (CURL is not used under Windows and macOS) | In-memory cookie storage with CURL on Linux, platform native storage on Windows and macOS |

Additionally, it provides the following setter functions:

| Setter            | Description                                                                | Default                           |
| ----------------- | -------------------------------------------------------------------------- | --------------------------------- |
| `set_http_client` | Unique pointer to a custom HTTP client to send GET and POST requests with. | Platform-specific implementation. |

### Emitter configuration using "EmitterConfiguration"

`EmitterConfiguration` brings additional settings for the constructor. It provides two constructors that accept the event store either as a path to the SQLite database or a custom `EventStore` object. The following configurations are identical:

```cpp
EmitterConfiguration config1("sp.db");

auto storage = std::make_shared<SqliteStorage>("sp.db");
EmitterConfiguration config2(storage); // you can also pass a custom `EventStore`
```

Additionally, it provides the following setter functions:

| Setter                             | Description                                                                                                                                  | Default     |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| `set_batch_size`                   | The maximum amount of events to send at a time.                                                                                              | 250 events  |
| `set_byte_limit_get`               | The byte limit when sending a GET request.                                                                                                   | 40000 bytes |
| `set_byte_limit_post`              | The byte limit when sending a POST request.                                                                                                  | 40000 bytes |
| `set_request_callback`             | Set a callback to call after emit requests are made with the resulting emit status (see page about Emitter for more info).                   | None        |
| `set_custom_retry_for_status_code` | Set a custom retry rule for when the HTTP status code is received in emit response from Collector (see page about Emitter for more details). | None        |

### Session configuration using "SessionConfiguration"

`SessionConfiguration` is an optional attribute in `create_tracker`. If passed, session tracking will be enabled with the given configuration. If not passed, session tracking will not be enabled. The constructor takes 3 attributes:

| Constructor attribute        | Description                                                                                                                                                                         | Default    |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `session_store` or `db_name` | You may either pass a path to an SQLite database to be used for session storage or a custom implementation of `SessionStore` similar as for `EventStore` in `EmitterConfiguration`. | None       |
| `foreground_timeout`         | Timeout in ms for updating the session when the app is in background.                                                                                                               | 30 minutes |
| `background_timeout`         | Timeout in ms for updating the session when the app is in foreground.                                                                                                               | 30 minutes |

## Option 3: Managing "Tracker", "Emitter", and "ClientSession" directly

The third option to initialise a new tracker is to instantiate it and the related components directly. This option is suitable in case you want supply a custom emitter or client session implementation. If you don't want to do that, we recommend using Option 2 which gives you the same configuration options with a simpler API.

The following example takes you through the steps needed to initialize a storage, emitter, subject, client session, and tracker:

```cpp
#include <snowplow/snowplow.hpp>

// 1. create storage for event queue and session information
auto storage = std::make_shared<SqliteStorage>("sp.db");
// 2. create an emitter for sending event to Snowplow collector
auto emitter = std::make_shared<Emitter>(storage, "com.acme.collector");
// 3. create a subject with user and device information
auto subject = std::make_shared<Subject>();
subject->set_user_id("a-user-id");
// 4. optionally create a client_session for session tracking
auto client_session = std::make_shared<ClientSession>(storage, 5000, 5000);
// 5. finally, create the tracker instance
auto tracker = std::make_shared<Tracker>(emitter, subject, client_session, "pc", "app_id", "ns");
```

Once you initialize the tracker, you can optionally register it with the Snowplow interface:

```cpp
Snowplow::register_tracker(tracker);
```

This will make accessible from anywhere using `Snowplow::get_tracker("ns")`.

The following will explain the individual components shown in the above code sample in more detail.

### Storage

The storage has two functions in the example above – it is used by the `emitter` to persist an event queue with events to be sent, and it is used by the `client_session` to persist the current session. The tracker provides an SQL storage (`SqliteStorage`) implementation, but you may introduce your own storage as described in ["Emitters"](/docs/sources/c-tracker/emitters/) and ["Client Sessions"](/docs/sources/c-tracker/client-sessions/).

### Emitter

Emitter is a required component responsible for sending tracked events to the collector.

Accepts an argument of an Emitter instance pointer; if the object is `NULL` will throw an exception. See Emitters for more on emitter configuration.

| **Argument Name**  | **Description**                                                                                                                                                               | **Required?** | **Default**                                                                               |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------- |
| `event_store`      | Defines the database to use for event queue                                                                                                                                   | Yes           | –                                                                                         |
| `uri`              | The collector URI (excluding protocol) to send events to                                                                                                                      | Yes           | –                                                                                         |
| `method`           | The request type to use (GET or POST)                                                                                                                                         | No            | POST                                                                                      |
| `protocol`         | The protocol to use (HTTP or HTTPS)                                                                                                                                           | No            | HTTPS                                                                                     |
| `batch_size`       | The maximum amount of events to send at a time                                                                                                                                | No            | 250 events                                                                                |
| `byte_limit_post`  | The byte limit when sending a POST request                                                                                                                                    | No            | 40000 bytes                                                                               |
| `byte_limit_get`   | The byte limit when sending a GET request                                                                                                                                     | No            | 40000 bytes                                                                               |
| `http_client`      | Unique pointer to a custom HTTP client to send GET and POST requests with                                                                                                     | No            | Platform-specific implementation.                                                         |
| `curl_cookie_file` | Path to a file where to store cookies in case http\_client is nullptr and the CURL HTTP client is used – only relevant under Linux (CURL is not used under Windows and macOS) | No            | In-memory cookie storage with CURL on Linux, platform native storage on Windows and macOS |

### Subject

The user which the Tracker will track. Accepts an argument of a Subject instance pointer.

You don’t need to set this during Tracker construction; you can use the `tracker.set_subject(...)` method afterwards. In fact, you don’t need to create a subject at all. If you don’t, though, your events won’t contain user-specific data such as timezone and language.

### Client session

The client sessions which the Tracker will attach to each event. Accepts an argument of a ClientSession instance pointer.

Adds the ability to attach a ClientSession context to each event that leaves the Tracker. This object will persistently store information about the sessions that have occurred for the life of the application – unless the database is destroyed.

### Tracker

The tracker instance lets you track events. You should maintain a reference to the initialized tracker during the app lifetime.

| **Argument Name** | **Description**                                                                                                                           | **Required?** | **Default** |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------- |
| `emitter`         | The emitter to which events are sent                                                                                                      | Yes           | –           |
| `subject`         | The user being tracked                                                                                                                    | No            | nullptr     |
| `client_session`  | Client session recording                                                                                                                  | No            | nullptr     |
| `platform`        | The platform the Tracker is running on                                                                                                    | No            | "srv"       |
| `app_id`          | The application ID                                                                                                                        | No            | ""          |
| `name_space`      | The namespace of the tracker instance used to identify the tracker.                                                                       | No            | ""          |
| `use_base64`      | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64). Defaults to true to ensure that no data is lost or corrupted. | No            | true        |
| `desktop_context` | Whether to add a `desktop_context` to events                                                                                              | No            | true        |

The `desktop_context` gathers extra information about the device it is running on and sends it along with every event that is made by the Tracker.

An example of the data in this context:

```json
{
    "deviceManufacturer": "Apple Inc.",
    "deviceModel": "MacPro3,1",
    "deviceProcessorCount": 8,
    "osIs64Bit": true,
    "osServicePack": "",
    "osType": "macOS",
    "osVersion": "10.11.2"
}
```

For more information the raw JsonSchema can be found [here](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.snowplow/desktop_context/jsonschema/1-0-0).

---

# Install the C++ tracker
> Install the C++ tracker using CMake, FetchContent, or by copying source files. Supports macOS, Windows, and Linux with C++ 11 minimum. Configure dependencies including SQLite3, CURL, and LibUUID.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/setup/

The Snowplow C++ Tracker has been built and tested using C++ 11 as a minimum.

Supported operating systems:

- macOS
- Windows
- Linux

## Installing in your project

There are three ways to install the tracker in your app:

1. By adding the project into your `CMakeLists.txt` as a subdirectory.
2. By installing the project and importing it into your app using CMake's `find_package` command.
3. By copying source files inside the `include` folder into your codebase.

### As an imported target in your CMake project

First, build and install the project. Make sure the project uses the external JSON libraries (`SNOWPLOW_USE_EXTERNAL_JSON=ON`). If you're building a static library (`SNOWPLOW_USE_EXTERNAL_SQLITE=ON`) you also need to use SQLite3 as an external library (`SNOWPLOW_USE_EXTERNAL_SQLITE=ON`).

If you have `SQLite3`, `CURL` or `LibUUID` available as system libraries but you need to use them from a different package (e.g. from Conan) you need to set `CMAKE_FIND_PACKAGE_PREFER_CONFIG=ON` to prevent linking to the system libraries.

```cmake
cmake [...] -DCMAKE_INSTALL_PREFIX=[...]
    -DSNOWPLOW_USE_EXTERNAL_JSON=ON -DSNOWPLOW_USE_EXTERNAL_SQLITE=ON \
    -DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON \
    -DSNOWPLOW_BUILD_TESTS=0 -DSNOWPLOW_BUILD_EXAMPLE=0 -DSNOWPLOW_BUILD_PERFORMANCE=0
```

After building and installing the project you can use `find_package` to import it into your `CMakeLists.txt`:

```cmake
find_package(snowplow REQUIRED CONFIG)
...
target_link_libraries(your-target snowplow::snowplow)
```

Make sure your project finds the same dependencies what was visible for Snowplow when you were building and installing it. For example, if you have both system and local SQlite3 installations and `CMAKE_FIND_PACKAGE_PREFER_CONFIG` was `ON` for Snowplow but `OFF` for your project, Snowplow will be built with the local SQLite3 while during `find_package(snowplow)` in your project it will find the system one.

### As a subdirectory in your CMake project

Cmake version 3.15 or greater is required. You may add the library to your project target (`your-target`) using `FetchContent` like so:

```cmake
include(FetchContent)
FetchContent_Declare(
  snowplow
  GIT_REPOSITORY https://github.com/snowplow/snowplow-cpp-tracker
  GIT_TAG        2.0.0
)
FetchContent_MakeAvailable(snowplow)
target_link_libraries(your-target snowplow)
```

### Copying files to your project

Download the most recent release from the [releases section](https://github.com/snowplow/snowplow-cpp-tracker/releases). Everything in the `include` folder will need to be included in your application.

The project has two dependencies that need to be included in your project: [nlohmann/json](https://github.com/nlohmann/json) and [the amalgamated version of sqlite3](https://www.sqlite.org/download.html). You will need to update the include paths in headers `include/snowplow/thirdparty/json.hpp` and `include/snowplow/thirdparty/sqlite3.hpp`.

### Additional requirements under Linux

Additionally, under Linux, the following libraries need to be installed:

- curl (using `apt install libcurl4-openssl-dev` on Ubuntu)
- uuid (using `apt install uuid-dev` on Ubuntu)

---

# Track specific events with the C++ tracker
> Track custom behavioral events using SelfDescribingEvent, StructuredEvent, ScreenViewEvent, and TimingEvent. Add custom context entities and true timestamps to events sent to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/tracking-specific-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in event classes to help you track different kinds of events. When instantiated, their objects can be passed to the `tracker.track()` methods to send events to the Snowplow collector. The event classes range from single purpose ones, such as `ScreenViewEvent`, to the more complex but flexible `SelfDescribingEvent`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribingEvent` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

The tracker provides the following event classes for tracking events out of the box:

| **Function**          | **Description**                                         |
| --------------------- | ------------------------------------------------------- |
| `StructuredEvent`     | Tracks a Snowplow custom structured event               |
| `SelfDescribingEvent` | Tracks a Snowplow custom unstructured event             |
| `ScreenViewEvent`     | Tracks the user viewing a screen within the application |
| `TimingEvent`         | Tracks a timing event                                   |

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

## Custom event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event's entities together form the event context. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a screen view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Each tracking type provides a `set_context()` method which accepts a vector of context entities:

```cpp
vector<SelfDescribingJson> context;
```

If a visitor arrives on a page advertising a movie, the dictionary for a single custom context entity in the list might look like this:

```cpp
vector<SelfDescribingJson> context;
SelfDescribingJson sdj(
  "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
  "{\"movie_name\":\"Solaris\",\"poster_country\":\"JP\"}"_json
);
context.push_back(sdj);
```

This is how to fire a structured event with the above custom context entity:

```cpp
StructuredEvent se("category", "action");
se.set_context(context);

Snowplow::get_default_tracker()->track(se);
```

Note that even though there is only one custom context entity attached to the event, it still needs to be placed in a list.

**INFO**: We use the excellent json library from Github community member [Niels (nlohmann)](https://github.com/nlohmann) for all JSON parsing. For more information on using this library please visit the [Technical information on Github](https://github.com/nlohmann/json).

## Optional true-timestamp argument

Each event class supports an optional true-timestamp argument; this allows you to provide the true-timestamp attached to this event to help with the timing of events in multiple timezones. The timestamp should be in milliseconds since the Unix epoch.

Here is an example tracking a structured event and supplying the optional true-timestamp argument:

```cpp
StructuredEvent se("category", "action");

// As it is optional you will need to pass the address for this value
unsigned long long true_tstamp = "1368725287000";
se.set_true_timestamp(&true_tstamp);

Snowplow::get_default_tracker()->track(se);
```

## Track SelfDescribing/Unstructured events with "SelfDescribingEvent"

Use the `SelfDescribingEvent` type to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

`SelfDescribingEvent` provides the following properties:

| **Argument** | **Description**             | **Required?** | **Validation**     |
| ------------ | --------------------------- | ------------- | ------------------ |
| `event`      | The properties of the event | Yes           | SelfDescribingJson |

Example:

```cpp
// Create a JSON containing your data
json data = "{\"level\":5,\"saveId\":\"ju302\",\"hardMode\":true}"_json;

// Create a new SelfDescribingJson
SelfDescribingJson sdj("iglu:com.example_company/save-game/jsonschema/1-0-2", data);

SelfDescribingEvent sde(sdj);
Snowplow::get_default_tracker()->track(sde);
```

For more on JSON schema, refer to [this page](/docs/fundamentals/schemas/).

## Track screen views with "ScreenViewEvent"

Use the `ScreenViewEvent` type to track a user viewing a screen (or equivalent) within your app. This is the page view equivalent for apps that are not webpages.

| **Argument** | **Description**                     | **Required?** | **Type** |
| ------------ | ----------------------------------- | ------------- | -------- |
| `name`       | Human-readable name for this screen | No            | \*string |
| `id`         | Unique identifier for this screen   | No            | \*string |

Although name and id are not individually required, at least one must be provided or the event will fail validation and subsequently throw an exception.

The event uses the following Iglu schema: [iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

Example:

```cpp
string name = "Screen ID - 5asd56";

ScreenViewEvent sve;
sve.name = &name;

Snowplow::get_default_tracker()->track(sve);
```

## Track structured events with `StructuredEvent`

Use the `StructuredEvent` type to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required). This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| **Argument** | **Description**                                                  | **Required?** | **Validation** |
| ------------ | ---------------------------------------------------------------- | ------------- | -------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | string         |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | string         |
| `label`      | A string to provide additional dimensions to the event data      | No            | \*string       |
| `property`   | A string describing the object or the action performed on it     | No            | \*string       |
| `value`      | A value to provide numerical data about the event                | No            | \*double       |

Example:

```cpp
StructuredEvent se("shop", "add-to-basket"); // constructor takes category and action
se.property = "pcs";
se.value = 25.6;

Snowplow::get_default_tracker()->track(se);
```

## Track timing events with "TimingEvent"

Use the `TimingEvent` type to track user timing events such as how long resources take to load.

Its properties are as follows:

| **Argument** | **Description**           | **Required?** | **Validation** |
| ------------ | ------------------------- | ------------- | -------------- |
| `category`   | The category of the event | Yes           | \*string       |
| `variable`   | The variable of the event | Yes           | \*string       |
| `timing`     | The timing of the event   | Yes           | \*int64        |
| `label`      | The label of the event    | No            | \*string       |

The event uses the following Iglu schema: [iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0).

Example:

```cpp
TimingEvent te("category", "variable", 123);
Snowplow::get_default_tracker()->track(te);
```

---

# Upgrade to newer versions of the C++ tracker
> Migration instructions for upgrading to newer C++ tracker versions. Learn about breaking changes in v2.0.0, v1.0.0, v0.3.0, and v0.2.0 including CMake updates and API changes.
> Source: https://docs.snowplow.io/docs/sources/c-tracker/upgrading-to-newer-versions/

This page gives instructions for upgrading to newer versions of the C++ tracker.

## Upgrading to v2.0.0

There have been the following breaking changes:

- CMake minimum version changed 3.14 -> 3.15
- libcurl has been removed as a dependency of Snowplow on macOS and the CURL HTTP client is no longer available on macOS. If a project that is using Snowplow as a subdirectory relies on getting libcurl through Snowplow then it will fail now.
- Also, sqlite3, libcurl, libuuid are now private dependencies. If a project relies on getting the sqlite3/libcurl/libuuid include paths through the Snowplow tracker library, then it will fail now.

## Upgrading to v1.0.0

The package added support for the cmake build system. You may make use of this and import the package in your cmake build configuration. Importing source code by copying files is still supported. However, please note that the code was moved from the `src` folder to the `include` folder and you will need to import the sqlite3 and nlohmann/json dependencies on your own and update their location in `include/snowplow/thirdparty/sqlite3.hpp` and `include/snowplow/thirdparty/json.hpp` files. See instructions on [the Setup page](/docs/sources/c-tracker/setup/) for more information.

This version changed how the `Tracker` and `Emitter` instances are initialized and introduced the `Snowplow::create_tracker()` API. Please refer to [the page about](/docs/sources/c-tracker/initialisation/) initialization to learn more about the new API. The `Tracker::init()` calls should be replaced with `Snowplow::create_tracker()` and `Tracker::instance()` with `Snowplow::get_tracker(namespace)` or `Snowplow::get_default_tracker()`.

Also note that the default protocol has changed from HTTP to HTTPS.

## Upgrading to v0.3.0

There are a few breaking changes in this release. Please make sure to:

1. Remove the `Tracker::` prefix when referring to event types, e.g., use `ScreenViewEvent()` instead of `Tracker::ScreenViewEvent()`.
2. Use the common `tracker->track(event)` function to track events (instead of `tracker->track_self_describing_event(event)`, `tracker->track_screen_view(event)`, ...).
3. Use the `event.set_context(context)` and `event.set_true_timestamp(tt)` setters instead of accessing `event.contexts` and `event.true_timestamp` directly.
4. Event IDs are no longer accessible from the event objects but are returned from the `tracker->track(event)` function.
5. `Emitter` and `ClientSession` no longer accept database path string for storage but require an instance of `SqliteStorage` (or other storage implementation, see the docs).
6. Use `#include "snowplow.hpp"` to import all public APIs of the tracker instead of including individual files.

## Upgrading to v0.2.0

This version adds a `snowplow` namespace to all tracker components. You will need to import specific or all types from the namespace:

```cpp
using namespace snowplow;
```

The constructor for `ClientSession` also changed, dropping the last argument for `check_interval` that is no longer necessary. Simply remove the last argument when creating an instance of `ClientSession`.

---

# First-party tracking
> Configure a custom collector domain matching your primary domain to enable first-party cookies and bypass browser tracking limitations.
> Source: https://docs.snowplow.io/docs/sources/first-party-tracking/

This feature lets you configure a custom collector domain to match your primary domain, enabling first-party cookies.

When your collector domain (e.g. `collector.snwplow.net`) does not match your primary domain (e.g. `flowershop.ai`), web browsers can impose limitations on the collected data. For example, cookies set by the collector might be only kept for 7 days.

With first-party tracking, you can configure a custom collector domain (e.g. `c.flowershop.ai`) to match your primary domain (e.g. `flowershop.ai`), sidestepping these limitations.

Note that in light of the [latest ITP restrictions](https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense), you will also need to [use a Cookie Lifetime Extension service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/) to fully persist the cookies.

> **Info:** Before starting, ensure you can access and edit the configuration of your hosting/DNS provider. This is necessary to complete the implementation.

> **Info:** The flow described below might differ depending on the version of Snowplow you are using.

## Selecting the domain names

When configuring first-party tracking, you first need to set a collector domain where the events will be sent to by the web browser. This must be a subdomain of your primary domain. For example, for `flowershop.ai` it could be `t.flowershop.ai` or `c.app.flowershop.ai`.

> **Tip:** Avoid words such as “track”, “collect” or “analytics”, as ad blockers may prevent data from being sent to domains containing these words.

You also need to select a cookie domain. This will determine on which websites the cookie will be available. For example, if your cookie domain is `flowershop.ai`, the cookie will be available on `flowershop.ai`, as well as any subdomains.

> **Note:** For Snowplow collector to be able to set the cookie, the cookie domain must be a parent domain of the collector domain.

Here are a few examples:

| Collector domain      | Cookie domain       | Cookie available on                                |
| --------------------- | ------------------- | -------------------------------------------------- |
| `t.flowershop.ai`     | `flowershop.ai`     | `flowershop.ai`, `app.flowershop.ai`, etc          |
| `c.app.flowershop.ai` | `app.flowershop.ai` | `app.flowershop.ai`, `beta.app.flowershop.ai`, etc |

When you enter a collector domain, we will automatically suggest a cookie domain. You can edit it to suit your use case.

**How to choose the cookie domain**

Setting the cookie domain to your primary domain has the benefit of allowing you to track users across any subdomains. For example, a cookie domain of `flowershop.ai` would work across all the following subdomains (and would allow you to track the same user ids):

- `docs.flowershop.ai`
- `app.flowershop.ai`
- `test.app.flowershop.ai`
- `flowershop.ai`

In some cases, you may wish to separate the tracking of user behavior for different subdomains.

For example, if you own both `gardening.primary-domain.co.uk` and `insurance.primary-domain.co.uk` and only want to track users on `gardening.primary-domain.co.uk`, you can select that as your cookie domain and `c.gardening.primary-domain.co.uk` as your collector domain.

![enter\_domain](/assets/images/Screenshot_enter_domain-f90bda00daea98a474af1e6bb6a0881d.png)

## Configuring DNS records

In the next step, Snowplow will generate the required DNS records. This may take several minutes.

When the records are ready, you will receive a confirmation by email.

![create\_dns\_records](/assets/images/Screenshot_create_dns_records-5c11e37bafecca032938c1d40f8025d4.png)

Once the DNS records are available, copy them into your domain provider.

The set of DNS records will contain a special record that allows Snowplow to verify that the setup is correct.

> **Info:** You will have to conclude this step within 72 hours. If Snowplow is unable to verify the setup within that timeframe, you will have to restart the process from the beginning.

Once Snowplow verifies your DNS setup, the status in the bottom left will change and you will receive a confirmation by email.

![dns\_records](/assets/images/Screenshot_dns_records-fd5ce76357c4e4726d9032ce85f974d3.png)

## Updating your tracking code

Once the new custom domain is verified, you can implement the tracking code.

This can be done in several ways:

- If you have not yet implemented tracking and wish to implement the JavaScript tracking snippet, navigate to “Start tracking events”. You can select the new custom domain from the dropdown and the code snippet will automatically update.
- If you already have tracking implemented and are using the JavaScript tracker, follow the above step or update the collector URL manually.
- If you wish to implement tracking or already have tracking with one of our other trackers then you will need to manually change the collector URL.

![implement\_new\_tracking\_snippet](/assets/images/Screenshot_implement_new_tracking_snippet-fcb7544f202315342935e5967e2ba8a9.png)

---

# Add entities and subject data to Flutter tracker events
> Enrich Flutter events with custom context entities using self-describing JSON schemas. Configure subject information including user ID, platform data, timezone, and screen resolution.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset:

1. Event context using self-describing data: Attach event context, describing anything you like, in the form of self-describing JSONs.
2. Subject: Include information about the user, or the platform on which the event occurred.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```cpp
tracker.track(
    Structured(category: 'shop', action: 'add-to-basket'),
    contexts: [
        const SelfDescribing(
            schema: 'iglu:com.my_company/movie_poster/jsonschema/1-0-0',
            data: {
                'movie_name': 'Solaris',
                'poster_country': 'JP',
                'poster_date': '1978-01-01'
            }
        ),
        const SelfDescribing(
            schema: 'iglu:com.my_company/customer/jsonschema/1-0-0',
            data: {
                'p_buy': 0.23,
                'segment': 'young_adult'
            }
        )
    ]
);
```

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

This information can be entered during tracker initialization by passing a `SubjectConfiguration` instance to the `Snowplow.createTracker` method. All of the information is optional.

Some subject information is filled automatically by the tracker. This includes the platform of the user, timezone, language, resolution, and viewport.

Please refer to the section on subject configuration on the [Configuration page](/docs/sources/flutter-tracker/initialization-and-configuration/) to learn more.

---

# Enable anonymous tracking with the Flutter tracker
> Anonymize user identifiers including domain_userid, session IDs, network_userid, and IP addresses. Configure client-side and server-side anonymization using userAnonymisation and serverAnonymisation flags.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/anonymous-tracking/

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

This table shows the [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) that can be anonymized on web and mobile:

| Identifier          | Location in event        | Tracked on web                              | Tracked on mobile  |
| ------------------- | ------------------------ | ------------------------------------------- | ------------------ |
| `user_id`           | Atomic                   | ✅                                           | ✅                  |
| `domain_userid`     | Atomic                   | ✅ Non-configurable                          | ✅                  |
| `domain_sessionid`  | Atomic                   | ✅ Non-configurable                          | ❌                  |
| `domain_sessionidx` | Atomic                   | ✅ Non-configurable                          | ❌                  |
| `userId`            | Session entity           | ✅ Non-configurable, same as `domain_userid` | ✅ Non-configurable |
| `sessionId`         | Session entity           | ✅ Non-configurable                          | ✅ Non-configurable |
| `previousSessionId` | Session entity           | ✅ Non-configurable                          | ✅ Non-configurable |
| `appleIdfa`         | Mobile (platform) entity | ❌                                           | ✅                  |
| `appleIdfv`         | Mobile (platform) entity | ❌                                           | ✅                  |
| `androidIdfa`       | Mobile (platform) entity | ❌                                           | ✅                  |
| `network_userid`    | Atomic                   | ✅ Non-configurable                          | ✅                  |
| `user_ipaddress`    | Atomic                   | ✅ Non-configurable                          | ✅                  |

You can set some of these properties at [tracker initialization](/docs/sources/flutter-tracker/initialization-and-configuration/).

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property at initialization, that value has priority.
>
> Similarly, if you set the `network_userid` property, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

There are several levels to the anonymization depending on which of the categories are affected.

## 1. Full client-side anonymization

In this case, we want to anonymize both the client-side user identifiers as well as the client-side session identifiers. Here the Session entity is not configured: no session information will be tracked at all.

```dart
const TrackerConfiguration()
    .userAnonymisation(true)
    .sessionContext(false)
    .platformContext(true) // only relevant for mobile
```

On web, setting `userAnonymisation` stops any user identifiers or session information being stored in cookies or localStorage. This means that `domain_userid`, `domain_sessionid`, and `domain_sessionidx` will be anonymized. These properties are already not present in any mobile events.

On mobile, setting `userAnonymisation` affects properties in the Session and Platform entities. In this example, the Platform entity is enabled. The `appleIdfv` Platform property would be anonymized, as well as `appleIdfa`/`androidIdfa` if your app is configured to track those.

**Web:**

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | ❌                   |
| `domain_sessionidx` | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | N/A                 |
| `appleIdfv`         | Mobile (platform) entity | N/A                 |
| `androidIdfa`       | Mobile (platform) entity | N/A                 |
| `network_userid`    | Atomic                   | ✅                   |
| `user_ipaddress`    | Atomic                   | ✅                   |

**Mobile:**

| Identifier          | Location in event        | Included in event?          |
| ------------------- | ------------------------ | --------------------------- |
| `user_id`           | Atomic                   | ❌                           |
| `domain_userid`     | Atomic                   | ❌                           |
| `domain_sessionid`  | Atomic                   | N/A                         |
| `domain_sessionidx` | Atomic                   | N/A                         |
| `userId`            | Session entity           | ❌ no session entity         |
| `sessionId`         | Session entity           | ❌ no session entity         |
| `previousSessionId` | Session entity           | ❌ no session entity         |
| `appleIdfa`         | Mobile (platform) entity | ❌                           |
| `appleIdfv`         | Mobile (platform) entity | ❌                           |
| `androidIdfa`       | Mobile (platform) entity | ❌                           |
| `network_userid`    | Atomic                   | ✅/❌ removed if you set this |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you set this |

***

## 2. Client-side anonymization with session tracking

This setting disables client-side user identifiers but tracks session information.

```dart
const TrackerConfiguration()
    .userAnonymisation(true)
    .sessionContext(true)
    .platformContext(true) // only relevant for mobile
```

Enabling both `userAnonymisation` and `sessionContext` means that events will have the Session entity, but the `userId` property will be anonymised to a null UUID (`00000000-0000-0000-0000-000000000000`). The `sessionId` and `sessionIndex` are still present, as are `domain_sessionid` and `domain_sessionidx` for web events. As above, if the Platform entity is enabled, the IDFA identifiers will not be present.

**Web:**

| Identifier          | Location in event        | Included in event? |
| ------------------- | ------------------------ | ------------------ |
| `user_id`           | Atomic                   | ❌                  |
| `domain_userid`     | Atomic                   | ❌                  |
| `domain_sessionid`  | Atomic                   | ✅                  |
| `domain_sessionidx` | Atomic                   | ✅                  |
| `userId`            | Session entity           | ❌ null UUID        |
| `sessionId`         | Session entity           | ✅                  |
| `previousSessionId` | Session entity           | ❌                  |
| `appleIdfa`         | Mobile (platform) entity | N/A                |
| `appleIdfv`         | Mobile (platform) entity | N/A                |
| `androidIdfa`       | Mobile (platform) entity | N/A                |
| `network_userid`    | Atomic                   | ✅                  |
| `user_ipaddress`    | Atomic                   | ✅                  |

**Mobile:**

| Identifier          | Location in event        | Included in event?          |
| ------------------- | ------------------------ | --------------------------- |
| `user_id`           | Atomic                   | ❌                           |
| `domain_userid`     | Atomic                   | ❌                           |
| `domain_sessionid`  | Atomic                   | N/A                         |
| `domain_sessionidx` | Atomic                   | N/A                         |
| `userId`            | Session entity           | ❌ null UUID                 |
| `sessionId`         | Session entity           | ✅                           |
| `previousSessionId` | Session entity           | ❌                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                           |
| `appleIdfv`         | Mobile (platform) entity | ❌                           |
| `androidIdfa`       | Mobile (platform) entity | ❌                           |
| `network_userid`    | Atomic                   | ✅/❌ removed if you set this |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you set this |

***

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set server-side. In particular, these are the `network_userid` property set in server-side cookie and the user IP address (`user_ipaddress`). You can anonymize the properties using the `serverAnonymisation` flag in `EmitterConfiguration`:

```dart
const EmitterConfiguration()
    .serverAnonymisation(true)
```

Setting this will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow collector. The Snowplow pipeline will anonymize the identifiers.

On mobile, `serverAnonymisation` and `userAnonymisation` are separate. This means you can anonymize only the server-side identifiers without also anonymizing the client-side identifiers.

On web, setting `serverAnonymisation` provides full anonymization. It's effectively a subset of `userAnonymisation`. If `serverAnonymisation` is enabled, this will also set `userAnonymisation` to `true` (even if set to `false` in your `TrackerConfiguration`).

**Web:**

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | ❌                   |
| `domain_sessionidx` | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | N/A                 |
| `appleIdfv`         | Mobile (platform) entity | N/A                 |
| `androidIdfa`       | Mobile (platform) entity | N/A                 |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

**Mobile:**

| Identifier          | Location in event        | Included in event?                |
| ------------------- | ------------------------ | --------------------------------- |
| `user_id`           | Atomic                   | ✅                                 |
| `domain_userid`     | Atomic                   | ✅                                 |
| `domain_sessionid`  | Atomic                   | N/A                               |
| `domain_sessionidx` | Atomic                   | N/A                               |
| `userId`            | Session entity           | ✅                                 |
| `sessionId`         | Session entity           | ✅                                 |
| `previousSessionId` | Session entity           | ✅                                 |
| `appleIdfa`         | Mobile (platform) entity | ✅                                 |
| `appleIdfv`         | Mobile (platform) entity | ✅                                 |
| `androidIdfa`       | Mobile (platform) entity | ✅                                 |
| `network_userid`    | Atomic                   | ❌/✅ still present if you set this |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you set this |

***

## 4. Full anonymization for mobile

For full anonymization on mobile, set both client-side and server-side anonymization.

```dart
const TrackerConfiguration()
    .userAnonymisation(true)
    .sessionContext(false)
    .platformContext(true) // only relevant for mobile

const EmitterConfiguration()
    .serverAnonymisation(true)
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `domain_sessionid`  | Atomic                   | N/A                 |
| `domain_sessionidx` | Atomic                   | N/A                 |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

---

# Get started with the Flutter tracker
> Install the Snowplow Flutter tracker from pub.dev and initialize tracking in your Flutter app. Create a tracker instance with namespace and collector endpoint to track events across Web, Android, and iOS platforms.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/getting-started/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

The following steps will guide you through setting up the Flutter tracker in your project and tracking a simple event.

## Installation

Add the Snowplow tracker as a dependency to your Flutter application:

```bash
flutter pub add snowplow_tracker
```

This will add a line with the dependency to your pubspec.yaml:

```yaml
dependencies:
  snowplow_tracker: ^0.8.0
```

Import the package into your Dart code:

```dart
import 'package:snowplow_tracker/snowplow_tracker.dart'
```

### Installation on Web

If using the tracker within a Flutter app for Web, you will also need to import the Snowplow JavaScript Tracker in your `index.html` file. Please load the JS tracker with the Snowplow tag as [described here](/docs/sources/web-trackers/tracker-setup/). Do not change the global function name `snowplow` that is used to access the tracker – the Flutter APIs assume that it remains the default as shown in documentation.

Make sure to use JavaScript tracker version `3.5` or newer. You may also refer to the [example project](https://github.com/snowplow/snowplow-flutter-tracker/tree/main/example) in the Flutter tracker repository to see this in action.

## Initialization

Instantiate a tracker using the `Snowplow.createTracker` function. At its most basic, the function takes two required arguments: `namespace` and `endpoint`. Tracker namespace identifies the tracker instance, you may create multiple trackers with different namespaces. The endpoint is the URI of the Snowplow collector to send the events to. This tracker creation is asynchronous and uses the `await` keyword; therefore it must occur inside a function labelled `async`. You could create the tracker in the `main()` of your main widget.

```dart
SnowplowTracker tracker = await Snowplow.createTracker(
    namespace: 'ns1',
    endpoint: 'http://...'
);

// Creating a tracker in the main() function
// It is passed to the other widgets as necessary
Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();

  SnowplowTracker tracker = await Snowplow.createTracker(
      namespace: "namespace",
      endpoint: "http://0.0.0.0:9090");

  runApp(MyApp(tracker: tracker));
}
```

There are additional optional arguments to configure the tracker. To learn more about configuring how events are sent, check out [this page](/docs/sources/flutter-tracker/initialization-and-configuration/).

## Tracking events

To track events, simply instantiate their respective types (e.g., `ScreenView`, `SelfDescribing`, `Structured`) and pass them to the `tracker.track` or `Snowplow.track` methods.

```dart
tracker.track(ScreenView(
    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
    name: 'home',
    type: 'full',
    transitionType: 'none'));
```

Visit documentation about [tracking events](/docs/sources/flutter-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/flutter-tracker/adding-data/).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

---

# Track events from hybrid Flutter apps with WebViews
> Enable event tracking from WebView content in Flutter apps using webview_flutter package. Forward Web tracker events to Flutter tracker or use WebView tracker directly to share sessions.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/hybrid-apps/

> **Info:** This feature is available since v0.8 of the Flutter tracker and supports WebView implemented using the [`webview_flutter` package](https://pub.dev/packages/webview_flutter).

Hybrid apps are mobile apps that in addition to a Flutter interface, provide part of the UI through an embedded Web view. Snowplow events are tracked from both the Flutter code as well as the Web view. Our goal is to have both events tracked from both places to share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the Flutter tracker.

1. Implement the Flutter tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages. Here is a simplified example usage:

   ```dart
   import 'package:flutter/services.dart';
   import 'package:snowplow_tracker/snowplow_tracker.dart';

   class MainPage extends StatefulWidget {
       const MainPage({super.key, required this.tracker});

       // Snowplow tracker instance
       final SnowplowTracker tracker;

       @override
       State<MainPage> createState() => _MainPageState();
   }

   class _MainPageState extends State<MainPage> with WidgetsBindingObserver {
       late final WebViewController _webViewController;

       @override
       void initState() {
           _webViewController = WebViewController()
               ..setJavaScriptMode(JavaScriptMode.unrestricted)
               ..loadRequest(Uri.parse(const String.fromEnvironment('WEBVIEW_URL',
                   defaultValue: 'http://localhost:3000')));

           // register the webview controller with the tracker to set up JavaScript channel
           widget.tracker.registerWebViewJavaScriptChannel(
               webViewController: _webViewController);
       }

       @override
       Widget build(BuildContext context) {
           return SizedBox(
               height: 300,
               child: WebViewWidget(controller: _webViewController),
           );
       }
   }
   ```

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the Flutter tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the Flutter mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the Flutter tracker.
2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.
3. Subscribe to WebView event messages. This step is the same as shown above.
4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# Flutter tracker SDK
> The Snowplow Flutter Tracker enables cross-platform event tracking from Flutter applications across Web, Android, and iOS. Track screen views, custom events, and user sessions with unified data model support.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/

The Snowplow Flutter Tracker allows you to add analytics to your Flutter apps for Web, Android, and iOS when using a Snowplow pipeline.

The tracker is published on pub.dev as [snowplow\_tracker](https://pub.dev/packages/snowplow_tracker).

---

# Initialize and configure the Flutter tracker
> Initialize the Flutter tracker with createTracker method and configure TrackerConfiguration, SubjectConfiguration, GdprConfiguration, and EmitterConfiguration. Set up app identifiers, context entities, session tracking, and anonymous tracking options.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/initialization-and-configuration/

The package provides a single method to initialize and configure a new tracker, the `Snowplow.createTracker` method. It accepts configuration parameters for the tracker and returns a `SnowplowTracker` instance.

```dart
SnowplowTracker tracker = await Snowplow.createTracker(
    namespace: 'ns1',
    endpoint: 'http://...',
    trackerConfig: const TrackerConfiguration(...),
    gdprConfig: const GdprConfiguration(...),
    subjectConfig: const SubjectConfiguration(...));
);
```

The method returns a `SnowplowTracker` instance. This can be later used for tracking events, or accessing tracker properties. However, all methods provided by the `SnowplowTracker` instance are also available as static functions in the `Snowplow` class but they require passing the tracker namespace as string.

The only required attributes of the `Snowplow.createTracker` method are `namespace` used to identify the tracker, and the Snowplow collector `endpoint`. Additionally, one can configure the HTTP method to be used when sending events to the collector, as well as a custom POST path, and provide configuration by instantiating classes for `TrackerConfiguration`, `SubjectConfiguration`, or `GdprConfiguration`. By default, events are sent by POST. The following arguments are accepted by the `Snowplow.createTracker` method:

| Attribute        | Type                    | Description                                                                          |
| ---------------- | ----------------------- | ------------------------------------------------------------------------------------ |
| `namespace`      | `String`                | Tracker namespace to identify the tracker.                                           |
| `endpoint`       | `String`                | URI for the Snowplow collector endpoint.                                             |
| `method`         | `Method?`               | HTTP method to use: `Method.get` or `Method.post` (`Method.post` is default).        |
| `customPostPath` | `String?`               | Custom POST path.                                                                    |
| `requestHeaders` | `Map<String, String>?`  | Map of custom HTTP headers to add to requests to the collector.                      |
| `trackerConfig`  | `TrackerConfiguration?` | Configuration of the tracker and the core tracker properties.                        |
| `gdprConfig`     | `GdprConfiguration?`    | Determines the GDPR context that will be attached to all events sent by the tracker. |
| `subjectConfig`  | `SubjectConfiguration?` | Subject information about tracked user and device that is added to events.           |
| `emitterConfig`  | `EmitterConfiguration?` | Configuration for how the events are sent.                                           |

> **Note:** The ability to set `customPostPath` was added in v0.2.0. Setting a custom POST path can be useful in avoiding adblockers; it replaces the default "com.snowplowanalytics/snowplow/tp2". Your event collector must also be configured to accept the custom path.

> **Note:** The `EmitterConfiguration` class was added in v0.3.0.

## Configuration of tracker properties: `TrackerConfiguration`

`TrackerConfiguration` provides options to configure properties and features of the tracker. In addition to setting the app identifier and device platform, the configuration enables turning several automatic context entities on and off.

| Attribute                      | Type                         | Description                                                                                                                                                                                | Android | iOS | Web | Default                                       |
| ------------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | --- | --- | --------------------------------------------- |
| `appId`                        | `String?`                    | Identifier of the app.                                                                                                                                                                     | ✔       | ✔   | ✔   | null on Web, bundle identifier on iOS/Android |
| `devicePlatform`               | `DevicePlatform?`            | The device platform the tracker runs on. Available options are provided by the `DevicePlatform` enum.                                                                                      | ✔       | ✔   | ✔   | "web" on Web, "mob" on iOS/Android            |
| `base64Encoding`               | `bool?`                      | Indicates whether payload JSON data should be base64 encoded.                                                                                                                              | ✔       | ✔   | ✔   | true                                          |
| `platformContext`              | `bool?`                      | Indicates whether [platform](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-2) (mobile) entity should be attached to tracked events.          | ✔       | ✔   |     | true                                          |
| `geoLocationContext`           | `bool?`                      | Indicates whether [geo-location](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-1-0) entity should be attached to tracked events.          | ✔       | ✔   | ✔   | false                                         |
| `sessionContext`               | `bool?`                      | Indicates whether [session](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) entity should be attached to tracked events.                    | ✔       | ✔   | ✔   | true                                          |
| `webPageContext`               | `bool?`                      | Indicates whether a context entity about current [web page](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0) should be attached to tracked events. |         |     | ✔   | true                                          |
| `screenContext`                | `bool?`                      | Indicates whether [screen](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/screen/jsonschema/1-0-0) entity should be attached to tracked events.                               | ✔       | ✔   |     | true                                          |
| `applicationContext`           | `bool?`                      | Indicates whether [application](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/application/jsonschema/1-0-0) entity should be attached to tracked events.                     | ✔       | ✔   |     | true                                          |
| `webActivityTracking`          | `WebActivityTracking?`       | Enables activity tracking using page views and pings on the Web.                                                                                                                           |         |     | ✔   | true                                          |
| `userAnonymisation`            | `bool?`                      | Anonymizes certain user identifiers.                                                                                                                                                       | ✔       | ✔   | ✔   | false                                         |
| `lifecycleAutotracking`        | `bool?`                      | Indicates whether the [lifecycle](iglu:com.snowplowanalytics.mobile/application_lifecycle/jsonschema/1-0-0) entity and foreground and background events should be autotracked.             | ✔       | ✔   |     | true                                          |
| `screenEngagementAutotracking` | `bool?`                      | Indicates whether to enable tracking of the screen end event and the screen summary context entity.                                                                                        | ✔       | ✔   |     | true                                          |
| `platformContextProperties`    | `PlatformContextProperties?` | Overrides for the values for properties of the platform context entity.                                                                                                                    | ✔       | ✔   |     | null                                          |

> **Note:** The ability to enable `userAnonymisation`, or the screen and application context entities, was added in v0.3.0.

> **Note:** The ability to enable `lifecycleAutotracking` was added in v0.5.0.

The optional `WebActivityTracking` property configures page tracking on Web. Initializing the configuration will inform `SnowplowObserver` observers (see section on auto-tracking in "Tracking events") to auto track `PageViewEvent` events instead of `ScreenView` events on navigation changes. Further, setting the `minimumVisitLength` and `heartbeatDelay` properties of the `WebActivityTracking` instance will enable activity tracking using 'page ping' events on Web.

Activity tracking monitors whether a user continues to engage with a page over time, and record how they digest content on the page over time. That is accomplished using 'page ping' events. If activity tracking is enabled, the web page is monitored to see if a user is engaging with it. (E.g. is the tab in focus, does the mouse move over the page, does the user scroll etc.) If any of these things occur in a set period of time (`minimumVisitLength` seconds from page load and every `heartbeatDelay` seconds after that), a page ping event fires, and records the maximum scroll left / right and up / down in the last ping period. If there is no activity in the page (e.g. because the user is on a different browser tab), no page ping fires.

Lifecycle autotracking is only available on mobile apps (iOS and Android). When configured (it is enabled by default), a Lifecycle context entity is attached to all events. It records whether the app was visible or not when the event was tracked. In addition, a `Background` event will be tracked when the app is moved to background, and a `Foreground` event when the app moves back to foreground (becomes visible on the screen).

Screen engagement autotracking is also only available on mobile apps (iOS and Android). When configured (it is enabled by default), a screen summary context entity will be tracked along with screen end, foreground and background events. Make sure that you have lifecycle autotracking enabled for screen summary to have complete information.

See [this page](/docs/sources/flutter-tracker/anonymous-tracking/) for information about anonymous tracking.

## Configuration of emitter properties: `EmitterConfiguration`

This Configuration class was added in v0.3.0. Currently, the only property is `serverAnonymisation`.

| Attribute             | Type    | Description                                        | Android | iOS | Web | Default |
| --------------------- | ------- | -------------------------------------------------- | ------- | --- | --- | ------- |
| `serverAnonymisation` | `bool?` | Prevents tracking of server-side user identifiers. | ✔       | ✔   | ✔   | false   |

## Configuration of subject information: `SubjectConfiguration`

Subject information are persistent and global information about the tracked device or user. They apply to all events and are assigned as event properties.

Some of the properties are only configurable on iOS and Android and are automatically assigned on the Web.

| Attribute          | Type      | Description                                                     | Android | iOS | Web                              | Default |
| ------------------ | --------- | --------------------------------------------------------------- | ------- | --- | -------------------------------- | ------- |
| `userId`           | `String?` | Business ID of the user.                                        | ✔       | ✔   | ✔                                |         |
| `networkUserId`    | `String?` | Network user ID (UUIDv4).                                       | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `domainUserId`     | `String?` | Domain user ID (UUIDv4).                                        | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `userAgent`        | `String?` | Custom user-agent. It overrides the user-agent used by default. | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `ipAddress`        | `String?` | IP address.                                                     | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `timezone`         | `String?` | The timezone label.                                             | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `language`         | `String?` | The language set on the device.                                 | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `screenResolution` | `Size?`   | The screen resolution on the device.                            | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `screenViewport`   | `Size?`   | The screen viewport.                                            | ✔       | ✔   | Non-configurable, auto-assigned. |         |
| `colorDepth`       | `double?` | The color depth.                                                | ✔       | ✔   | Non-configurable, auto-assigned. |         |

The configured attributes are mapped to Snowplow event properties described in the [Snowplow Tracker Protocol](/docs/events/). They are mapped as follows:

| Attribute                 | Event Property      |
| ------------------------- | ------------------- |
| `userId`                  | `uid`               |
| `networkUserId`           | `network_userid`    |
| `domainUserId`            | `domain_userid`     |
| `userAgent`               | `useragent`         |
| `ipAddress`               | `user_ipaddress`    |
| `timezone`                | `os_timezone`       |
| `language`                | `lang`              |
| `screenResolution.width`  | `dvce_screenwidth`  |
| `screenResolution.height` | `dvce_screenheight` |
| `screenViewport.width`    | `br_viewwidth`      |
| `screenViewport.height`   | `br_viewheight`     |
| `colorDepth`              | `br_colordepth`     |

## GDPR context entity configuration: `GdprConfiguration`

Determines the GDPR context that will be attached to all events sent by the tracker.

| Attribute             | Type     | Description                  | Android | iOS | Web | Default |
| --------------------- | -------- | ---------------------------- | ------- | --- | --- | ------- |
| `basisForProcessing`  | `String` | Basis for processing.        | ✔       | ✔   | ✔   |         |
| `documentId`          | `String` | ID of a GDPR basis document. | ✔       | ✔   | ✔   |         |
| `documentVersion`     | `String` | Version of the document.     | ✔       | ✔   | ✔   |         |
| `documentDescription` | `String` | Description of the document. | ✔       | ✔   | ✔   |         |

---

# Configure session tracking with the Flutter tracker and Unified Digital package
> Enable session tracking with client_session context entity across Android, iOS, and Web platforms. Configure foreground and background timeouts to track user sessions consistently.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/sessions-and-data-model/

The Flutter tracker gives you the option to adopt the [Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) across all supported platforms – Android, iOS, and Web.

In addition to adopting screen view events, the unified data model defines that sessions are represented using a [context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-1) where it exists. Concretely, the `client_session` context entity is added to all tracked events if session tracking is enabled in the tracker configuration (through the `sessionContext` property). This entity consists of the following properties:

| Attribute           | Description                                                               | Required? |
| ------------------- | ------------------------------------------------------------------------- | --------- |
| `userId`            | An identifier for the user of the session.                                | Yes       |
| `sessionId`         | An identifier (UUID) for the session.                                     | Yes       |
| `sessionIndex`      | The index of the current session for this user.                           | Yes       |
| `previousSessionId` | The previous session identifier (UUID) for this user.                     | No        |
| `storageMechanism`  | The mechanism that the session information has been stored on the device. | Yes       |
| `firstEventId`      | The optional identifier (UUID) of the first event id for this session.    | No        |

Behind the scenes, the Flutter tracker uses the default configuration for session management on the Android, iOS, and Web trackers.

On Android and iOS, session data is maintained for the life of the application being installed on a device. Essentially it will update if it is not accessed within a configurable timeout. There are two inactivity timeouts that result in updates to the `sessionId`: foreground inactivity, and background inactivity timeout. The default 30 minutes setting is used for both.

On the Web, the tracker uses domain (`duid`) and session cookies (`sid`) as implemented by the JavaScript tracker. The session cookie expires after 30 minutes of inactivity. This means that a user leaving the site and returning in under 30 minutes does not change the session. In contrast with the JavaScript tracker, the Flutter tracker also adds the `client_session` context entity that wraps the domain and session IDs.

---

# Track events with the Flutter tracker
> Track events in Flutter using SelfDescribing, Structured, ScreenView, Timing, and Consent events. Enable auto-tracking of page views with SnowplowObserver or manually track custom events with schemas.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in event classes to help you track different kinds of events. When instantiated, their objects can be passed to the `Snowplow.track()` method to send events to the Snowplow collector. The event classes range from single purpose ones, such as `ScreenView`, to the more complex but flexible `SelfDescribing`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribing` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

## Auto-tracked view events

There is an option to automatically track view events when currently active pages change through the [Navigator API](https://api.flutter.dev/flutter/widgets/Navigator-class.html).

To activate this feature, one has to register a `SnowplowObserver` retrieved from the tracker instance using `SnowplowTracker.getObserver()`. The retrieved observer can be added to `navigatorObservers` in `MaterialApp`:

```dart
MaterialApp(
  navigatorObservers: [
    tracker.getObserver()
  ],
  ...
);
```

If using the `Router` API with the `MaterialApp.router` constructor, add the observer to the `observers` of your `Navigator` instance, e.g.:

```dart
Navigator(
  observers: [tracker.getObserver()],
  ...
);
```

The `SnowplowObserver` automatically tracks `PageViewEvent` and `ScreenView` events when the currently active `ModalRoute` of the navigator changes.

By default, `ScreenView` events are tracked on all platforms. In case `TrackerConfiguration.webActivityTracking` is configured when creating the tracker, `PageViewEvent` events will be tracked on Web instead of `ScreenView` events (`ScreenView` events will still be tracked on other platforms).

The `SnowplowTracker.getObserver()` function takes an optional `nameExtractor` function as argument which is used to extract a name from new routes that is used in tracked `ScreenView` or `PageViewEvent` events.

The following operations will result in tracking a view event:

```dart
Navigator.pushNamed(context, '/contact/123');

Navigator.push<void>(context, MaterialPageRoute(
  settings: RouteSettings(name: '/contact/123'),
  builder: (_) => ContactDetail(123)));

Navigator.pushReplacement<void>(context, MaterialPageRoute(
  settings: RouteSettings(name: '/contact/123'),
  builder: (_) => ContactDetail(123)));

Navigator.pop(context);
```

## Manually-tracked events

Event classes supported by the Flutter Tracker:

| Method             | Event type tracked                                          |
| ------------------ | ----------------------------------------------------------- |
| `SelfDescribing`   | Custom event based on "self-describing" JSON schema         |
| `Structured`       | Semi-custom structured event                                |
| `ScreenView`       | View of a screen in the app                                 |
| `Timing`           | User timing events such as how long resources take to load. |
| `ConsentGranted`   | User opting into data collection.                           |
| `ConsentWithdrawn` | User withdrawing consent for data collection.               |

All the methods share common features and parameters. Every type of event can have an optional context added. See the [next page](/docs/sources/flutter-tracker/adding-data/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event types, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `selfDescribing`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `SelfDescribing`

Use the `SelfDescribing` type to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

Creating an instance of `SelfDescribing` takes a schema name and a dictionary of event data.

Example (assumes that `tracker` is a tracker instance created using `Snowplow.createTracker`):

```dart
tracker.track(SelfDescribing(
    schema: 'iglu:com.example_company/save_game/jsonschema/1-0-2',
    data: {
        'saveId': '4321',
        'level': 23,
        'difficultyLevel': 'HARD',
        'dlContent': true
    }
));
```

## Track structured events with `Structured`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument   | Description                                                    | Required in event? |
| ---------- | -------------------------------------------------------------- | ------------------ |
| `category` | The grouping of structured events which this action belongs to | Yes                |
| `action`   | Defines the type of user interaction which this event involves | Yes                |
| `label`    | Often used to refer to the 'object' the action is performed on | No                 |
| `property` | Describing the 'object', or the action performed on it         | No                 |
| `value`    | Provides numerical data about the event                        | No                 |

Example:

```dart
tracker.track(Structured(
    category: 'shop',
    action: 'add-to-basket',
    label: 'Add To Basket',
    property: 'pcs',
    value: 2.00,
));
```

## Track page views with `PageViewEvent`

The `PageViewEvent` may be used to track page views on the Web. The event is designed to track web page views and automatically captures page title, referrer and URL. Being Web-only, it is not implemented on Android and iOS where the app is not displayed as a Web page.

Page view events are the basic building blocks for the [Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/).

## Track screen views with `ScreenView`

Use `ScreenView` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`. The `name` and `id` properties are required. "Name" is the human-readable screen name, and "ID" should be the unique screen ID (UUID v4).

Screen view events are also used in the[Snowplow Unified data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/). Nevertheless, the Flutter tracker also implements them on Web. You may adopt them for use in the unified data model to provide consistent event tracking across all platforms.

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

| Argument         | Description                                                 | Required in event? |
| ---------------- | ----------------------------------------------------------- | ------------------ |
| `name`           | The name of the screen viewed.                              | Yes                |
| `id`             | The id (UUID v4) of screen that was viewed.                 | Yes                |
| `type`           | The type of screen that was viewed.                         | No                 |
| `previousName`   | The name of the previous screen that was viewed.            | No                 |
| `previousType`   | The type of screen that was viewed.                         | No                 |
| `previousId`     | The id (UUID v4) of the previous screen that was viewed.    | No                 |
| `transitionType` | The type of transition that led to the screen being viewed. | No                 |

Example:

```dart
tracker.track(ScreenView(
    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
    name: 'home',
    type: 'full',
    transitionType: 'none'));
```

### Screen engagement tracking

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

#### Updating list item view and scroll depth information

To update the list item viewed and scroll depth information tracked in the screen summary entity, you can track the `ListItemView` and `ScrollChanged` events with this information. When tracked, the tracker won't send these events individually to the collector, but will process the information into the next `screen_summary` entity and discard the events. You may want to track the events every time a new list item is viewed on the screen, or whenever the scroll position changes.

To update the list items viewed information:

```dart
Event event = const ListItemView(index: 1, itemsCount: 10);
tracker.track(event);
```

To update the scroll depth information:

```dart
Event event = const ScrollChanged(
  yOffset: 10,
  xOffset: 20,
  viewHeight: 100,
  viewWidth: 200,
  contentHeight: 300,
  contentWidth: 400,
);
tracker.track(event);
```

## Track timing events with `Timing`

Use the `Timing` type to track user timing events such as how long resources take to load. These events take a timing `category`, the `variable` being measured, and the `timing` time measurement. An optional `label` can be added to further identify the timing event

| Argument   | Description                                              | Required in event? |
| ---------- | -------------------------------------------------------- | ------------------ |
| `category` | Defines the timing category.                             | Yes                |
| `variable` | Defines the timing variable measured.                    | Yes                |
| `timing`   | Represents the time.                                     | Yes                |
| `label`    | An optional string to further identify the timing event. | No                 |

Example:

```dart
tracker.track(Timing(
    category: 'category',
    variable: 'variable',
    timing: 1,
    label: 'label',
));
```

## Track user consent with `ConsentGranted` and `ConsentWithdrawn`

Use the `ConsentGranted` to track a user opting into data collection and `ConsentWithdrawn` to track a user withdrawing their consent for data collection.

For both events, a consent document context will be attached to the event using the `documentId` and `version` arguments supplied. To specify that a user opts out of all data collection using the `ConsentWithdrawn` event, the `all` property should be set to true.

Properties of `ConsentGranted`:

| Argument              | Description                            | Required in event? |
| --------------------- | -------------------------------------- | ------------------ |
| `expiry`              | The expiry date-time of the consent.   | Yes                |
| `documentId`          | The consent document ID.               | Yes                |
| `version`             | The consent document version.          | Yes                |
| `name`                | Optional consent document name.        | No                 |
| `documentDescription` | Optional consent document description. | No                 |

Example:

```dart
tracker.track(ConsentGranted(
    expiry: DateTime.now(),
    documentId: '1234',
    version: '5',
    name: 'name1',
    documentDescription: 'description1',
));
```

Properties of `ConsentWithdrawn`:

| Argument              | Description                                   | Required in event? |
| --------------------- | --------------------------------------------- | ------------------ |
| `all`                 | Whether user opts out of all data collection. | Yes                |
| `documentId`          | The consent document ID.                      | Yes                |
| `version`             | The consent document version.                 | Yes                |
| `name`                | Optional consent document name.               | No                 |
| `documentDescription` | Optional consent document description.        | No                 |

Example:

```dart
tracker.track(ConsentWithdrawn(
    all: false,
    documentId: '1234',
    version: '5',
    name: 'name1',
    documentDescription: 'description1',
));
```

---

# Media tracking with the Flutter tracker
> Track media playback events in Flutter applications using the built-in media tracking capabilities available since version 0.7.0.
> Source: https://docs.snowplow.io/docs/sources/flutter-tracker/tracking-events/media-tracking/

> **Note:** Media tracking is available in the Flutter tracker since version 0.7.0.

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

```dart
// 1. Start tracking when player loads
final id = const Uuid().v4();
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: MediaTrackingConfiguration(
      id: id,
      player: const MediaPlayerEntity(
          duration: 300,
          label: 'My Video Title',
          mediaType: MediaType.video,
      )
  )
);

// 2. Update player state every second
await mediaTracking.update(
  player: MediaPlayerEntity(currentTime: currentTime)
);

// 3. Track events as they occur
await mediaTracking.track(MediaPlayEvent());
await mediaTracking.track(MediaPauseEvent());
await mediaTracking.track(MediaSeekStartEvent());
await mediaTracking.track(MediaSeekEndEvent());
await mediaTracking.track(MediaEndEvent());

// 4. End tracking when done
await tracker.endMediaTracking(id: id);
```

> **Info:** To track media events on web with the Flutter tracker, configure the JavaScript plugin URL:```dart
> SnowplowTracker tracker = await Snowplow.createTracker(
>   namespace: namespace,
>   endpoint: endpoint,
>   trackerConfig: const TrackerConfiguration(
>       jsMediaPluginURL: "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js"
>   )
> );
> ```

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(
      id: id,
      player: MediaPlayerEntity(
          currentTime: 0,          // Current playback position in seconds
          duration: 300,           // Total duration in seconds
          ended: false,            // Whether playback has ended
          livestream: false,       // Whether this is a live stream
          label: 'My Video',       // Human-readable title (recommended)
          loop: false,             // Whether to restart after ending
          mediaType: MediaType.video, // MediaType.video or MediaType.audio
          muted: false,            // Whether audio is muted
          paused: true,            // Whether playback is paused
          pictureInPicture: false, // Whether in picture-in-picture mode
          playerType: 'html5',     // Player type identifier
          playbackRate: 1.0,       // Playback speed (1 = normal)
          quality: '1080p',        // Quality level
          volume: 100              // Volume percentage (0-100)
      )
  )
);
```

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(
      id: id,
      pingInterval: 30,   // Seconds between pings (default: 30)
      maxPausedPings: 1   // Max pings while paused (default: 1)
  )
);

// Or turn off pings entirely
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(id: id, pings: false)
);
```

### Percentage progress

Track events when playback reaches specified percentage boundaries.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(
      id: id,
      boundaries: [10, 25, 50, 75, 90]  // Fire events at these percentages
  )
);
```

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(id: id, session: true)
);
```

### Custom entities

You can attach custom entities to all events for this media tracking instance.

```dart
MediaTracking mediaTracking = await tracker.startMediaTracking(
  configuration: const MediaTrackingConfiguration(
      id: id,
      contexts: [
          SelfDescribing(
              schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",
              data: {"contentId": "abc123", "category": "tutorial"}
          )
      ]
  )
);
```

### Filter events

Media event filtering isn't available for the Flutter tracker.

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

```dart
await mediaTracking.track(MediaReadyEvent());        // Player is ready
await mediaTracking.track(MediaPlayEvent());         // Playback started
await mediaTracking.track(MediaPauseEvent());        // Playback paused
await mediaTracking.track(MediaEndEvent());          // Playback ended
await mediaTracking.track(MediaSeekStartEvent());    // Seek operation started
await mediaTracking.track(MediaSeekEndEvent());      // Seek operation ended
await mediaTracking.track(MediaBufferStartEvent());  // Buffering started
await mediaTracking.track(MediaBufferEndEvent());    // Buffering ended
```

### Player state events

```dart
await mediaTracking.track(const MediaPlaybackRateChangeEvent(newRate: 1.5));
await mediaTracking.track(const MediaVolumeChangeEvent(newVolume: 80));
await mediaTracking.track(const MediaFullscreenChangeEvent(fullscreen: true));
await mediaTracking.track(const MediaPictureInPictureChangeEvent(pictureInPicture: true));
await mediaTracking.track(const MediaQualityChangeEvent(
  newQuality: '1080p',
  bitrate: 5000,
  framesPerSecond: 30,
  automatic: true
));
await mediaTracking.track(const MediaErrorEvent(
  errorCode: '500',
  errorName: 'NetworkError',
  errorDescription: 'Failed to load media'
));
```

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

```dart
// Ad break events
await mediaTracking.track(
  MediaAdBreakStartEvent(),
  adBreak: const MediaAdBreakEntity(
      breakId: 'break-1',
      name: 'pre-roll',
      breakType: MediaAdBreakType.linear,
      podSize: 3
  )
);
await mediaTracking.track(MediaAdBreakEndEvent());

// Ad events
await mediaTracking.track(
  MediaAdStartEvent(),
  ad: const MediaAdEntity(
      adId: 'ad-1',
      name: 'Product Ad',
      creativeId: 'creative-1',
      duration: 30,
      skippable: true
  )
);
await mediaTracking.track(MediaAdFirstQuartileEvent());
await mediaTracking.track(MediaAdMidpointEvent());
await mediaTracking.track(MediaAdThirdQuartileEvent());
await mediaTracking.track(MediaAdCompleteEvent());
await mediaTracking.track(MediaAdSkipEvent());
await mediaTracking.track(MediaAdClickEvent());
await mediaTracking.track(MediaAdPauseEvent());
await mediaTracking.track(MediaAdResumeEvent());
```

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

```dart
await mediaTracking.track(
  const SelfDescribing(
      schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",
      data: {"action": "share", "platform": "twitter"}
  )
);
```

## Update player state

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```dart
await mediaTracking.update(
  player: MediaPlayerEntity(currentTime: currentTime)
);
```

---

# Track the Subject with the Golang tracker
> Attach user and device information to Golang tracker events using the Subject class. Set user ID, screen resolution, viewport, timezone, language, IP address, and user-agent for enriched event data.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/adding-extra-data-the-subject-class/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

Create a subject like this:

```go
subject := sp.InitSubject()
```

The Subject class has a set of `Set...()` methods to attach extra data relating to the user to all tracked events:

- `SetUserId`
- `SetScreenResolution`
- `SetViewport`
- `SetColorDepth`
- `SetTimezone`
- `SetLanguage`
- `SetIpAddress`
- `SetUseragent`
- `SetDomainUserId`
- `SetNetworkUserId`

### Tracking the Subject

The `Subject` can be used into two ways.

When initialising the `Tracker`:

```go
tracker := sp.InitTracker(
  sp.RequireEmitter(emitter),
  sp.OptionSubject(subject)
)
```

Or when you track an event:

```go
tracker.TrackPageView(sp.PageViewEvent{
  PageUrl: sphelp.NewString("acme.com"),
  Subject: subject
})
```

When setting the `Subject` as you track as event, you will override the `Tracker` level `Subject` for that specific event.

### Set user ID with `SetUserId`

You can set the user ID to any string:

```go
s.SetUserId( "{{USER ID}}" );
```

Example:

```go
s.SetUserId("alexd");
```

### Set screen resolution with `SetScreenResolution`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```go
s.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```go
s.SetScreenResolution(1366, 768);
```

### Set viewport dimensions with `SetViewport`

If your code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```go
s.SetViewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```go
s.SetViewport(300, 200);
```

### Set color depth with `SetColorDepth`

If your code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```go
s.SetColorDepth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```go
s.SetColorDepth(32);
```

### Set timezone with `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```go
s.timezone( {{TIMEZONE}} );
```

The timezone should be a string:

```go
s.SetColorDepth("Europe/London");
```

### Set the language with `SetLang`

This method lets you pass a user's language in to Snowplow:

```go
s.SetLang( {{LANGUAGE}} );
```

The language should be a string:

```go
s.SetLang('en');
```

### Set the IP Address with `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```go
s.SetIpAddress( {{IP ADDRESS}} );
```

The IP Address should be a string:

```go
s.SetIpAddress('127.0.0.1');
```

### Set the useragent with `SetUseragent`

This method lets you pass a user's useragent string in to Snowplow:

```go
s.SetUseragent( {{USERAGENT}} );
```

The useragent should be a string:

```go
s.SetUseragent('some useragent');
```

### Set the Domain User ID with `SetDomainUserId`

This method lets you pass a user's Domain User ID string in to Snowplow:

```go
s.SetDomainUserId( {{DOMAIN USER ID}} );
```

The Domain User ID should be a string:

```go
s.SetDomainUserId('domain-uid-12');
```

### Set the Domain User ID with `SetNetworkUserId`

This method lets you pass a user's Network User ID string in to Snowplow:

```go
s.SetNetworkUserId( {{NETWORK USER ID}} );
```

The Network User ID should be a string:

```go
s.SetNetworkUserId('network-uid-12');
```

---

# Configure emitters with the Golang tracker
> Configure emitters with collector URI, storage implementation (memory or SQLite3), request types, and byte limits. Set up custom HTTP clients and callbacks for tracking event delivery.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/emitters/

Tracker instances must be initialized with an emitter. This section will go into more depth about the Emitter and how it works under the hood.

The simplest Emitter setup requires only the collector URI to be passed to it:

```go
emitter := sp.InitEmitter(RequireCollectorUri("com.acme"), sp.RequireStorage(*storagememory.Init()))
```

There are other optional builder functions:

| **Function Name**     | **Description**                                | **Required?** | **Default** |
| --------------------- | ---------------------------------------------- | ------------- | ----------- |
| `RequireCollectorUri` | The URI to send events to                      | Yes           | `nil`       |
| `RequireStorage`      | The storage integration to use                 | Yes           | `nil`       |
| `OptionRequestType`   | The request type to use (GET or POST)          | No            | `POST`      |
| `OptionProtocol`      | The protocol to use (http or https)            | No            | `http`      |
| `OptionSendLimit`     | The maximum amount of events to send at a time | No            | `500`       |
| `OptionByteLimitGet`  | The byte limit when sending a GET request      | No            | `40000`     |
| `OptionByteLimitPost` | The byte limit when sending a POST request     | No            | `40000`     |
| `OptionCallback`      | Defines a custom callback function             | No            | `nil`       |
| `OptionHttpClient`    | A custom HTTP client                           | No            | `&Client{}` |

A more complete example:

```go
emitter := sp.InitEmitter(
  sp.RequireCollectorUri("com.acme"),
  sp.RequireStorage(*storagememory.Init()),
  sp.OptionRequestType("GET"),
  sp.OptionProtocol("https"),
  sp.OptionSendLimit(50),
  sp.OptionByteLimitGet(52000),
  sp.OptionByteLimitPost(52000),
  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {
    log.Println("Successes: " + IntToString(len(g)))
    log.Println("Failures: " + IntToString(len(b)))
  }),
)
```

As of Version 3 of the Tracker you must pass RequireStorage to set the Storage option for the Emitter. Out of the box we support the following:

- **StorageMemory**: A fully in-memory implementation utilising `go-memdb`
- **StorageSQLite3**: A local disk implementation leveraging `go-sqlite3`

You can also define your own Storage system by implementing the `Storage` interface found in `pkg/storage/storageiface`:

```go
type Storage interface {
  AddEventRow(payload Payload) bool
  DeleteAllEventRows() int64
  DeleteEventRows(ids []int) int64
  GetAllEventRows() []EventRow
  GetEventRowsWithinRange(eventRange int) []EventRow
}
```

To use `StorageMemory` the constructor looks like so:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory"

emitter := sp.InitEmitter(
  sp.RequireCollectorUri("com.acme"),
  sp.RequireStorage(*storagememory.Init()),
  sp.OptionRequestType("GET"),
  sp.OptionProtocol("https"),
  sp.OptionSendLimit(50),
  sp.OptionByteLimitGet(52000),
  sp.OptionByteLimitPost(52000),
  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {
    log.Println("Successes: " + IntToString(len(g)))
    log.Println("Failures: " + IntToString(len(b)))
  }),
)
```

To leverage `StorageSQLite3`:

```go
import storagesqlite3 "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/sqlite3"

emitter := sp.InitEmitter(
  sp.RequireCollectorUri("com.acme"),
  sp.RequireStorage(*storagesqlite3.Init("dbname_path.db")),
  sp.OptionRequestType("GET"),
  sp.OptionProtocol("https"),
  sp.OptionSendLimit(50),
  sp.OptionByteLimitGet(52000),
  sp.OptionByteLimitPost(52000),
  sp.OptionCallback(func(g []CallbackResult, b []CallbackResult) {
    log.Println("Successes: " + IntToString(len(g)))
    log.Println("Failures: " + IntToString(len(b)))
  }),
)
```

### Under the hood

Once the emitter receives an event from the Tracker a few things start to happen:

- The event is added to the storage implementation (either `memory` or `sqlite3`)
- A long running go routine is started which will continue to send events as long as they can be found in the database (asynchronous)
- The emitter loop will grab a range of events from the database up until the `SendLimit` as noted \__above_
- The emitter will send all of these events as determined by the Request, Protocol and ByteLimits
  - Each request is sent in its own go routine.
- Once sent it will process the results of all the requests sent and will remove all successfully sent events from the database

**IF** all of the requests failed this loop is terminated eagerly; this is seen as a network failure so attempting to send is a waste of resources. **IF** there are no more events in the database the loop is terminated.

### Builder methods

#### `OptionHttpClient`

An HTTP client can be set with custom settings appropriate for the use-case, such as timeouts and other connection settings.

This method accepts a reference to [http.Client](https://golang.org/pkg/net/http/#Client).

---

# Golang tracker SDK
> The Snowplow Golang Tracker enables server-side event tracking from Go applications. Create subjects, emitters, and trackers with persistent storage to send behavioral events to your Snowplow collector.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/

The [Snowplow Golang Tracker](https://github.com/snowplow/snowplow-golang-tracker) allows you to track Snowplow events from your Golang apps and servers.

There are four basic types of object you will create when using the Snowplow Golang Tracker: subjects, emitters, storage and trackers.

- A subject represents a user whose events are tracked.
- A tracker constructs events and sends them to an emitter.
- The emitter then sends the event to the endpoint you configure, a valid Snowplow Collector, which leverages a storage implementation to store them securely before sending to allow for recovery from failure.

> **Note:** As sending and processing of events is done asynchronously it is advised to create the Tracker as a singleton object. This is due to the fact that all events are first persistently stored in a local Sqlite3 database; if multiple Trackers are created there is the possibility of duplicate sending of events and overt consumption of resources.
>
> See [here for instructions](http://blog.ralch.com/tutorial/design-patterns/golang-singleton/) on building a Singleton in Golang.

---

# Initialize the Golang tracker
> Create a Golang tracker with InitTracker using collector URI and storage implementation. Configure namespace, app ID, platform, and base64 encoding with optional builder functions.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/initialization/

Import the Golang Tracker library like so:

```go
import "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

You will need to refer to the package as `tracker`. If you wish to use something shorter (or if `tracker` is already taken):

```go
import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

The package can now be referred to as `sp` rather than `tracker`.

You will also need to import a `storage` implementation for the `emitter` - you can select from two that we have created or implement your own following the supplied interface.

The `in-memory` or `SQLite3` implementations can be pulled from the following packages:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory" // Maps to tracker.InitStorageMemory() in v2

OR

import storagesqlite3 "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/sqlite3" // Maps to tracker.InitStorageSQLite3(<dbname>) in v2
```

The `Storage` interface can be found in `github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/storageiface`.

That's it - you are now ready to initialize a tracker instance.

## Creating a tracker

The simplest tracker initialization only requires you to provide the URI of the collector to which the tracker will log events and a storage implementation:

```go
import storagememory "github.com/snowplow/snowplow-golang-tracker/v3/pkg/storage/memory"
import sp "github.com/snowplow/snowplow-golang-tracker/v3/tracker"

emitter := sp.InitEmitter(sp.RequireCollectorUri("com.acme"), sp.RequireStorage(*storagememory.Init()))
tracker := sp.InitTracker(sp.RequireEmitter(emitter))
```

There are other optional builder functions:

| **Function Name**    | **Description**                                                            | **Required?** | **Default** |
| -------------------- | -------------------------------------------------------------------------- | ------------- | ----------- |
| `RequireEmitter`     | The emitter to which events are sent                                       | Yes           | `nil`       |
| `OptionSubject`      | The user being tracked                                                     | No            | `nil`       |
| `OptionNamespace`    | The name of the tracker instance                                           | No            | \`\`        |
| `OptionAppId`        | The application ID                                                         | No            | \`\`        |
| `OptionPlatform`     | The platform the Tracker is running on                                     | No            | `srv`       |
| `OptionBase64Encode` | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `true`      |

A more complete example:

```go
subject := sp.InitSubject()
emitter := sp.InitEmitter(
  sp.RequireCollectorUri("com.acme"),
  sp.RequireStorage(*storagememory.Init()),
)
tracker := sp.InitTracker(
  sp.RequireEmitter(emitter),
  sp.OptionSubject(subject),
  sp.OptionNamespace("namespace"),
  sp.OptionAppId("app-id"),
  sp.OptionPlatform("mob"),
  sp.OptionBase64Encode(false),
)
```

### `RequireEmitter`

Accepts an argument of an Emitter instance pointer; if the object is `nil` will `panic`. See Emitters for more on emitter configuration.

### `OptionSubject`

The user which the Tracker will track. Accepts an argument of a Subject instance pointer.

You don't need to set this during Tracker construction; you can use the `tracker.SetSubject()` method afterwards. In fact, you don't need to create a subject at all. If you don't, though, your events won't contain user-specific data such as timezone and language.

### `OptionNamespace`

If provided, the `namespace` argument will be attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

### `OptionAppId`

The `appId` argument lets you set the application ID to any string.

### `OptionPlatform`

By default we assume the Tracker will be running in a server environment. To override this provide your own platform string.

### `OptionBase64Encode`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the Boolean `OptionBase64Encode` function with either `true` or `false` passed in.

---

# Install the Golang tracker
> Install the Golang tracker using go modules with support for Go 1.17.x through 1.19.x. Import the tracker package and start tracking server-side events from your Go applications.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/setup/

The latest version of the Snowplow Golang Tracker is v3.1.0 which has been built and tested using Golang versions `1.19.x`, `1.18.x` and `1.17.x`.

## Setup

All of the tracker code is hosted on [GitHub](https://github.com/snowplow/snowplow-golang-tracker) and versions can be fetched using Golang modules with both `v2` and `v3` available via this method.

To get the package, execute:

```bash
$host go get "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

To import the package, add the following line to your code:

```go
import "github.com/snowplow/snowplow-golang-tracker/v3/tracker"
```

---

# Track specific events with the Golang tracker
> Track behavioral events including SelfDescribing, ScreenView, PageView, EcommerceTransaction, Structured, and Timing events. Add custom contexts and optional timestamps to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/golang-tracker/tracking-specific-events/

Tracking methods supported by the Golang Tracker at a glance:

| **Function**                                                                                           | **Description**                                        |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| [`TrackSelfDescribingEvent()`](#track-selfdescribingunstructured-events-with-trackselfdescribingevent) | Track a Snowplow custom unstructured event             |
| [`TrackScreenView()`](#track-screen-views-with-trackscreenview)                                        | Track the user viewing a screen within the application |
| [`TrackPageView()`](#track-pageviews-with-trackpageview)                                               | Track and record views of web pages.                   |
| [`TrackEcommerceTransaction()`](#track-ecommerce-transactions-with-trackecommercetransaction)          | Track an ecommerce transaction                         |
| [`TrackStructEvent()`](#track-structured-events-with-trackstructevent)                                 | Track a Snowplow custom structured event               |
| [`TrackTiming()`](#track-timing-events-with-tracktiming)                                               | Track a timing event                                   |

> **Note:** All event structs require pointer values as a way of asserting properly whether or not a value has been passed that might have been required. As such there are three functions provided in `helper` package that allow you to inline pointer values:
>
> - `NewString`
> - `NewInt64`
> - `NewFloat64`
>
> These all accept their respective raw value and return a pointer to this value.
>
> To import the helper package:
>
> ```text
> import sphelp "github.com/snowplow/snowplow-golang-tracker/v3/pkg/common"
> ```

### Common

All events are tracked with specific methods on the tracker instance, of the form `TrackXXX()`, where `XXX` is the name of the event to track.

### Custom contexts

In short, custom contexts let you add additional information about the circumstances surrounding an event in the form of an array of SelfDescribingJson structs (`[]SelfDescribingJson`). Each tracking method accepts an additional optional contexts parameter, which should be a list of contexts:

```go
contextArray := []sp.SelfDescribingJson{}
```

If a visitor arrives on a page advertising a movie, the dictionary for a single custom context in the list might look like this:

```go
contextArray := []sp.SelfDescribingJson{
  *sp.InitSelfDescribingJson(
    "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
    map[string]interface{}{
      "movie_name": "Solaris",
      "poster_country": "JP",
    },
  ),
}
```

This is how to fire a page view event with the above custom context:

```go
tracker.TrackPageView(
  sp.PageViewEvent{
    PageUrl: sphelp.NewString("acme.com"),
    Contexts: contextArray,
  },
)
```

Note that even though there is only one custom context attached to the event, it still needs to be placed in an array.

### Optional timestamp argument

Each `Track...()` method supports an optional timestamp argument; this allows you to manually override the timestamp attached to this event. The timestamp should be in milliseconds since the Unix epoch.

If you do not pass this timestamp in as an argument, then the Golang Tracker will use the current time to be the timestamp for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{
  Category: sphelp.NewString("some category"),
  Action: sphelp.NewString("some action"),
  Timestamp: sphelp.NewInt64(1368725287000),
})
```

### Optional true-timestamp argument

Each `Track...()` method supports an optional true-timestamp argument; this allows you to provide the true-timestamp attached to this event to help with the timing of events in multiple timezones. The timestamp should be in milliseconds since the Unix epoch.

Here is an example tracking a structured event and supplying the optional true-timestamp argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{
  Category: sphelp.NewString("some category"),
  Action: sphelp.NewString("some action"),
  TrueTimestamp: sphelp.NewInt64(1368725287000),
})
```

### Optional event ID argument

Each `Track...()` method supports an optional event id argument; this allows you to manually override the event ID attached to this event. The event ID should be a valid version 4 UUID string.

Here is an example tracking a structured event and supplying the optional event ID argument:

```go
tracker.TrackStructEvent(sp.StructuredEvent{
  Category: sphelp.NewString("some category"),
  Action: sphelp.NewString("some action"),
  EventId: sphelp.NewString("486820fb-e722-4311-b33d-d2f319b511f6"),
})
```

### Optional Subject argument

Each `Track...()` method supports an optional Subject argument; this allows you to manually override the Subject information that is usually pulled from the `Tracker` object.

Here is an example of tracking a Page View event and supplying the optional Subject argument:

```go
eventSubject := sp.InitSubject()
eventSubject.SetUserId("987654321")

tracker.TrackPageView(sp.PageViewEvent{
	PageUrl:  sphelp.NewString("acme.com"),
	Subject:  eventSubject,
})
```

### Track SelfDescribing/Unstructured events with `TrackSelfDescribingEvent()`

Use `TrackSelfDescribingEvent()` to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**    | **Description**               | **Required?** | **Validation**        |
| --------------- | ----------------------------- | ------------- | --------------------- |
| `Event`         | The properties of the event   | Yes           | \*SelfDescribingJson  |
| `Timestamp`     | When the event occurred       | No            | \*int64               |
| `EventId`       | The event ID                  | No            | \*string              |
| `TrueTimestamp` | The true time of event        | No            | \*int64               |
| `Contexts`      | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject        | No            | Subject               |

Example:

```go
// Create a data map
data := map[string]interface{}{
  "level": 5,
  "saveId": "ju302",
  "hardMode": true,
}

// Create a new SelfDescribingJson
sdj := sp.InitSelfDescribingJson("iglu:com.example_company/save-game/jsonschema/1-0-2", data)

tracker.TrackSelfDescribingEvent(sp.SelfDescribingEvent{
  Event: sdj,
})
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

### Track screen views with `TrackScreenView()`

Use `TrackScreenView()` to track a user viewing a screen (or equivalent) within your app:

| **Argument**    | **Description**                     | **Required?** | **Type**              |
| --------------- | ----------------------------------- | ------------- | --------------------- |
| `Name`          | Human-readable name for this screen | No            | \*string              |
| `Id`            | Unique identifier for this screen   | No            | \*string              |
| `Timestamp`     | When the event occurred             | No            | \*int64               |
| `EventId`       | The event ID                        | No            | \*string              |
| `TrueTimestamp` | The true time of event              | No            | \*int64               |
| `Contexts`      | Custom contexts for the event       | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject              | No            | Subject               |

Although name and id are not individually required, at least one must be provided or the event will fail validation.

Example:

```go
tracker.TrackScreenView(sp.ScreenViewEvent{
  Id: sphelp.NewString("Screen ID"),
})
```

### Track pageviews with `TrackPageView()`

Use `TrackPageView()` to track a user viewing a page within your app:

| **Argument**    | **Description**                      | **Required?** | **Validation**        |
| --------------- | ------------------------------------ | ------------- | --------------------- |
| `PageUrl`       | The URL of the page                  | Yes           | \*string              |
| `PageTitle`     | The title of the page                | No            | \*string              |
| `Referrer`      | The address which linked to the page | No            | \*string              |
| `Timestamp`     | When the event occurred              | No            | \*int64               |
| `EventId`       | The event ID                         | No            | \*string              |
| `TrueTimestamp` | The true time of event               | No            | \*int64               |
| `Contexts`      | Custom contexts for the event        | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject               | No            | Subject               |

Example:

```go
tracker.TrackPageView(sp.PageViewEvent{
  PageUrl: sphelp.NewString("acme.com"),
})
```

### Track ecommerce transactions with `TrackEcommerceTransaction()`

Use `TrackEcommerceTransaction()` to track an ecommerce transaction:

| **Argument**    | **Description**                 | **Required?** | **Validation**                   |
| --------------- | ------------------------------- | ------------- | -------------------------------- |
| `OrderId`       | ID of the eCommerce transaction | Yes           | \*string                         |
| `TotalValue`    | Total transaction value         | Yes           | \*float64                        |
| `Affiliation`   | Transaction affiliation         | No            | \*string                         |
| `TaxValue`      | Transaction tax value           | No            | \*float64                        |
| `Shipping`      | Delivery cost charged           | No            | \*float64                        |
| `City`          | Delivery address city           | No            | \*string                         |
| `State`         | Delivery address state          | No            | \*string                         |
| `Country`       | Delivery address country        | No            | \*string                         |
| `Currency`      | Transaction currency            | No            | \*string                         |
| `Items`         | Items in the transaction        | Yes           | \[]EcommerceTransactionItemEvent |
| `Timestamp`     | When the event occurred         | No            | \*int64                          |
| `EventId`       | The event ID                    | No            | \*string                         |
| `TrueTimestamp` | The true time of event          | No            | \*int64                          |
| `Contexts`      | Custom contexts for the event   | No            | \[]SelfDescribingJson            |
| `Subject`       | Event specific Subject          | No            | Subject                          |

The `items` argument is an Array of TransactionItems. `TrackEcommerceTransaction` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `Items` list. Each transaction item event will have the same timestamp, true timestamp, order ID, and currency as the main transaction event.

These are the fields with which a TransactionItem can be created.

| **Field**  | **Description**               | **Required?** | **Validation**        |
| ---------- | ----------------------------- | ------------- | --------------------- |
| `Sku`      | Item SKU                      | Yes           | \*string              |
| `Price`    | Item price                    | Yes           | \*float64             |
| `Quantity` | Item quantity                 | Yes           | \*int64               |
| `Name`     | Item name                     | No            | \*string              |
| `Category` | Item category                 | No            | \*string              |
| `EventId`  | The event ID                  | No            | \*string              |
| `Contexts` | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`  | Event specific Subject        | No            | Subject               |

Example of tracking a transaction containing two items:

```go
items := []sp.EcommerceTransactionItemEvent{
  sp.EcommerceTransactionItemEvent{
    Sku: sphelp.NewString("pbz0026"),
    Price: sphelp.NewFloat64(20),
    Quantity: sphelp.NewInt64(1),
  },
  sp.EcommerceTransactionItemEvent{
    Sku: sphelp.NewString("pbz0038"),
    Price: sphelp.NewFloat64(15),
    Quantity: sphelp.NewInt64(1),
    Name: sphelp.NewString("red hat"),
    Category: sphelp.NewString("menswear"),
  },
}

tracker.TrackEcommerceTransaction(sp.EcommerceTransactionEvent{
  OrderId: sphelp.NewString("6a8078be"),
  TotalValue: sphelp.NewFloat64(35),
  Affiliation: sphelp.NewString("some-affiliation"),
  TaxValue: sphelp.NewFloat64(6.12),
  Shipping: sphelp.NewFloat64(30),
  City: sphelp.NewString("Dijon"),
  State: sphelp.NewString("Bourgogne"),
  Country: sphelp.NewString("France"),
  Currency: sphelp.NewString("EUR"),
  Items: items,
})
```

### Track structured events with `TrackStructEvent()`

Use `TrackStructEvent()` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**    | **Description**                                                  | **Required?** | **Validation**        |
| --------------- | ---------------------------------------------------------------- | ------------- | --------------------- |
| `Category`      | The grouping of structured events which this `action` belongs to | Yes           | \*string              |
| `Action`        | Defines the type of user interaction which this event involves   | Yes           | \*string              |
| `Label`         | A string to provide additional dimensions to the event data      | No            | \*string              |
| `Property`      | A string describing the object or the action performed on it     | No            | \*string              |
| `Value`         | A value to provide numerical data about the event                | No            | \*float64             |
| `Timestamp`     | When the event occurred                                          | No            | \*int64               |
| `EventId`       | The event ID                                                     | No            | \*string              |
| `TrueTimestamp` | The true time of event                                           | No            | \*int64               |
| `Contexts`      | Custom contexts for the event                                    | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject                                           | No            | Subject               |

Example:

```go
tracker.TrackStructEvent(sp.StructuredEvent{
  Category: sphelp.NewString("shop"),
  Action: sphelp.NewString("add-to-basket"),
  Property: sphelp.NewString("pcs"),
  Value: sphelp.NewFloat64(2),
})
```

### Track timing events with `TrackTiming()`

Use `TrackTiming()` to track a timing event.

The arguments are as follows:

| **Argument**    | **Description**               | **Required?** | **Validation**        |
| --------------- | ----------------------------- | ------------- | --------------------- |
| `Category`      | The category of the event     | Yes           | \*string              |
| `Variable`      | The variable of the event     | Yes           | \*string              |
| `Timing`        | The timing of the event       | Yes           | \*int64               |
| `Label`         | The label of the event        | No            | \*string              |
| `Timestamp`     | When the event occurred       | No            | \*int64               |
| `EventId`       | The event ID                  | No            | \*string              |
| `TrueTimestamp` | The true time of event        | No            | \*int64               |
| `Contexts`      | Custom contexts for the event | No            | \[]SelfDescribingJson |
| `Subject`       | Event specific Subject        | No            | Subject               |

Example:

```go
tracker.TrackTiming(sp.TimingEvent{
  Category: sphelp.NewString("Timing Category"),
  Variable: sphelp.NewString("Some var"),
  Timing: sphelp.NewInt64(124578),
})
```

---

# Google AMP tracker
> Integrate Snowplow tracking into Google AMP (Accelerated Mobile Pages) using amp-analytics. Track page views, structured events, page pings, and custom events from mobile-optimized AMP pages with cross-domain session linking.
> Source: https://docs.snowplow.io/docs/sources/google-amp-tracker/

Snowplow is collaborating with Google on their [Accelerated Mobile Pages Project](https://www.ampproject.org/) (AMPP or AMP for short). AMP is an open source (Apache License 2.0) initiative to improve the mobile web experience by optimizing web pages for mobile devices.

To learn more about analytics for AMP pages see the [amp-analytics](https://www.ampproject.org/docs/reference/extended/amp-analytics.html) reference. For general information about AMP see [What Is AMP?](https://www.ampproject.org/docs/get_started/about-amp.html) on the [Accelerated Mobile Pages (AMP) Project](https://www.ampproject.org/) site.

Snowplow is natively integrated into the project, so pages optimized with AMP HTML can be tracked in Snowplow by adding the appropriate `amp-analytics` tag to your pages:

```html

<body>
  ...
  <amp-analytics type="snowplow_v2" id="snowplow_v2">
  <script type="application/json">
  {
    "vars": {
     "collectorHost": "collector.snplow.net",
     "appId": "amp-examples-v2.0.2",
     ...
    },
    "linkers": {
     "enabled": true,
     ...
    },
    "triggers": {
     "defaultPageview": {
       "on": "visible",
       "request": "pageView"
     }
    }
  }
  </script>
  </amp-analytics>
</body>
```

> **Note:** The tracker utilises AMP linker functionality to ensure that user identification can be done where a user may visit via the google domain or the site's own domain. In order to function, this requires that linkers are enabled in the amp-analytics configuration. Not doing so can result in changing AMP client Ids (which are the primary user identifier).

## Standard variables

`collectorHost` and `appId` must be provided in the `"vars"` section of the tag:

```javascript
"vars": {
  ...
},
```

The rest are optional

### `collectorHost`

Specify the host to your collector like so:

```javascript
"vars": {
  "collectorHost": "snowplow-collector.acme.com",
  ...
```

Notes:

- Do _not_ include the protocol aka schema (`http(s)://`)
- Do _not_ include a trailing slash
- Use of HTTPS is mandatory in AMP, so your Snowplow collector **must** support HTTPS

### `appId`

You must set the application ID for the website you are tracking via AMP:

```javascript
"vars": {
  "appId": "campaign-microsite",
  ...
```

Notes:

- You do not have to use the `appId` to distinguish AMP traffic from other web traffic (unless you want to) - see the [Analytics](#analytics) section for an alternative approach.

### `userId`

Specify the optional `"userId"` var to set the uid/user\_id [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#user-fields) field.

```javascript
"vars": {
  "userId": "someUserId",
  ...
```

### `nameTracker`

Specify the optional "nameTracker" var to set the tna/name\_tracker [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields) field.

```javascript
"vars": {
  "nameTracker": "someTrackerName",
  ...
```

### `customContexts`

Custom contexts may be added by including Self-Describing JSON as a `"customContexts"` var, with `"` characters escaped:

```javascript
"vars": {
  "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}}"
  ...
```

Multiple custom contexts may be added by separating each self-describing JSON with a comma:

```javascript
"vars": {
  "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}},{\"schema\":\"iglu:com.acme/second_context/jsonschema/1-0-0\",\"data\":{\"someOtherKey\":\"someOtherValue\"}}"
  ...
```

Custom contexts may either be set globally for all events (as a top-level var, see above) or per-event trigger:

```javascript
...
"triggers": {
...
  "defaultPageview": {
    "on": "click",
    "selector": "visible",
    "request": "pageView",
    "vars": {
      "customContexts":  "{\"schema\":\"iglu:com.acme/first_context/jsonschema/1-0-0\",\"data\":{\"someKey\":\"someValue\"}}"
    }
  }
}
...
```

These approaches may not currently be mixed, however.

## Tracking events

The following trigger request values are supported for the Snowplow Analytics configuration:

- `pageView` for page view tracking
- `structEvent` for structured event tracking
- `ampPagePing` for page ping tracking
- `selfDescribingEvent` for custom event tracking

All event tracking is disabled by default; you can enable it on an event-by-event basis as follows:

### Page view

Enable the page view tracking like so:

```javascript
<amp-analytics type="snowplow_v2" id="event1">
<script type="application/json">
{
  "vars": {
    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host
    "appId": "campaign-microsite"                    // Replace with your app ID
  },
  "triggers": {
    "trackPageview": {  // Trigger names can be any string. trackPageview is not a required name
      "on": "visible",
      "request": "pageView"
    }
  }
}
</script>
</amp-analytics>
```

### Structured events

Structured events are user interactions with content that can be tracked independently from a web page or a screen load. "Structured" refers to the Google Analytics-style structure of having up to five fields (with only the first two required).

Events can be sent by setting the AMP trigger request value to event and setting the required event category and action fields.

The following example uses the selector attribute of the trigger to send an event when a particular element is clicked:

```javascript
<amp-analytics type="snowplow_v2" id="event2">
<script type="application/json">
{
  "vars": {
    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host
    "appId": "campaign-microsite"                    // Replace with your app ID
  },
  "triggers": {
    "trackClickOnHeader" : {
      "on": "click",
      "selector": "#header",
      "request": "structEvent",
      "vars": {
        "structEventCategory": "ui-components",
        "structEventAction": "header-click"
      }
    }
  }
}
</script>
</amp-analytics>
```

You can set key-value pairs for the following event fields in the vars attribute of the trigger:

| **Argument**          | **Description**                                                  | **Required?** | **Validation** |
| --------------------- | ---------------------------------------------------------------- | ------------- | -------------- |
| `structEventCategory` | The grouping of structured events which this `action` belongs to | Yes           | String         |
| `structEventAction`   | Defines the type of user interaction which this event involves   | Yes           | String         |
| `structEventLabel`    | A string to provide additional dimensions to the event data      | No            | String         |
| `structEventProperty` | A string describing the object or the action performed on it     | No            | String         |
| `structEventValue`    | A value to provide numerical data about the event                | No            | Int or Float   |

### Page pings

Enable page ping tracking like so:

```javascript
<amp-analytics type="snowplow_v2" id="event3">
<script type="application/json">
{
  "vars": {
    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host
    "appId": "campaign-microsite"                    // Replace with your app ID
  },
  "triggers": {
    "trackPagePings": {
      "on": "timer",
      "request": "ampPagePing",
      "timerSpec": {
        "interval": 30,      // Set to desired ping interval
        "maxTimerLength": 1800,
        "immediate": false,
        "startSpec": {
          "on": "visible",
          "selector": ":root"
        },
        "stopSpec": {
          "on": "hidden",
          "selector": ":root"
        }
      }
    }
  }
}
</script>
</amp-analytics>
```

AMP page ping events will be sent as AMP-specific page ping events (rather than Javascript tracker page pings), against the [AMP page ping schema](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_page_ping/jsonschema/1-0-0), since the data available on AMP is defined differently to Javascript tracker page pings.

All events are sent with an [AMP web page](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_web_page/jsonschema/1-0-0) context, for aggregation of pings by page view id.

### Custom events

Custom events may be sent via the AMP tracker by passing the schema vendor, and version, and an escaped JSON of the desired data, as follows:

```javascript
<amp-analytics type="snowplow_v2" id="event3">
<script type="application/json">
{
  "vars": {
    "collectorHost": "snowplow-collector.acme.com",  // Replace with your collector host
    "appId": "campaign-microsite"                    // Replace with your app ID
  },
  "triggers": {
    "trackSelfDescribingEvent": {
      "on": "click",
      "selector": "a",
      "request": "selfDescribingEvent",
      "vars": {
        "customEventSchemaVendor": "com.snowplowanalytics.snowplow",
        "customEventSchemaName": "link_click",
        "customEventSchemaVersion": "1-0-1",
        "customEventSchemaData": "{\"targetUrl\":\"${targetUrl}\",\"elementId\":\"TEST\",\"elementContent\":\"${elementContent}\"}"
      }
    }
  }
}
</script>
</amp-analytics>
```

## Analytics

v2 of the tracker brings with it significant improvements in our ability to model and gain insights.

### Standard fields

All events sent via this tracker will have:

- `v_tracker` set to `amp-1.1.0`
- `platform` set to `web`

If you want to analyze events sent via this tracker, you may prefer to query for `v_tracker LIKE 'amp-%'` to future-proof your query against future releases of this tracker (which may change the version number).

### Page view and ping aggregation

By default, the [AMP web page context](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_web_page/jsonschema/1-0-0) is attached to every event. This will contain the AMP-defined [PAGE\_VIEW\_ID\_64](https://github.com/ampproject/amphtml/blob/master/spec/amp-var-substitutions.md#page-view-id-64), which defined as "intended to be random with a high entropy and likely to be unique per URL, user and day".

Users can aggregate page views, page pings and other events on-page by this ID to aggregate engaged time, and model events to a page view level, by combining it with the url, amp client ID, and date.

Note that page pings and the page view ID itself are not defined by Snowplow's logic, but by what's made available by AMP - therefore applying the same logic to this data as that produced by the Javascript tracker is liable to produce different results.

### Session information

> **Note:** Available from version 1.1.0.

By default the [AMP Session](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_session/jsonschema/) context is attached to every event. This context allows for tracking information related to session analytics capabilities, as implemented in the [AMP framework](https://github.com/ampproject/issues/3399). The attributes included are the following:

- `ampSessionId`: An identifier for the AMP session.
- `ampSessionIndex`: The index of the current session for this user.
- `sessionEngaged`: If there has been any kind of user engagement in the AMP session. Engagement in this context means if the page is visible, has focus and is in the foreground.
- `sessionCreationTimestamp`: Timestamp at which the session was created in milliseconds elapsed since the UNIX epoch.
- `lastSessionEventTimestamp`: Timestamp at which the last event took place in the session in milliseconds elapsed since the UNIX epoch.

### User Identification

By default, the [AMP ID](https://github.com/snowplow/iglu-central/blob/master/schemas/dev.amp.snowplow/amp_id/jsonschema/1-0-0) context is attached to every event. This contains the [AMP Client ID](https://github.com/ampproject/amphtml/blob/master/spec/amp-var-substitutions.md#client-id), the `user_id` (if set via the `userId` var), and the domain\_userid (if passed to an AMP page via cross-domain linking - more detail below).

This provides a map between the main relevant identifiers, which can be used to model user journeys across platforms. Users can choose to instrument further user identification methods using custom contexts.

In order for the AMP Client ID to behave as expected, the 'linker' parameter of the configuration JSON must have `'enabled': true` set. Failing to do so can result in changing AMP client IDs, even where a user remains on AMP pages for the whole journey. (This can happen because AMP pages can be served from Google's AMP cache).

The tracker is designed to handle user journeys as follows:

#### JS-tracker page to AMP page

Where a user moves from a standard web page, tracked by the Javascript tracker, to an AMP page, the domain userid from the Javascript tracker can be passed to the AMP tracker by enabling the [Javascript tracker's crossDomainLinker](/docs/sources/web-trackers/tracker-setup/initialization-options/). The AMP tracker will parse the value from the querystring, and attach it to all events, along with the AMP client ID, via the AMP ID context.

The AMP tracker uses a combination of cookies and the AMP linker to attempt to retain the value, however due to the nature of AMP pages, there is no guarantee that the value will be retained across sessions. To ensure best possible retention of the value within the session, make sure the tracker config has linker pages enabled for your AMP domains:

```javascript
...
"linkers": {
  "enabled": true,
  "proxyOnly": false,
  "destinationDomains": ["ampdomain"]
},
...
```

Data models should be designed for an 'at least once' identification structure - in other words, configuring the cross-domain linker correcttly will ensure at least one event contains both AMP ID and domain User ID. Models should aim to map this across all events. Of course, this method of identification depends on the user traveling directly from a JS-tracked page to an AMP page at least once.

#### AMP page to JS-tracker page

Where a user moves frrom an AMP page to a standard web page which is tracked by the Javascript tracker, the AMP tracker will use AMP's linker functionality to append the AMP CLient ID to the querystring, as long as linkers are enabled for the destination domain:

```javascript
...
"linkers": {
  "enabled": true,
  "proxyOnly": false,
  "destinationDomains": ["destDomain"]
},
...
```

This will add a querystring parameter 'linker=' to the destination url, which contains the amp\_id value, base-64 encoded. This will look something like this: `?sp_amp_linker=1*1c1wx43*amp_id*amp-a1b23cDEfGhIjkl4mnoPqr`.

The structure of this param is explained in the [AMP documentation](https://github.com/ampproject/amphtml/blob/master/extensions/amp-analytics/linker-id-receiving.md) - models can extract the base64-encoded AMP Client ID, decode it, and map it to the domain userid (or any other user value) from the Javascript tracker.

## Reporting Issues and Contributing

A fork of the AMP project can be found in the [Snowplow Incubator Github](https://github.com/snowplow-incubator/amphtml) Repo. Please submit issues, bugreports and PRs to this repo.

---

# Configure the Snowplow Ecommerce Tag
> Configure the Snowplow Ecommerce Tag Template in GTM using native Snowplow Ecommerce API or transitional GA4/UA adapter APIs. Set tracking parameters, custom contexts, and product data for ecommerce event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/ecommerce-tag-template/configuration/

Use the native Snowplow Ecommerce API or [transitional GA4/UA ecommerce adapter APIs](/docs/sources/web-trackers/tracking-events/ecommerce/#ga4ua-ecommerce-transitional-api) for existing dataLayer implementations using those formats. To get full value from the [Snowplow Ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/) we recommend using the native API when possible.

![](/assets/images/01_ecommerce_api-5ccc2f11a8b43b04dbc188a27da29ea1.png)

## Tracking Parameters

**Snowplow Ecommerce:**

![](/assets/images/02_sp_tracking_parameters-79d633f456f8c14286ec3ed73e2fd3fb.png)

#### Snowplow Ecommerce Function

In this section you can select the [Snowplow Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### Snowplow Ecommerce Argument

In this textbox you can specify the argument to the ecommerce function. This can be a Variable that evaluates to a corresponding object.

#### Additional Tracking Parameters

##### Add Custom Context Entities

Use this table to attach [custom context entities](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-entity) to the Snowplow event. Each row can be set to a Google Tag Manager variable that returns an array of custom contexts to add to the event hit.

##### Set Custom Timestamp

Set this to a UNIX timestamp in case you want to [override the default timestamp](/docs/sources/web-trackers/tracking-events/#custom-timestamp) used by Snowplow.

**GA4 Ecommerce:**

![](/assets/images/02_ga4_tracking_parameters-62661b55514ba47b36edf5fbc50c3282.png)

#### GA4 Ecommerce Function

In this section you can select the [Google Analytics 4 Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### GA4 Ecommerce Arguments

##### DataLayer ecommerce

Here you can specify the dataLayer ecommerce variable to use, i.e. a variable that returns the `ecommerce` object itself.

##### Options object

Here you can specify a variable returning an object holding additional information for the ecommerce event (e.g. including `currency`, `finalCartValue`, `step`, etc).

**Universal Analytics Enhanced Ecommerce:**

![](/assets/images/02_ua_tracking_parameters-1c48c247c61794f48eac816ad435b34c.png)

#### Universal Analytics Enhanced Ecommerce Function

In this section you can select the [Universal Analytics Enhanced Ecommerce function](/docs/sources/web-trackers/tracking-events/ecommerce/) to use.

#### Universal Analytics Enhanced Ecommerce Arguments

##### DataLayer ecommerce

Here you can specify the dataLayer ecommerce variable to use.

##### Options object

Here you can specify a variable returning an object holding additional information for the ecommerce event (e.g.including currency, finalCartValue, step etc).

***

## Snowplow Tracker and Ecommerce Plugin Settings

![](/assets/images/04_tracker_plugin_settings-383e984e9035913a2f284e466f2723fe.png)

### Tracker Settings

The Snowplow Ecommerce tag template **requires** a Snowplow Settings Variable to be setup. In this section you can select the Google Tag Manager variable of type [Snowplow Settings](/docs/sources/google-tag-manager/quick-start/) to use.

### Plugin Settings

In this section you can select how the plugin will be added. The available options are:

- `jsDelivr`: To get the plugin URL from jsDelivr CDN. Choosing this option allows you to specify the plugin version to be used.
- `unpkg`: To get the plugin URL from unpkg CDN. Choosing this option allows you to specify the plugin version to be used.
- `Self-hosted`: To get the plugin library from a specified URL. This option requires a [Permission](https://developers.google.com/tag-platform/tag-manager/templates/permissions) change to allow injecting the plugin script from the specified URL.
- `Do not add`: To not add the plugin (e.g. when using a [Custom Bundle](/docs/sources/web-trackers/plugins/configuring-tracker-plugins/) with the plugin already included). The default plugins bundled with the JavaScript tracker has changed from v3 to v4. Ensure that all plugins that you require are included. A list of the default plugins is available [here](/docs/sources/web-trackers/plugins/).

---

# Snowplow GTM Ecommerce template
> Install and configure the Snowplow Ecommerce Tag Template in Google Tag Manager to track product views, transactions, cart actions, and checkout events using the Snowplow Ecommerce plugin for v3 and v4 trackers.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/ecommerce-tag-template/

The Ecommerce Template is a separate Tag Template that can be added to your GTM workspace to track ecommerce events. This was done to keep the main Snowplow Tag Template a bit lighter, and for ease of tag management.

This template implements the [Snowplow Ecommerce plugin](/docs/sources/google-tag-manager/ecommerce-tag-template/) for the Snowplow JavaScript tracker and can be used alongside either [v3](/docs/sources/google-tag-manager/previous-versions/) or [v4](/docs/sources/google-tag-manager/) of the Snowplow tag.

## Template Installation

### Tag Manager Template Gallery

Search for "Snowplow Ecommerce v3" in the [Tag Manager Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-ecommerce-v3) and click `Add to Workspace`.

### Manual Installation

1. Download [template.tpl](https://github.com/snowplow/snowplow-gtm-tag-template-ecommerce-v3)
2. Create a new Tag template in the Templates section of your GTM container
3. Click the More Actions menu and select Import
4. Import the `template.tpl` file downloaded in Step 1
5. Click Save

## Tag Setup

With the template installed, you can now add the Snowplow Ecommerce Tag to your GTM Container.

1. From the Tag tab, select `New`, then select the Snowplow Ecommerce Tag as your tag type
2. Select your desired Trigger for the ecommerce events you want to track
3. [Configure the Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/configuration/)
4. Click Save

---

# Snowplow Google Tag Manager templates
> Deploy the Snowplow JavaScript tracker through Google Tag Manager using custom templates for v4. Configure event tracking, ecommerce, and custom variables with server-side and client-side GTM tags.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/

Using the Snowplow GTM custom templates you can deploy, implement, and configure the Snowplow [JavaScript tracker](/docs/sources/web-trackers/) directly on the website using Google Tag Manager.

The main Tag template that you will need to use when setting up the JavaScript Tracker v4 in GTM is available in the [Tag Manager Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4). To setup the Snowplow v4 Tag, you will also need the Snowplow v4 Settings Variable template. The templates you will need are:

1. [Snowplow v4](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4): Load, configure, and deploy the Snowplow JavaScript tracker library. It supports the full functionality of the JavaScript SDK.
2. [Snowplow v4 Settings](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-variable-template-v4): A variable template which can be used to easily apply a set of tracker configuration parameters to tags created with the Snowplow v4 tag template.

For Ecommerce tracking, the [Snowplow Ecommerce Tag](https://github.com/snowplow/snowplow-gtm-tag-template-ecommerce-v3) is available on GitHub.

---

# Quick start guide for using Snowplow in Google Tag Manager
> Add Snowplow v4 custom templates to your Google Tag Manager workspace and configure basic tracking. Import templates from the GTM gallery and set up your collector endpoint for event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/quick-start/

This guide will walk you through the initial setup for Snowplow in Google Tag Manager.

## Adding the Templates

To get started with Snowplow in Google Tag Manager, you will need to add the Snowplow Tag Template and the Snowplow Settings Variable Template to your GTM workspace.

Links to both templates can be found [here](/docs/sources/google-tag-manager/).

### Snowplow Tag Template

1. Navigate to the `Templates` tab in your GTM workspace and click the `Search Gallery` button in the `Tag Templates` section.

![](/assets/images/tag-template-search-0ec39a0b52c7894a89449492ad0308ce.png)

2. Search for "Snowplow v4"

![](/assets/images/search-27677eb5bfee0e80f905e1f996b157ce.png)

3. Click on the template, and then click `Add to Workspace` in the next screen. Review the permissions and click `Add` to finalize the import.

### Snowplow Settings Variable Template

The Snowplow Settings Variable template is used to configure the Snowplow tracker, such as the collector endpoint, privacy options, and tracker version. Although possible to use the Snowplow tag without the settings variable, it's highly recommended to use it for ease of configuration, along with keeping the tracker configuration separate from the tag.

1. Again in the `Templates` tab in your GTM workspace, click the `Search Gallery` button in the `Variable Templates` section.

![](/assets/images/variable-template-search-2d20f633b21d310f8664a688d210f0d3.png)

2. Search for "Snowplow v4 Settings"

3. Click on the template, and then click `Add to Workspace` in the next screen. No permissions are required, so click `Add` to finalize the import.

## Configuring the Settings Variable

1. Navigate to the `Variables` tab in your GTM workspace and click `New` in `User-Defined Variables`.

![](/assets/images/variables-new-c32b98d0e9bac37687e92d6234177414.png)

2. Select `Snowplow v4 Settings` from the list of available variables.

3. Under `Tracker Options`, enter your Snowplow collector endpoint set up when [configuring your collector](/docs/pipeline/collector/).

![](/assets/images/variable-9147f9ce6c3ee4fe6ed7770f3eee1fa7.png)

> **Info:** You might consider using conditional variables to set the Collector endpoint based on the environment, e.g. sending data to [Snowplow Micro](/docs/testing/snowplow-micro/) during development.

4. Under `JavaScript Tracker`, choose a hosting option. To get started quickly, select either `unpkg` or `jsDelivr` and enter a library version.

5. Give your variable a name and click `Save`.

## Implementing the Snowplow Tag

In this section, we will create a simple tag to fire a page view event.

1. Navigate to the `Tags` tab in your GTM workspace and click `New`.

![](/assets/images/new-tag-e29e5a0837ec41a24d07c0bf2c89e5af.png)

2. Click on the `Tag Configuration` section and select `Snowplow v4`.

3. Set the `Tag Type` to `Page View`, if it is not already selected.

4. Under `Tracker Initialisation`, select the Snowplow Settings variable we created earlier.

5. Add a trigger to the tag. This will determine when the tag is fired. For a page view tag, you can use the built-in `All Pages` trigger.

6. Give your tag a name and click `Save`.

## Testing the Tag

To test the tag, you can use the GTM preview mode. Click the `Preview` button in the top right of the GTM interface. This will open a new tab with your website and the GTM preview console.

Ensure that you see the Page View event in your Snowplow pipeline. If you don't have a full pipeline set up yet, you can use [Snowplow Micro](/docs/testing/snowplow-micro/) or [Snowplow Inspector](/docs/testing/snowplow-inspector/) to check that the event is sent correctly.

---

# Snowplow GTM Settings template
> Configure the Snowplow Settings Variable template for GTM with tracker options, collector endpoints, privacy settings, cookies, dispatching methods, and predefined context entities for consistent tracking configuration.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/settings-template/

This page describes the settings available in the Snowplow Settings Variable template for Google Tag Manager.

## Tracker Options

### Tracker Name

It is important to set the tracker name. The reason you might have more than one tracker name generated on the site is if you have different configuration objects or tracking endpoints to which you want to send commands.

When the tag runs, it first checks if a tag with this name has already been initialized. If it has, it then proceeds to send the command to this tracker name. If a tracker with this name has _not_ been initialized, a new tracker is initialized with the tracker configuration from this settings variable.

This means that a tracker configuration is applied **only once** to the tracker. Thus if you have more than one tag running on the site, each with the same tracker name but different tracker configurations, only the configuration of the tag that fires _first_ will be applied to the tracker.

### Collector Endpoint Hostname

This needs to be set to the hostname/domain (e.g. `sp.domain.com`) on which you’ve configured your [Snowplow Collector](/docs/pipeline/collector/)

## JavaScript Tracker

### Snowplow JavaScript Tracker Library

This determines the source of the Snowplow JavaScript tracker library. You can choose to load the tracker from a CDN or host it on your own server. For production usage, we recommend [self-hosting the tracker](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/).

### Self-Hosted Library URL

This field is required if you choose to load the Snowplow JavaScript tracker from your own server. Enter the URL of the tracker library here.

> **Warning:** The default Tag doesn't have the permission to inject scripts from a custom URL.
>
> You will need to update the `Injects Scripts` permission to reflect the new location, by editing the `Snowplow Analytics v3/v4 Tag` template. Delete the content of the `Allowed URL Match Patterns` field, and type the full URL to the library there. Again, it must match what you input into the tag itself when creating it.
>
> ![modifying permissions](/assets/images/modifying_permissions-d485193d56ce3212e8c83daea275800b.png)
>
> Modifying permissions **breaks the gallery link** and you will no longer be notified about updates to the template.
>
> ![modifying permissions breaks gallery link](/assets/images/modifying_breaks_gallery_link-7870229aea74cb186cc082ab6b9b2c36.png)

> **Note:** Since v1.1.0, an alternative to prevent breaking the gallery update link is to use the `Do not load` option from the corresponding drop down menu:
>
> ![library host drop down \&#39;Do not load\&#39; option](/assets/images/host_drop_down_no_load-06e54c13559cb4115d3d5c4577540b8f.png)

### Library Version

This field is required if you choose to load the Snowplow JavaScript tracker from a CDN. Enter the version of the tracker library you want to load here.

## Application Settings

### Application ID

This is the unique identifier for your application. It is used to distinguish different applications in your Snowplow pipeline.

### Platform

This is the platform on which your application is running. This is used to distinguish different platforms in your Snowplow pipeline.

## Privacy Settings

### Respect 'Do Not Track'

This setting allows you to respect the Do Not Track setting in the user's browser. When enabled, the Snowplow JavaScript tracker will not track users who have enabled the Do Not Track setting in their browser, along with preventing any cookies from being set.

### Anonymous Tracking

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

#### Server Anonymisation

Server-side anonymisation affects user identifiers set server-side. In particular, these are the network\_userid property set in server-side cookie and the user IP address.

Setting the flag will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow collector. The Snowplow pipeline will take care of anonymising the identifiers.

#### Anonymous Session Tracking

This setting disables client-side user identifiers but tracks session information. In practice, this means that events track the Session context entity but the userId property is a null UUID (00000000-0000-0000-0000-000000000000). In case Platform context is enabled, the IDFA identifiers will not be present.

#### Cookie Lifetime Extension Service

This allows you to set the endpoint for the [Cookie Lifetime Extension Service](/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/).

## Cookie Settings

### State Storage Strategy

This setting allows you to choose the strategy for storing the Snowplow tracker state. The available options are:

- `Cookie and Local Storage`: The Snowplow tracker will store the state in both cookies and local storage.
- `Cookie`: The Snowplow tracker will store the state in cookies only.
- `Local Storage`: The Snowplow tracker will store the state in local storage only.
- `None`: The Snowplow tracker will not store the state.

### Cookie Domain

This setting allows you to specify the domain for which the Snowplow tracker cookies will be set. This is useful when you want to track users across subdomains.

By default, `auto` will be used, which will set the domain to the root domain.

See [here](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-domain) for more information.

### Cookie Lifetime

This setting allows you to specify the lifetime of the Snowplow tracker cookies. By default, the cookies will be active for 2 years.

If the lifetime is set to `0`, the cookie will expire at the end of the session (when the browser closes). If set to `-1`, the first-party cookies will be disabled.

### Cookie SameSite

This setting allows you to specify the SameSite attribute for the Snowplow tracker cookies. The SameSite attribute is used to prevent CSRF attacks.

### Session Cookie Timeout

This setting allows you to specify the timeout for the session cookie. By default, the session cookie will expire after 30 minutes of inactivity.

### Synchronously Write Cookies

This setting allows you to specify whether the Snowplow tracker should [write cookies synchronously](/docs/sources/web-trackers/configuring-how-events-sent/#synchronous-cookie-writes). By default, the tracker will write cookies asynchronously.

## Dispatching

### Common

#### Dispatch Method

This setting allows you to choose the method for sending events to the Snowplow collector. The available options are:

- `POST`
- `GET`

It is recommended to use the default `POST` method for sending events to the collector as it allows for larger payloads, unless you have specific requirements for using the `GET` method.

#### Encode Into Base64

This setting allows you to encode the payload into Base64 before sending it to the collector. If you are using the `GET` method for sending events to the collector, it is recommended to enable this setting, as it will help prevent issues with special characters in the payload. Otherwise, it is recommended to leave this setting disabled to reduce the payload size.

#### Connection Timeout

This setting allows you to specify the timeout for the connection to the Snowplow collector. By default, the timeout is set to 5000 milliseconds.

### `POST` Specific

#### Buffer Size

This setting allows you to specify the number of events to buffer before sending them to the collector. By default, the buffer size is set to 1.

If you set the buffer size to a value greater than 1, the tracker will buffer events and send them in batches to the collector. Although this can help reduce the number of requests made to the collector, it comes at the expense of potential data loss for non-returning visitors.

#### POST Path

This setting allows you to specify the path to which the events will be sent. By default, the events will be sent to the `/com.snowplowanalytics.snowplow/tp2` path.

#### Maximum POST Payload Size

This setting allows you to specify the maximum size of the payload that will be sent to the collector. By default, the maximum payload size is set to 40000 bytes.

If an event is generated that is over the maximum payload size, the event will bypass the buffer and be sent immediately to the collector. This means that if it fails, it will not be retried.

#### Enable keepalive

This setting allows you to enable or disable the [keepalive](/docs/sources/web-trackers/configuring-how-events-sent/#keepalive-option-for-collector-requests) feature. This will enable requests to continue to be sent, even if the user navigates away from the page that sent the request.

Defaults to `false`.

## Predefined Contexts

Predefined contexts provide additional metadata for your Snowplow events. By including these contexts, you can capture common data points like device information, session details, or geolocation without having to define them manually.

Available predefined contexts are:

| Name                          | Description                                                                                                                              | Source Plugin                                                                                                |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `webPage`                     | Information about the web page where the event occurred.                                                                                 | [Web Page tracking](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity) |
| `gaCookies`                   | Information about the Google Analytics cookies.                                                                                          | [Google Analytics Cookies Plugin](/docs/sources/web-trackers/tracking-events/ga-cookies/)                    |
| `clientHints`                 | Information about the client's device.                                                                                                   | [Client Hints Plugin](/docs/sources/web-trackers/tracking-events/client-hints/)                              |
| `geolocation`                 | Information about the client's geolocation.                                                                                              | [Geolocation](/docs/events/ootb-data/geolocation/)                                                           |
| `session`                     | Information about the user session.                                                                                                      | [Session](/docs/events/ootb-data/user-and-session-identification/#session-entity)                            |
| `performanceNavigationTiming` | Retrieves data from the [PerformanceNavigationTiming](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming) API. | [Performance Navigation Timing](/docs/sources/web-trackers/tracking-events/timings/)                         |

---

# Snowplow GTM template
> Configure Snowplow v4 tag types in Google Tag Manager including ad tracking, button clicks, cart events, site search, timing, enhanced consent, ecommerce, error tracking, page views, link clicks, and form tracking with custom commands.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/snowplow-template/

This template implements the [Snowplow JavaScript tracker](/docs/sources/web-trackers/) for Google Tag Manager. It allows for the sending of [Snowplow events](/docs/events/) from your website to your Snowplow collector.

Tag Types are the kinds of events that can be tracked with the Snowplow v4 Tag Template. Each tag type has its own set of options and parameters that can be configured.

You can also configure [plugins](/docs/sources/google-tag-manager/snowplow-template/plugins/) to use with this template.

## Ad Tracking

The Ad Tracking tag is used to track impressions and ad clicks. This can used by, for example, ad networks to identify which sites and web pages users visit across a network, so that they can be segmented.

**Ad Tracking Parameters**

All ad tracking events take the following common parameters:

| Name           | Required? | Description                                                            | Example |
| -------------- | --------- | ---------------------------------------------------------------------- | ------- |
| `advertiserId` | No        | The advertiser ID                                                      | 201     |
| `campaignId`   | No        | The campaign ID                                                        | 12      |
| `cost`         | No        | The cost of the ad                                                     | 5.5     |
| `costModel`    | No        | The cost model for the campaign. Must be one of `cpc`, `cpm`, or `cpa` | cpc     |

The ad tracking tag includes three event types, each with its own set of additional parameters:

### Impression Event

| Name           | Required? | Description                                                      | Example                   |
| -------------- | --------- | ---------------------------------------------------------------- | ------------------------- |
| `impressionId` | No        | Identifier for the particular impression instance                | 67965967893               |
| `targetUrl`    | No        | The destination URL                                              | <https://www.example.com> |
| `bannerId`     | No        | Adserver identifier for the ad banner (creative) being displayed | 23                        |
| `zoneId`       | No        | Adserver identifier for the zone where the ad banner is located  | 7                         |

### Click Event

| Name        | Required? | Description                                                      | Example                   |
| ----------- | --------- | ---------------------------------------------------------------- | ------------------------- |
| `targetUrl` | Yes       | The destination URL                                              | <https://www.example.com> |
| `clickId`   | No        | Identifier for the particular click instance                     | 12243253                  |
| `bannerId`  | No        | Adserver identifier for the ad banner (creative) being displayed | 23                        |
| `zoneId`    | No        | Adserver identifier for the zone where the ad banner is located  | 7                         |

### Conversion Event

| Name           | Required? | Description                                       | Example   |
| -------------- | --------- | ------------------------------------------------- | --------- |
| `conversionId` | No        | Identifier for the particular conversion instance | 743560297 |
| `category`     | No        | Conversion category                               | ecommerce |
| `action`       | No        | The type of user interaction                      | purchase  |
| `property`     | No        | Describes the object of the conversion            | shoes     |
| `initialValue` | No        | How much the conversion is initially worth        | 99        |

## Button Click Tracking

This tag will enable the tracking of clicks on buttons, covering both `<button>` and `<input type="button">` elements.

> **Info:** This does not track track a button click event directly, but rather will enable listen for button clicks.

### Filter

The `Filter` option allows you to specify a list of CSS classes that will be used to filter out elements that should not be tracked. This can be useful if you have a large number of buttons on your site, but only want to track clicks on a subset of them.

There are two types of filters you can use:

- Allow: Only elements that match the specified CSS classes will be tracked.
- Block: Elements that match the specified CSS classes will not be tracked.

Each filter takes a comma-separated list of CSS classes. For example, to only track clicks on buttons with the classes "track-me" and "click-me" you would add enter `track-me, click-me` into `Classes` when the `Allow` filter type is selected.

## Cart Tracking

The Cart Tracking tag is used to track interactions with a shopping cart. This can be used to track the addition and removal of items to a cart.

**Cart Tracking Parameters**

Add To Cart and Remove From Cart events take the following parameters:

| Name        | Required? | Description                                                     | Example    |
| ----------- | --------- | --------------------------------------------------------------- | ---------- |
| `sku`       | Yes       | The stock keeping unit (SKU) of the product being added/removed | 000345     |
| `unitPrice` | Yes       | The price of the product being added/removed                    | 12.99      |
| `quantity`  | Yes       | The quantity of the product being added/removed                 | 2          |
| `name`      | No        | The name of the product being added/removed                     | 'blue tie' |
| `category`  | No        | The category of the product being added/removed                 | 'clothing' |
| `currency`  | No        | The currency of the product being added/removed                 | 'USD'      |



> **Info:** We recommend migrating to the [Snowplow Ecommerce Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/) template to take advantage of the latest ecommerce tracking features.

## Site Search

The Site Search tag can be used to track searches on your website, using the [Site Search Schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/site_search/jsonschema/1-0-0)

**Site Search Parameters**

| Name           | Required? | Description                     | Example                  |
| -------------- | --------- | ------------------------------- | ------------------------ |
| `terms`        | Yes       | Search terms                    | \['unified, log']        |
| `filters`      | No        | Search filters                  | {'category': 'clothing'} |
| `totalResults` | No        | Results found                   | 10                       |
| `pageResults`  | No        | Results displayed on first page | 5                        |

## Timing

This is used to track user timing events, such as how long resources take to load.

**Timing Parameters**

| Name       | Required? | Description                    | Example            |
| ---------- | --------- | ------------------------------ | ------------------ |
| `category` | Yes       | Timing category                | 'load'             |
| `variable` | Yes       | Timed variable                 | 'map\_loaded'      |
| `timing`   | Yes       | Number of milliseconds elapsed | 50                 |
| `label`    | No        | Label for the event            | 'Map loading time' |

## Enhanced Consent

The Enhanced Consent tag is used to track user consent preferences. This can be used to track whether a user has given consent to certain tracking activities, such as tracking cookies or personalized ads.

**Enhanced Consent Events**

| Name           | Description                                             |
| -------------- | ------------------------------------------------------- |
| Allow All      | Track an acceptance of user consent.                    |
| Allow Selected | Track a specific selection of consented scopes.         |
| Pending        | Track the unconfirmed selection about user consent.     |
| Implicit       | Track the implicit consent on user consent preferences. |
| Deny           | Track a denial of user consent.                         |
| Expired        | Track the expiration of a consent selection.            |
| Withdrawn      | Track the withdrawal of user consent.                   |

**Enhanced Consent Parameters**

| Attribute            | Description                                                                         | Example                 |
| -------------------- | ----------------------------------------------------------------------------------- | ----------------------- |
| `basisForProcessing` | GDPR lawful basis for data collection & processing.                                 | Consent                 |
| `consentUrl`         | URI of the privacy policy related document.                                         | <https://example.com>   |
| `consentVersion`     | Version of the privacy policy related document.                                     | 1.0                     |
| `consentScopes`      | The scopes allowed after the user finalized their selection of consent preferences. | 'analytics, functional' |
| `domainsApplied`     | The domains for which this consent allows these preferences to persist to.          | <https://example.com>   |
| `gdprApplies`        | Determine if GDPR applies based on the user's geo-location.                         | -                       |

## Enhanced Ecommerce

For Google Analytics 4 compatible ecommerce tracking, see the dedicated [Snowplow v3 Ecommerce Tag](/docs/sources/google-tag-manager/previous-versions/v3/v3-tags/ecommerce-tag-template/) template instead.

> **Info:** The following Enhanced Ecommerce plugin has been deprecated in favor of the [Snowplow Ecommerce Tag](/docs/sources/google-tag-manager/ecommerce-tag-template/) template, it is recommended to use the new template for the latest ecommerce tracking features. This tag remains for compatibility with existing setups.

The Enhanced Ecommerce tag is used to track ecommerce events, such as product views, add to cart, and purchases.

When you select [Enhanced Ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/enhanced/), you are left with two options: Use Data Layer or Choose Variable. The way it works is very similar to Enhanced Ecommerce in Google Analytics.

If you choose the first, the template will look into the `dataLayer` structure for the most recently pushed [Enhanced Ecommerce object](https://www.simoahava.com/analytics/enhanced-ecommerce-guide-for-google-tag-manager/#data-types-actions), and map this object to the request to Snowplow Analytics.

If you selected `Choose Variable`, you need to provide a GTM variable that returns an object in the correct, expected format.

## Error Tracking

The Errors tracker tag provides a way to manually track any errors that occur on your website.

**Error Tracking Parameters**

| **Name**   | **Required?** | **Description**                     | **Example**              |
| ---------- | ------------- | ----------------------------------- | ------------------------ |
| `message`  | Yes           | Error message                       | 'Cannot get user object' |
| `filename` | No            | Filename or URL                     | 'shop.js'                |
| `lineno`   | No            | Line number of problem code chunk   | 23                       |
| `colno`    | No            | Column number of problem code chunk | 45                       |
| `error`    | No            | JS `ErrorEvent`                     | {Stack trace string}     |

## Page View

You can provide a `Custom Page Title` if you wish, and you can add a [custom entity](/docs/events/custom-events/#custom-entities) to the request, as also described [here](/docs/sources/web-trackers/tracking-events/page-views/).

If you enable `Page Activity Tracking`, page pings will be sent at intervals that you can specify.

The `Callback Function` is something you can set to a JavaScript function. If you set the callback, then instead of sending the page ping to Snowplow, the function gets invoked instead. See the [JavaScript tracker documentation](/docs/sources/web-trackers/tracking-events/activity-page-pings/#activity-tracking-callback) for more information.

## Link Click Tracking

The `Track Link Click` event is similar to regular parameter-based events, as it lets you add parameters and track a link click as a manually encoded hit.

The `Enable Automatic Link Click Tracking` adds listeners to the page, which will track clicks on links permitting they adhere to the allow/denylisted class names you can optionally provide. It is intended to trigger [early on a page](https://support.google.com/tagmanager/answer/7679319) to begin listening for click events, _not_ to trigger on click events like [Click Triggers](https://support.google.com/tagmanager/answer/7679320).

The `Fix Middle-click Tracking` adds a fix for some browsers where middle-clicks were not tracked properly.

If you check `Track HTML Content Of Clicked Link`, then the full text content of the link element will be sent to Snowplow as well.

> **Note:** This setting enables the tracker's native [Link Click Tracking plugin](/docs/sources/web-trackers/tracking-events/link-click/), it does not directly fire an event.

## Form Tracking

Form tracking has just two options, because filters and transformations won’t work with the custom template (due to lack of support for processing HTML elements). The options are to set form and/or field denylists and allowlists.

Denylists and allowlists for forms are a list of comma-separated HTML class names. If denylisted, then any form element with a listed class will not trigger the form event. If allowlisted, then _only_ form elements with a listed classname will be tracked.

For fields, denylists and allowlists work similarly, except they use the `name` attribute rather than the class.

Note that this tag enables the tracker's native [Form Tracking plugin](/docs/sources/web-trackers/tracking-events/form-tracking/), it does not directly fire an event. It is intended to trigger [early on a page](https://support.google.com/tagmanager/answer/7679319) to begin listening for form events, _not_ to trigger on form events like [Form Submit Triggers](https://support.google.com/tagmanager/answer/7679217).

## Custom commands

For any other commands which are supported by the Snowplow JavaScript Tracker v4, you can select the `[Custom Command]` option. Once selected, you can enter any function name and the associated parameter for that function. The parameter can either be a simple string, in examples such as `setUserId`:

![setUserId](/assets/images/setUserId-7128fb7529a6fb0148cccf87e6ddebf4.png)

Or it can be set to a Custom JavaScript Variable in the instances where an Object should be passed to the function, such as with `enableAnonymousTracking`:

![enableAnonymousTracking Custom JavaScript variable](/assets/images/enableAnonymousTracking_custom_variable-37f00a0b1fdc34f552bf62fb02e2983b.png)

And then use this variable as your Command Argument:

![enableAnonymousTracking Custom Command argument](/assets/images/enableAnonymousTracking_argument-2250e7ff142e56e8beb72b3674138c8b.png)

For a list of available commands, consult the [API documentation](https://github.com/snowplow/snowplow-javascript-tracker/blob/master/trackers/browser-tracker/src/api.ts) for the Browser Tracker

## Additional Tracking Parameters

![](/assets/images/additional_tracking_parameters-7f4abc16eb1463fe636661c2141bf131.png)

### Add custom context entities

Using the Context Entities table allows you to attach [custom context entities](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-entity) to the Snowplow event. Each row should be set to a variable value that must be an array of custom context objects that will all be concatenated to add to the Event.

For example to manually attach the web page context, create a custom JavaScript variable that returns an array with the custom context object:

```js
function() {
    return [{
        schema: 'iglu:com.example/web_page/jsonschema/1-0-0',
        data: {
            id: '12345',
            title: 'Example Page',
            url: 'https://example.com'
        }
    }];
}
```

Then set this variable as the value in the Context Entities table.

### Set Custom Timestamp

You can also choose to [set the True Timestamp](/docs/sources/web-trackers/tracking-events/#custom-timestamp) with this field. The format must be UNIX time in milliseconds.

## Parameter Object

Tags that can derive their parameters from a Google Tag Manager variable are:

- [Ad Tracking](/docs/sources/web-trackers/tracking-events/ads/)
- [Cart Tracking](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/ecommerce/original/)
- [Error Tracking](/docs/sources/web-trackers/tracking-events/errors/)
- [Self-describing Event](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-event-self-describing)
- [Site Search](/docs/sources/web-trackers/tracking-events/site-search/)
- [Social Interaction](/docs/sources/web-trackers/tracking-events/social-media/)
- [Structured Event](/docs/fundamentals/canonical-event/#structured-events)
- [Timing](/docs/sources/web-trackers/tracking-events/timings/generic/)

You can set the Retrieve Parameters From Variable setting to a Google Tag Manager variable. This parameter _must_ return an object. In the object, the key-value pairs should reflect the named parameters in the [event documentation](/docs/sources/web-trackers/tracking-events/). For example, to have the variable populate an [Error event](/docs/sources/web-trackers/tracking-events/errors/), you could use a Custom JavaScript variable like this:

```javascript
function() {
  return {
    message: 'Some Error Happened',
    filename: 'somefile.js',
    lineno: 5,
    colno: 236,
    error: null
  }
}
```

Alternatively, you can set the drop-down to the value `No`, and add the parameters manually instead:

![adding parameters manually](/assets/images/adding_parameters_manually-c874b2064a6048631c221f1f3375a18e.png)

Some tag types will add additional selections to this section. Follow the [official tracker documentation](/docs/sources/web-trackers/tracking-events/) for more information about what each option does.

---

# Snowplow GTM plugins
> Load JavaScript tracker plugins in Google Tag Manager to extend tracking functionality. Configure external and inline custom plugins with CDN URLs or custom JavaScript variables for enhanced event tracking.
> Source: https://docs.snowplow.io/docs/sources/google-tag-manager/snowplow-template/plugins/

Plugins are supported to provide an easy way to extend the functionality of the tracker. This is similar to the [JavaScript Tracker](/docs/sources/web-trackers/plugins/creating-your-own-plugins/) but with the caveat that plugins may be more limited in their functionality due to the constraints of the Google Tag Manager environment.

Each plugin can be loaded from external URLs or provided inline within GTM, and can optionally take some custom configuration. A plugin may add new methods to the tracker that can be called via [Custom Commands](/docs/sources/google-tag-manager/snowplow-template/#custom-commands).

> **Note:** Snowplow plugins are updated alongside the tracker version to ensure compatibility. For best results, keep the tracker and plugin versions in sync:
>
> - CDN Users: Use a GTM Variable to manage versions in URLs.
> - Self-Hosting: Update plugin files whenever you update the tracker version.

## `Load Plugins` table

Plugins are configured by using the `Load Plugins` table, within the Snowplow tag. Each row takes three values to load a plugin.

> **Tip:** Plugins are loaded by the tracker directly, not via the Tag Template: You do not need to adjust the Template permissions to allow loading plugins from these URLs.

### Plugin URL

The URL to load the plugin from, e.g. `https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-link-click-tracking@latest/dist/index.umd.min.js`

The tracker will try to load plugins each time they are requested. This is usually acceptable as the browser will have cached the first request; if you don't want this behavior we suggest creating an empty `Custom Command` Tag with no commands that loads the plugins; you can make this Tag fire [once per page](https://support.google.com/tagmanager/answer/6279951) and add it as a [Setup Tag](https://support.google.com/tagmanager/answer/6238868) to any other Tags that use its commands.

### Plugin Configuration

The name of the plugin to load. These values can be found on the respective plugin documentation.

For example, for the `Link Click Tracking` plugin, the values (`'snowplowLinkClickTracking,LinkClickTrackingPlugin'`) can be found in the snippet [here](/docs/sources/web-trackers/tracking-events/link-click/#toggle-link-click-tracking).

### Additional Configuration (optional)

Finally, the optional `Additional Configuration` field allows you to add any configuration the plugin may require.

There are two ways to format additional configuration:

- A single comma-seperated string value, which will be split into an array of strings and passed as arguments to the plugin. (e.g. `arg1,arg2,arg3`)
- A reference to a GTM Variable containing an Array of arguments to pass to the plugin.

Plugins are loaded in order, and processed before the configured `Tag Type` configuration is executed -- so you can load a plugin in the same Tag that uses its functionality via Custom Commands.

The plugins will remain configured in the tracker and be accessible to later Tags.

> **Warning:** The Tag Template will try to call `.indexOf(',')` on the `Additional Configuration` value, so values of types other than Array or String may fail and break the Tag. String values (before or after splitting) of `true`, `false`, `null`, and numeric values will become their respective typed JSON values. It is not possible to pass a single `null`, `undefined`, or empty string value as a parameter to a plugin, instead no arguments will be passed to the plugin.

## Custom plugins

[Inline plugins](/docs/sources/web-trackers/plugins/creating-your-own-plugins/#inline-plugins) are plugins that don't require being fetched from an external file to load.

You can create Inline plugins in GTM by using Custom JavaScript Variables in the `Plugin URL` field. The Variable should return an Object with a method that returns another Object meeting the [plugin interface](/docs/sources/web-trackers/plugins/creating-your-own-plugins/#plugin-interface) (any other methods on the outer Object will become tracker methods).

For `Plugin Configuration`, the UI enforces the comma-seperated values syntax required for external plugins and unconditionally calls `.split(',')` on the string. The tracker requires that for inline-plugins only a single string may be used. To work around this limitation, create another Custom JavaScript Variable that returns your constructor method name wrapped in an Object with a fake `split()` method:

```javascript
function() {
  return {
    split: function() {
      return "myInlineConstructorMethodName";
    }
  };
}
```

> **Info:** If your configuration includes functions, GTM will wrap those functions in [its sandbox](https://developers.google.com/tag-platform/tag-manager/templates/sandboxed-javascript), even when passed to the tracker for execution.
>
> Complex values like DOM elements will be replaced by `null` when passed to or returned from your function, which may make some plugins not function as intended.

---

# Snowplow sources overview
> Overview of Snowplow tracker SDKs for web, mobile, server-side platforms, and third-party webhook integrations to collect event data.
> Source: https://docs.snowplow.io/docs/sources/

Snowplow supports a wide range of sources which send events to your Collector endpoint.

Snowplow tracker SDKs are client- or server-side libraries that enable you to collect events from your own applications. For tracking events from third-party applications, use [webhooks](/docs/sources/webhooks/).

You can find the source code for each tracker on GitHub, where possible. The JavaScript tracker repository is a multirepository containing the web (JavaScript and Browser), Node.js, and React Native trackers.

All Snowplow trackers have open-source licenses (Apache 2.0 License, except for the JavaScript tracker which is BSD 3-Clause License).

## Client-side trackers

| Tracker                                                 | Language                     | Used in                                                 | [Snowtype](/docs/event-studio/implement-tracking/snowtype/)? | Supported [data models](/docs/modeling-your-data/modeling-your-data-with-dbt/) | GitHub                                                                                 | Installed using                                                                                                        |
| ------------------------------------------------------- | ---------------------------- | ------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| [Web](/docs/sources/web-trackers/) (JavaScript)         | TypeScript, JavaScript       | Web                                                     | ✅                                                            | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker) | Script tag                                                                                                             |
| [Web](/docs/sources/web-trackers/) (Browser)            | TypeScript, JavaScript       | Web                                                     | ✅                                                            | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker) | [npm](https://www.npmjs.com/package/@snowplow/browser-tracker), yarn, or pnpm                                          |
| [Google Tag Manager](/docs/sources/google-tag-manager/) | JavaScript                   | Web                                                     | ❌                                                            | Unified Digital, Attribution, Ecommerce                                        | n/a                                                                                    | [Tag Template Gallery](https://tagmanager.google.com/gallery/#/owners/snowplow/templates/snowplow-gtm-tag-template-v4) |
| [Google AMP](/docs/sources/google-amp-tracker/)         | HTML                         | AMP HTML webpages                                       | ❌                                                            |                                                                                | n/a                                                                                    | Script tag                                                                                                             |
| [Pixel](/docs/sources/pixel-tracker/)                   | HTML                         | Email, web environments where JavaScript is unavailable | ❌                                                            |                                                                                | n/a                                                                                    | Script tag                                                                                                             |
| [Android](/docs/sources/mobile-trackers/)               | Kotlin, Java                 | Mobile                                                  | ✅                                                            | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-android-tracker](https://github.com/snowplow/snowplow-android-tracker)       | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-android-tracker)                             |
| [iOS](/docs/sources/mobile-trackers/)                   | Swift, Objective-C           | Mobile (iOS), macOS, tvOS, watchOS, visionOS            | ✅                                                            | Unified Digital, Attribution, Media Player, Ecommerce                          | [snowplow-ios-tracker](https://github.com/snowplow/snowplow-ios-tracker)               | SPM or [Cocoapods](https://cocoapods.org/pods/SnowplowTracker)                                                         |
| [React Native](/docs/sources/react-native-tracker/)     | TypeScript, JavaScript       | Mobile                                                  | ✅                                                            | Unified Digital, Attribution                                                   | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker) | [npm](https://www.npmjs.com/package/@snowplow/react-native-tracker)                                                    |
| [Flutter](/docs/sources/flutter-tracker/)               | Dart                         | Mobile                                                  | ✅                                                            | Unified Digital, Attribution, Media Player                                     | [snowplow-flutter-tracker](https://github.com/snowplow/snowplow-flutter-tracker)       | [pub.dev](https://pub.dev/packages/snowplow_tracker)                                                                   |
| [Roku](/docs/sources/roku-tracker/)                     | BrightScript, BrighterScript | Media player applications                               | ❌                                                            | Media Player                                                                   | [snowplow-roku-tracker](https://github.com/snowplow/snowplow-roku-tracker)             | [npm](https://www.npmjs.com/package/@snowplow/roku-tracker)                                                            |
| [WebView](/docs/sources/webview-tracker/)               | TypeScript, JavaScript       | Hybrid mobile applications                              | ❌                                                            | Unified Digital, Attribution                                                   | [snowplow-webview-tracker](https://github.com/snowplow/snowplow-webview-tracker)       | [npm](https://www.npmjs.com/package/@snowplow/webview-tracker) or script tag                                           |

## Server-side trackers

| Tracker                                              | Language                    | [Snowtype](/docs/event-studio/implement-tracking/snowtype/)? | GitHub                                                                                 | Installed using                                                                                     |
| ---------------------------------------------------- | --------------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [Node.js](/docs/sources/node-js-tracker/)            | TypeScript, JavaScript      | ✅                                                            | [snowplow-javascript-tracker](https://github.com/snowplow/snowplow-javascript-tracker) | [npm](https://www.npmjs.com/package/@snowplow/node-tracker)                                         |
| [Python](/docs/sources/python-tracker/)              | Python                      | ❌                                                            | [snowplow-python-tracker](https://github.com/snowplow/snowplow-python-tracker)         | [PyPI](https://pypi.org/project/snowplow-tracker/)                                                  |
| [Golang](/docs/sources/golang-tracker/)              | Go                          | ✅                                                            | [snowplow-golang-tracker](https://github.com/snowplow/snowplow-golang-tracker)         | [GitHub](https://github.com/snowplow/snowplow-golang-tracker/releases)                              |
| [Java](/docs/sources/java-tracker/)                  | Java                        | ✅                                                            | [snowplow-java-tracker](https://github.com/snowplow/snowplow-java-tracker)             | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-java-tracker)             |
| [.NET](/docs/sources/net-tracker/)                   | C#                          | ❌                                                            | [snowplow-dotnet-tracker](https://github.com/snowplow/snowplow-dotnet-tracker)         | [NuGet](https://www.nuget.org/packages/Snowplow.Tracker)                                            |
| [Unity](/docs/sources/unity-tracker/)                | C#                          | ❌                                                            | [snowplow-unity-tracker](https://github.com/snowplow/snowplow-unity-tracker)           | [GitHub](https://github.com/snowplow/snowplow-unity-tracker/releases)                               |
| [C++](/docs/sources/c-tracker/)                      | C++                         | ❌                                                            | [snowplow-cpp-tracker](https://github.com/snowplow/snowplow-cpp-tracker)               | CMake                                                                                               |
| [Scala](/docs/sources/scala-tracker/)                | Scala                       | ❌                                                            | [snowplow-scala-tracker](https://github.com/snowplow/snowplow-scala-tracker)           | [Maven](https://mvnrepository.com/artifact/com.snowplowanalytics/snowplow-scala-tracker) or JCenter |
| [Ruby](/docs/sources/ruby-tracker/)                  | Ruby                        | ❌                                                            | [snowplow-ruby-tracker](https://github.com/snowplow/snowplow-ruby-tracker)             | [RubyGems](https://rubygems.org/gems/snowplow-tracker/)                                             |
| [PHP](/docs/sources/php-tracker/)                    | PHP                         | ❌                                                            | [snowplow-php-tracker](https://github.com/snowplow/snowplow-php-tracker)               | [Packagist](https://packagist.org/packages/snowplow/snowplow-tracker)                               |
| [Rust](/docs/sources/rust-tracker/)                  | Rust                        | ❌                                                            | [snowplow-rust-tracker](https://github.com/snowplow/snowplow-rust-tracker)             | [crates.io](https://crates.io/crates/snowplow_tracker)                                              |
| [Lua](/docs/sources/lua-tracker/)                    | Lua                         | ❌                                                            | [snowplow-lua-tracker](https://github.com/snowplow/snowplow-lua-tracker)               | [LuaRocks](https://luarocks.org/modules/snowplow/snowplowtracker)                                   |
| [Tracking CLI](/docs/sources/snowplow-tracking-cli/) | Command line, shell scripts | ❌                                                            | [snowplow-tracking-cli](https://github.com/snowplow/snowplow-tracking-cli/)            | [GitHub](https://github.com/snowplow/snowplow-tracking-cli/releases)                                |

## Webhooks

| Webhook                                          | Track                                                                |
| ------------------------------------------------ | -------------------------------------------------------------------- |
| [Adjust](/docs/sources/webhooks/adjust-webhook/) | Which marketing channels are driving mobile app installations        |
| [Iglu](/docs/sources/webhooks/iglu-webhook/)     | Any [Iglu](/docs/api-reference/iglu/)-compatible GET or POST request |
| [Iterable](/docs/sources/webhooks/iterable/)     | Events provided by Iterable                                          |
| [MailGun](/docs/sources/webhooks/mailgun/)       | Email activity logged by MailGun                                     |
| [Mandrill](/docs/sources/webhooks/mandrill/)     | Email activity logged by Mandrill                                    |
| [SendGrid](/docs/sources/webhooks/sendgrid/)     | Email activity logged by SendGrid                                    |
| [Zendesk](/docs/sources/webhooks/zendesk/)       | Events logged by Zendesk                                             |

---

# Configure how events are sent for the Java tracker
> Configure network connections, event batching, buffering, and retry logic for the Java tracker to optimize event delivery to your collector.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/configuring-how-events-are-sent/

A user interacts with your app: an event is generated and tracked using `Tracker.track()`. But the event must be sent to your event collector, to enter your pipeline, before it has any value.

The Java tracker allows configuration of the network connection, event sending, and buffering of events. All of these configurations are contained within the `NetworkConfiguration` and `EmitterConfiguration` classes. The default configurations will be sufficient for many Snowplow users. The tables below show the different configuration options you can set.

Using `NetworkConfiguration`:

| NetworkConfiguration | Description                | Required?                                      |
| -------------------- | -------------------------- | ---------------------------------------------- |
| collectorUrl         | Event collector endpoint   | Yes, unless an `HttpClientAdapter` is provided |
| httpClientAdapter    | `HttpClientAdapter` object | No                                             |
| cookieJar            | OkHttp `CookieJar` object  | No                                             |

The URL path for your collector endpoint should include the protocol, "http" or "https".

Using `EmitterConfiguration`:

| EmitterConfiguration      | Description                                | Required? | Default |
| ------------------------- | ------------------------------------------ | --------- | ------- |
| batchSize                 | Number of events to batch into one request | No        | 50      |
| bufferCapacity            | Maximum number of events buffered          | No        | 10 000  |
| eventStore                | `EventStore` object                        | No        |         |
| customRetryForStatusCodes | Map of HTTP status codes to retry or not   | No        |         |
| threadCount               | Number of threads to use                   | No        | 50      |
| requestExecutorService    | `ScheduledExecutorService` object          | No        |         |
| callback                  | `EmitterCallback` object                   | No        |         |

See the API docs for the full [NetworkConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/NetworkConfiguration.html) and [EmitterConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/EmitterConfiguration.html) details.

The default `Emitter` is the `BatchEmitter`, which sends events asynchronously in batches using POST requests. If you need to send events synchronously, or with GET requests, we have provided an `Emitter` interface so you can create your own.

To create a `Tracker` manually, start by initialising an `Emitter`. The simplest `BatchEmitter` initialisation looks like this:

```java
BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://collectorEndpoint"));
```

When an `Event` is tracked using `Tracker.track()`, a payload (`TrackerPayload`) is generated from the `Event`. The payload is added to the `BatchEmitter`'s `InMemoryEventStore` buffer. This triggers a check on the size of the buffer. The number of stored events is compared with the configured `batchSize`; the default is 50 events per batch. If there are enough events, a batch's worth is asynchronously removed from the buffer for sending. The `BatchEmitter` prepares a request payload containing all the event payloads, and attempts to send it. On receiving a successful HTTP response code (2xx), the events are considered sent, and permanently deleted from the buffer.

If the event buffer is full, the payload will be dropped. Priority is given to older events. In this case, `Tracker.track()` returns `null` rather than the payload `eventId`.

### What happens if an event fails to send?

After a sending attempt, the fate of an event depends on which status code was received. A 2xx code is always considered successful. If the `HttpClientAdapter` returns a failure code (anything other than 2xx, with certain exceptions, see below), the events (as `TrackerPayload` objects) are returned to the buffer. They will be retried in future sending attempts. The returned events are added to the start of the buffer queue, as older events are prioritised over newer ones.

To prevent unnecessary requests being made while the collector is unavailable, an exponential backoff is added to all subsequent event sending attempts. This resets after a request is successful. The maximum backoff time between attempts is 10 minutes.

The status codes `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, or `422 Unprocessable Entity` are the exceptions: they are not retried by default. Payloads in requests receiving these responses are not returned to the buffer for retry. They are just deleted.

Configure which codes to retry on or not using the `EmitterConfiguration` `customRetryForStatusCodes()` method when creating your tracker. This method takes a map of status codes and booleans (true for retry and false for not retry):

```java
// by default 403 isn't retried, but 500 is
Map<Integer, Boolean> customRetry = new HashMap<>();
customRetry.put(403, true);
customRetry.put(500, false);

Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration("https://collector"),
                new EmitterConfiguration().customRetryForStatusCodes(customCodes));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().customRetryForStatusCodes(customRetry));
```

See below for information about retries within HTTP clients.

### Configuring how many events to send in one request

The BatchEmitter sends events in batches. This is more efficient than sending event requests singly, as only one set of POST headers is required for a number of events. The event collector receives the request and separates out the individual event payloads.

The default batch size is 50 events. If you have a high event volume, larger batches may be more suitable. However, there is theoretically a risk of creating oversized requests, which could result in `413 Payload Too Large` responses from your collector. The Java tracker and Snowplow collector do not currently have the ability to split oversized requests into smaller ones.

Configure the batch size like this:

```java
Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration("https://collector"),
                new EmitterConfiguration().batchSize(100));

// Batch size can be updated after initialization
tracker.getEmitter().setBatchSize(10)

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().batchSize(100))
```

The `batchSize` property was called `bufferSize` in versions before 0.12.

To force send all buffered events, as a single request:

```java
emitter.flushBuffer()
```

### Configuring how events are buffered

The Java tracker sends events asynchronously: the application is not blocked waiting for the tracked event to be sent. The tracked events are stored in memory until there are enough to send, or while the network is down. The default event store is `InMemoryEventStore`, added in version 0.12. This class stores the payloads in a queue, specifically a `LinkedBlockingDeque`.

The default buffer capacity is 10 000 events. This is the number of events that can be stored. When the buffer is full, new tracked payloads are dropped, so choosing the right capacity is important. If events are accumulating in your buffer because of a very high event volume, rather than because of network outages, consider using more threads (see below).

The theoretical maximum capacity is that of a`LinkedBlockingDeque`: `Integer.MAX_VALUE`. It's likely your application would run out of memory before buffering that many events.

Creating a `BatchEmitter` with a custom maximum buffer capacity:

```java
Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration("https://collector-endpoint"),
                new EmitterConfiguration().bufferCapacity(100000));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().bufferCapacity(100000));
```

The `BatchEmitter` in this tracker will store 100 000 events before starting to lose data.

We also provide an `EventStore` interface. To use a custom EventStore:

```java
Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration("https://collector-endpoint"),
                new EmitterConfiguration().eventStore(EventStore));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().eventStore(EventStore));
```

If `bufferCapacity` is provided at the same time as `eventStore`, the `bufferCapacity` value will be discarded.

### Configuring the network connection

We currently offer two different HTTP clients that can be used to send events to your collector: OkHttp or Apache HTTP. Both libraries have broadly the same features, with some differences in their default configurations. If neither of these HTTP clients is suitable, we also provide an `HttpClientAdapter` interface. The `HttpClientAdapter` is a wrapper for HTTP client objects.

> **Note:** Gradle users: [different dependencies](/docs/sources/java-tracker/installation-and-set-up/) can be configured if you are using OkHttp or Apache HTTP.

By default, the Java tracker uses OkHttp; an `OkHttpClientAdapter` object is generated when a `BatchEmitter` is created. See below for how to use Apache HTTP instead, or how to customise the OkHTTP setup.

To specify a completely different client adapter, use the `HttpClientAdapter` interface and initialize the `Tracker` like this:

```java
HttpClientAdapter adapter = {{ your implementation here }}

Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration(adapter));
```

Note that `collectorUrl` is not a required parameter for `NetworkConfiguration` when an `HttpClientAdapter` is specified. The `collectorUrl` is normally used to create the default `OkHttpClientAdapter`, therefore if a URL was provided here, it would be ignored.

HTTP request retry can be configured within the HTTP clients, on top of the Java tracker's handling of unsuccessful requests. The default HTTP client, OkHttp, [retries after certain types of connection failure](https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/-builder/retry-on-connection-failure/) by default. The Apache HTTP Client [retries a request up to 3 times](https://www.javadoc.io/doc/org.apache.httpcomponents/httpclient/4.3.5/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html) by default.

#### OkHttpClient

The simplest OkHttpClient initialization looks like this:

```java
OkHttpClient client = new OkHttpClient();
```

This is the default as used in the `BatchEmitter`.

To add configuration, pass a `OkHttpClientAdapter` during Tracker initialization. For example, setting timeouts:

```java
OkHttpClient client = new OkHttpClient.Builder()
      .connectTimeout(5, TimeUnit.SECONDS)
      .readTimeout(5, TimeUnit.SECONDS)
      .writeTimeout(5, TimeUnit.SECONDS)
      .build();

OkHttpClientAdapter adapter = new OkHttpClientAdapter(
      "http://collector-endpoint.com",
      client
);

Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration(adapter));
```

The URL is the address for your collector. See [Square's API docs](https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpClient.Builder.html) for the full list of options.

#### Apache HTTP Client

The simplest Apache HTTP Client initialization looks like this:

```java
CloseableHttpClient client = HttpClients.createDefault();
```

Wrap the Client in an `ApacheHttpClientAdapter` to pass it to the tracker during initialization.

You are encouraged to research how best to set up your Apache Client for maximum performance. For example, by default the Apache Client will never time out, and will also allow only two outbound connections at a time. In this code block, a `PoolingHttpClientConnectionManager` is used to allow up to 50 concurrent outbound connections:

```java
PoolingHttpClientConnectionManager manager = new PoolingHttpClientConnectionManager();
manager.setDefaultMaxPerRoute(50);

CloseableHttpClient client = HttpClients.custom()
      .setConnectionManager(manager)
      .build();

ApacheHttpClientAdapter adapter = new ApacheHttpClientAdapter(
      "http://collector-endpoint.com",
      client
);

Tracker tracker = Snowplow.createTracker(
                new TrackerConfiguration("namespace", "appId"),
                new NetworkConfiguration(adapter));
```

The URL is the address for your collector. See [Apache's HttpClient docs](https://hc.apache.org/httpcomponents-client-4.5.x/index.html) for more information about configuring the client.

### Configuring the Java tracker threads

The `BatchEmitter` contains threads for concurrent event sending. This is managed by an `ScheduledThreadPoolExecutor`. By default, the thread pool has up to 50 threads, helpfully named e.g. "snowplow-emitter-pool-1-request-thread-1". The bigger the pool of threads, the faster events can be sent. Set the number of threads depending on many events you are sending, and how strong a computer the tracker is running on.

The process of getting events from the buffer, creating a request payload, and sending the POST request occurs within a single thread.

Specifying the maximum number of event sending threads, in this case to 1:

```java
Tracker tracker = Snowplow.createTracker(
                  new TrackerConfiguration("namespace", "appId"),
                  new NetworkConfiguration("https://collector"),
                  new EmitterConfiguration().threadCount(1));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().threadCount(1));
```

It's also possible to provide your own `ScheduledExecutorService`:

```java
Tracker tracker = Snowplow.createTracker(
                  new TrackerConfiguration("namespace", "appId"),
                  new NetworkConfiguration("https://collector"),
                  new EmitterConfiguration().requestExecutorService(ScheduledExecutorService));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().requestExecutorService(ScheduledExecutorService));
```

The default thread pool uses non-daemon threads. To stop the threads and shut down the `ExecutorService`, call `Emitter.close()` (or `Tracker.close()`). There is no way to restart the `Emitter` after this.

### Persisting cookies using a CookieJar

> **Note:** The `OkHttpClientWithCookieJarAdapter` was released in the version 2.0.0 of the Java tracker.

As described [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/), the event collector sets a third-party cookie. This cookie is extracted during event processing (enrichment phase) into the `network_userid` property, the server-side user identifier. To persist this cookie across requests, use the `OkHttpClientWithCookieJarAdapter` when creating your `BatchEmitter` or `Tracker`. Note that the `OkHttpClientWithCookieJarAdapter` uses an in-memory cookie jar, so the cookies, and `network_userid`, will not persist when it goes out of memory.

The simplest implementation looks like this:

```java
Tracker tracker = Snowplow.createTracker(
                  new TrackerConfiguration("namespace", "appId"),
                  new NetworkConfiguration(new OkHttpClientWithCookieJarAdapter("http://collector")));

// A BatchEmitter can also be created directly
BatchEmitter emitter = new BatchEmitter(networkConfig);
```

The specified `CookieJar` will be ignored if a custom `HttpClientAdapter` is provided. To use a `CookieJar` with a custom `OkHttpClientAdapter`, it must be added using the `OkHttpClient.Builder`:

```java
OkHttpClient httpClient = new OkHttpClient.Builder()
                  .cookieJar(new CollectorCookieJar())
                  .build();
adapter = new OkHttpClientAdapter("http://collectorEndpoint", httpClient);

Tracker tracker = Snowplow.createTracker(
                  new TrackerConfiguration("namespace", "appId"),
                  new NetworkConfiguration(adapter));
```

Any custom `OkHttp` `CookieJar` (other than the `CollectorCookieJar` provided in the tracker) could be used instead.

### Using the Emitter callback

To gain visibility on tracker activity, you may wish to take advantage of the `EmitterCallback` interface. This interface was added in v1.0.0, and requires `onSuccess()` and `onFailure()` methods.You can use `EmitterCallback` to create your own tracker metrics.

The `onSuccess()` callback is called when a request has successfully sent to the event collector (HTTP status code 2xx). It takes a list of the `TrackerPayload` objects that were batched into the request.

The `onFailure()` callback is called at several different points of failure. It also takes a list of the `TrackerPayload` objects involved, however, two other parameters are required. The first is the enum `FailureType`, explaining what kind of failure has occurred. The second is a boolean for whether or not the events will be returned to the buffer for sending retry.

The possible `FailureType` options are: `REJECTED_BY_COLLECTOR`, when HTTP status codes other than 2xx are received for a request; `TRACKER_STORAGE_FULL`, when events are lost because the `InMemoryEventStore` buffer is full; `HTTP_CONNECTION_FAILURE`, for unsuccessful requests or `HttpClientAdapter` exceptions; or `EMITTER_REQUEST_FAILURE`, for a `BatchEmitter` exception.

Configure a callback in your tracker like this:

```java
EmitterCallback callback = {{ your implementation here }}
Tracker tracker = Snowplow.createTracker(
                  new TrackerConfiguration("namespace", "appId"),
                  new NetworkConfiguration("https://collector"),
                  new EmitterConfiguration().callback(callback));
```

---

# Custom tracking with the Java tracker
> Track custom events and add context entities using self-describing JSON schemas with the Java tracker to capture business-specific data.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach custom data to any tracked event. It's one of the most powerful Snowplow features.

When tracking user behavior, the event describes the specific activity they performed, e.g. a user added an item to an eCommerce cart. To understand the meaning of the event, and how it relates to your business, it's ideal to also track the relatively persistent environment in which the activity was performed. For example, is the user a repeat customer? Which item did they add, and how many are in stock?

These environmental factors can be tracked as the event "context", using self-describing JSON. When self-describing JSON are tracked as part of an event, they are called "entities". All the entities of an event together form the context. Read more in this [thorough blog post](https://snowplowanalytics.com/blog/2020/03/25/what-are-snowplow-events-and-entities-and-what-makes-them-so-powerful/).

### Adding custom entities to any event

Every `Event.Builder` in the Java tracker allows for a list of `SelfDescribingJson` objects to be added to the `Event`. It's fine to add multiple entities of the same type. There's no official limit to how many entities you can add to a single event, but consider if the payload size could become problematic if you are adding a large number.

Context entities can be added to any event using the `customContext()` Builder method:

```java
// This example shows an SelfDescribing event, but all events can have context
SelfDescribing selfDescribing = SelfDescribing.builder()
            .eventData(dataAsSelfDescribingJson)
            .customContext(listOfEntitiesAsSelfDescribingJson)
            .build();
```

Event context is sent as a JSON inside the event payload. During [enrichment](/docs/pipeline/enrichments/), it is separated into columns for each different schema used.

A note on nomenclature: entities were originally called "context", with the context called "contexts". The old nomenclature is still used in some parts of Snowplow, meaning that enriched events in the data warehouse refer to "contexts"/"context" rather than "context"/"entity".

### Global context

The Java tracker does not yet provide the ability to automatically assign entities to specific events.

### Self-describing JSON in the Java tracker

The Java tracker provides the `SelfDescribingJson` class for custom events and entities. There is no in-built distinction between schemas used for events and those used for entities: they can be used interchangably.

Your schemas must be accessible to your pipeline, within an [Iglu server](/docs/api-reference/iglu/). Tracked events containing self-describing JSON are validated against their schemas during the enrichment phase of the pipeline. If the data don't match the schema, the events end up as [failed events](/docs/fundamentals/failed-events/).

A self-describing JSON needs two keys, `schema` and `data`. The `schema` key is the Iglu URI for the schema. The `data` value must match the properties described by the specified schema. It is usually provided as a map.

A simple initialisation looks like this:

```java
// This map will be used for the "data" key
Map<String, String> eventData = new HashMap<>();
eventData.put("targetUrl", "https://www.snowplowanalytics.com")

// This is the Iglu schema URI
String schema_uri = "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1"

// Wrap the map in a SelfDescribingJson
// The specified schema allows for a String property "targetUrl"
SelfDescribingJson sdj = new SelfDescribingJson(schema_uri, eventData);
```

See the API docs for the full [SelfDescribingJson](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/payload/SelfDescribingJson.html) constructor options.

JSON can be sent base-64 encoded or not. By default, they are encoded. This is set as part of the [Tracker configuration](/docs/sources/java-tracker/installation-and-set-up/#configuring-the-tracker).

---

# Example applications for the Java tracker
> Reference implementations and sample code to help you get started with the Java tracker.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

- [Simple Console](https://github.com/snowplow/snowplow-java-tracker/tree/master/examples/simple-console)
- [Benchmarking](https://github.com/snowplow/snowplow-java-tracker/tree/master/examples/benchmarking)

---

# Java tracker SDK
> Add event tracking to Java-based desktop apps, server applications, servlets, and games with the Snowplow Java tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/

The Snowplow Java Tracker lets you add analytics to your [Java](http://www.java.com/en/)-based desktop and server apps, servlets and games.

---

# Install the Java tracker
> Install and configure the Java tracker using Maven, Gradle, or direct download for JDK8+ applications.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/installation-and-set-up/

The Snowplow Java tracker has been built and tested using Java versions 8, 11 and 17, so should work within any Java application built using JDK8 upwards. The Java tracker is also usable from Scala.

The current tracker version is 2.1.0. New issues and pull requests are very welcome! Find the Github repository [here.](https://github.com/snowplow/snowplow-java-tracker)

## Installing

### Install using Maven

Add into your project’s `pom.xml`:

```xml
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-java-tracker</artifactId>
  <version>2.1.0</version>
</dependency>
```

### Install using Gradle

From version 0.10.1 onwards, we provide out-of-the-box support for sending events via OkHttp or Apache HTTP. The appropriate dependencies must be specified. The default tracker configuration uses OkHttp.

Add this into your project’s `build.gradle` for the default installation with OkHttp support:

```gradle
dependencies {
  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'
  implementation ('com.snowplowanalytics:snowplow-java-tracker:2.1.0') {
      capabilities {
          requireCapability 'com.snowplowanalytics:snowplow-java-tracker-okhttp-support'
      }
  }
}
```

Adding dependencies for Apache HTTP support instead (read [this page](/docs/sources/java-tracker/configuring-how-events-are-sent/) for how to configure this):

```gradle
dependencies {
  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'
  implementation ('com.snowplowanalytics:snowplow-java-tracker:2.1.0') {
      capabilities {
          requireCapability 'com.snowplowanalytics:snowplow-java-tracker-apachehttp-support'
      }
  }
}
```

If you are using your own `HttpClientAdapter` implementation and/or will be installing dependencies separately:

```gradle
dependencies {
  implementation 'com.snowplowanalytics:snowplow-java-tracker:2.1.0'
}
```

### Install by direct download

You can also manually insert the tracker by downloading the jar directly from [Maven Central](https://search.maven.org/search?q=a:snowplow-java-tracker).

### Install in Scala project (SBT)

The Snowplow Java tracker is also usable from Scala. Add this to your SBT config:

```scala
// Dependency
val snowplowTracker = "com.snowplowanalytics"  % "snowplow-java-tracker"  % "2.1.0"
```

## Setting up

The simplest initialization looks like this:

```java
import com.snowplowanalytics.snowplow.tracker.Snowplow;
import com.snowplowanalytics.snowplow.tracker.Tracker;

Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
```

The URL path for your collector endpoint should include the protocol, "http" or "https". The Java tracker is able to send events to either.

The `Snowplow` interface, added in v1, contains static methods to help initialise and manage `Tracker` objects. This is especially useful when multiple trackers are needed (see [here](/docs/sources/java-tracker/using-multiple-trackers/)). See the API docs for the full [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

The [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker) includes a mini demo, "simple-console". Follow the instructions in the README to send one event of each type to your event collector. Simple-console is provided as a simple reference app to help you set up the tracker.

These are the required objects for tracking using the Java tracker:

| Class                 | Function               |
| --------------------- | ---------------------- |
| `Tracker`             | Tracks events          |
| subclasses of `Event` | What you want to track |

### Configuring the `Tracker`

The `Tracker` class has the responsibility for tracking [events](/docs/sources/java-tracker/tracking-events/). Certain properties can or must also be set when creating a `Tracker`, which will be attached to all events. These are `trackerNamespace`, `appId`, and `platform`.

Both `trackerNamespace` and `appId` are required arguments when creating a `Tracker`. Snowplow events are designed to be stored in a single data warehouse/lake, regardless of their source, to make data modeling easier and provide a single valuable source of truth for your business. The tracker namespace allows you to distinguish events sent by this specific `Tracker`, if you are using multiple `Tracker` instances within your app. It's also the identifier for `Tracker` objects in the `Snowplow` class. The `appId` allows you to identify events from this specific application, if you are tracking from multiple places.

The other Tracker property that will be added to all tracked events is `platform`. This is set by default to `srv` - "server-side app". To set another valid platform type, use the `DevicePlatform` enum during construction.

The final two `Tracker` configuration options are whether to use base-64 encoding, and whether to add a `Subject` object (see [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/) for details about `Subject`). By default, JSONs within the event are sent base-64 encoded. This can be set to `false` here at `Tracker` initialization.

To create a `Tracker` with custom configuration, use the `TrackerConfiguration` class.

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("namespace", "appId")
        .base64Encoded(false)
        .platform(DevicePlatform.Desktop);

// Use the Snowplow class to create Trackers
Tracker tracker = Snowplow.createTracker(
    trackerConfig,
    new NetworkConfiguration("http://collector"));

// Alternatively, create an Emitter first, and then create a Tracker directly
// Emitters are introduced in the next section
BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://collector"));
Tracker tracker = new Tracker(trackerConfig, emitter);
```

This `Tracker` will produce events with the `platform` value `pc`.

See the API docs for the full [Tracker](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Tracker.html), [TrackerConfiguration](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/configuration/TrackerConfiguration.html) and [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

### Stopping the tracker

The default `Emitter`, `BatchEmitter`, contains a pool of non-daemon threads managed by a `ScheduledExecutorService`. To close the executor service and stop the threads, use the `close()` method. This method can be called directly on an `Emitter`, or via a `Tracker`:

```java
tracker.close()

// It's also possible to close the Emitter directly
tracker.getEmitter().close()
emitter.close()
```

There is no way to restart the `Emitter` after this. Note that if your app is not quit, events can still be tracked after `close()` has been called: they will accumulate in the `Emitter` buffer, unable to be sent.

### Logging

Logging in the Tracker is done using SLF4J. The majority of the logging set as `DEBUG` so it will not overly populate your own logging.

Since Java tracker v0.11, user-supplied values are only logged at `DEBUG` level.

---

# Migration guides for the Java tracker
> Upgrade guides to help you migrate between major versions of the Java tracker.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the Java tracker.

---

# From version 0.11 to 0.12
> Migration guide for upgrading the Java tracker from version 0.11 to 0.12 with breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/migration-guide-v0-12/

Several new features were added in v0.12. See the [Github](https://github.com/snowplow/snowplow-java-tracker) changelog for full details. Unfortunately, these improvements involve multiple breaking changes. See below for a guide to the changes.

### `BatchEmitter`

The `BatchEmitter.Builder` configuration options have changed.

**Updated**:

- `bufferSize` has been renamed `batchSize`.
- `requestExecutorService()` now requires a `ScheduledExecutorService`, rather than just `ExecutorService`.

**Removed**:

- `requestCallback()` is gone, since events that fail to send will now be automatically retried.

**Added** (these are not breaking changes):

- `eventStore()`. It's now possible to design your own event buffer using the `EventStore` interface. This is an optional setting, with a default of `InMemoryEventStore`.
- `bufferCapacity()`. Set the maximum capacity of the default `InMemoryEventStore`. Again, this setting is optional. The default is `Integer.MAX_VALUE` (the maximum capacity of the queue used).

**Example**: creating a BatchEmitter with the relevant options\
Old API:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()
        .url("http://collector.url")
        .bufferSize(20)
        .requestCallback( {{ A callback }} )
        .requestExecutorService(ExecutorService)
        .build();
```

Version 0.12:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()
        .url("http://collector.url")
        .batchSize(20)
        .eventStore(EventStore)
        .bufferCapacity(1000000) // this won't do anything since eventStore is specified
        .requestExecutorService(ScheduledExecutorService)
        .build();
```

### `SimpleEmitter`

`SimpleEmitter` has been deprecated. Please use `BatchEmitter` instead, or create your own `Emitter` using the provided interface. Like `SimpleEmitter`, the `BatchEmitter` sends events asynchronously. However, requests are made using POST, rather than GET. We strongly recommend sending events in batches, but to mimic SimpleEmitter in sending events one-by-one, use a `batchSize` of 1.

**Example**: replacing `SimpleEmitter` with `BatchEmitter`\
Old API:

```java
SimpleEmitter simpleEmitter = SimpleEmitter.builder()
        .url("http://collector.url")
        .build();

Tracker tracker = new Tracker.TrackerBuilder(simpleEmitter, "namespace", "appId")
        .build();
```

Version 0.12:

```java
BatchEmitter batchEmitter = BatchEmitter.builder()
        .url("http://collector.url")
        .batchSize(1)
        .build();

Tracker tracker = new Tracker.TrackerBuilder(batchEmitter, "namespace", "appId")
        .build();
```

### `Emitter` interface

Every method in the `Emitter` interface has been updated!

`setBufferSize()` and `getBufferSize()` have been renamed `setBatchSize()` and `getBatchSize()`.

`getBuffer()` now returns a list of `TrackerPayload` objects rather than `TrackerEvent` objects (which no longer exist).

Finally, `emit()` is now more accurately called `add()`, as in "add to buffer". It takes a `TrackerPayload` object, not a `TrackerEvent`, and has a new return type: a `boolean`, used to confirm that the `TrackerPayload` has been successfully added to the buffer.

### Getting an event's `eventId`

We are aware of some use cases involving exporting the `eventId`s of tracked Events to third-party apps. The `eventId` is now returned from `Tracker.track()`. It's returned in a list to allow for `EcommerceTransaction` events, which generate multiple payloads. If the event buffer is full, the event is lost. In this case, `null` will be returned instead of the `eventId`.

**Example**: getting the `eventId`\
Old API:

```java
PageView pageView = PageView.builder()
        .pageUrl("https://www.snowplowanalytics.com")
        .build();

String eventId = pageView.getEventId();
```

Version 0.12:

```java
PageView pageView = PageView.builder()
        .pageUrl("https://www.snowplowanalytics.com")
        .build();

List<String> eventIds = tracker.track(pageView);
String eventId = eventIds.get(0);
```

### `Event`, `AbstractEvent`, and child classes

Several methods have been removed: the `Event` interface and `AbstractEvent` methods `getDeviceCreatedTimestamp()`, and `getEventId()`; and the `AbstractEvent.Builder` methods `deviceCreatedTimestamp()`, and `eventId()`. The deprecated `timestamp()` and `getTimestamp()` methods have been removed from these classes too.

Having discovered them, we deleted these methods immediately (rather than deprecating) as we considered them very dangerous. Allowing custom UUIDs can accidentally lead to non-unique "unique" identifiers, which causes big problems for pipelines, and risks data loss.

Despite the name, `Event` objects are no longer associated with an `eventId` ; this is generated when the `Payload` object is made. The main purpose of the `eventId` is to provide a UUID for events once they have been received by the collector and are in the pipeline.

**Example**: creating a PageView with all the options\
Old API:

```java
PageView pageViewEvent = PageView.builder()
        .pageTitle("Snowplow Analytics")
        .pageUrl("https://www.snowplowanalytics.com")
        .referrer("https://www.google.com")
        .customContext(SelfDescribingJson)
        .subject(Subject)
        .trueTimestamp(1646834667343L)
        .deviceCreatedtimestamp(1646834667123L)
        .eventId("UUID")
        .build();
```

Version 0.12:

```java
PageView pageViewEvent = PageView.builder()
        .pageTitle("Snowplow Analytics")
        .pageUrl("https://www.snowplowanalytics.com")
        .referrer("https://www.google.com")
        .customContext(SelfDescribingJson)
        .subject(Subject)
        .trueTimestamp(1646834667343L)
        .build();
```

### `TrackerEvent` and callbacks

This class no longer exists. It was a wrapper around tracked `Event` objects to allow request callbacks. `Event`s were stored in the buffer as `TrackerEvent` until event sending, when a `TrackerPayload` would be extracted. `TrackerPayload` objects are now stored directly.

Callbacks allowed developers to re-track events that failed to send - if a HTTP response code other than 2xx was received. This put the burden on users to handle retry. Now that the tracker automatically retries, callbacks are no longer necessary, and have been removed.

---

# From version 0.12 to 1.0
> Migration guide for upgrading the Java tracker from version 0.12 to 1.0 with API improvements and configuration changes.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/migration-guides/migration-guide-v1/

We've made multiple improvements to the API, and to the configuration, for Java tracker v1. See the [Github changelog](https://github.com/snowplow/snowplow-java-tracker/releases/tag/1.0.0) for full details. See below for a guide to the changes.

## API improvements (breaking changes)

### `Snowplow` interface, `Configuration` classes, and deprecated Builders

This is a big change. We wanted an intuitive API going into v1, so we've added a `Snowplow` interface to easily create and manage trackers. We've added Configuration classes - `TrackerConfiguration`, `NetworkConfiguration`, `EmitterConfiguration` and `SubjectConfiguration` - and several new constructors for instantiating the tracker or tracker components. As part of this work, we've deprecated the builders for the `Tracker`, `BatchEmitter`, `HttpClientAdapter` and `Subject` classes.

Instantiating a tracker with default configuration only requires three strings. Use configuration objects to instantiate custom trackers.

**Example**: creating a tracker with default configuration

Old API:

```java
BatchEmitter emitter = BatchEmitter.builder()
        .url("http://collectorEndpoint")
        .build();
Tracker tracker = new Tracker
        .TrackerBuilder(emitter, "trackerNamespace", "appId")
        .build();
```

New v1 API:

```java
Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
```

**Example**: creating a tracker with custom configuration

Old API:

```java
BatchEmitter emitter = BatchEmitter.builder()
        .url("http://collectorEndpoint")
        .batchSize(10)
        .threadCount(1)
        .build();
Tracker tracker = new Tracker
        .TrackerBuilder(emitter, "trackerNamespace", "appId")
        .platform(DevicePlatform.Desktop)
        .base64(false)
        .build();
```

Version 1.0.0:

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("trackerNamespace", "appId")
        .platform(DevicePlatform.Desktop)
        .base64Encoded(false);
NetworkConfiguration networkConfig = new NetworkConfiguration("http://collectorEndpoint");
EmitterConfiguration emitterConfig = new EmitterConfiguration()
        .batchSize(10)
        .threadCount(1);
        
Tracker tracker = Snowplow.createTracker(trackerConfig, networkConfig, emitterConfig);
```

In the above examples, the `Tracker` is saved as a separate variable. That's not actually necessary with the new `Snowplow` class! Access stored `Tracker` objects by their namespace, or use the default tracker:

New v1 API:

```java
Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
Snowplow.getDefaultTracker().track(event);
```

### `SelfDescribing` events

The `Unstructured` class has been renamed to `SelfDescribing`, bringing consistency with other Snowplow trackers. These are the custom events, based on self-describing (self-referential) JSONs, that allow you to create the perfect data for your business needs.

**Example**: replacing `Unstructured` with `SelfDescribing`

Old API:

```java
Unstructured event = Unstructured.builder()
            .eventData(dataAsSelfDescribingJson)
            .build();
```

New v1 API:

```java
SelfDescribing event = SelfDescribing.builder()
            .eventData(dataAsSelfDescribingJson)
            .build();
```

### Custom retry logic

The event sending retry logic has been refined. By default, the tracker will now not retry requests with the HTTP status codes 400, 401, 403, 410, or 422. This can be configured using the Emitter builder option `customRetryForStatusCodes()`, which replaces `fatalResponseCodes()`. Note that 2xx status codes will always be deemed successful, and will never be retried.

`customRetryForStatusCodes()` takes a map of integer codes and booleans - true for "yes, this should be retried" and false for "no, this should not be retried".

**Example**: setting custom retry rules

Old API:

```java
List<Integer> noRetry = new ArrayList<>();
noRetry.add(500);

BatchEmitter emitter = BatchEmitter.builder()
        .url("https://collector")
        .fatalResponseCodes(noRetry)
        .build();
```

New v1 API:

```java
Map<Integer, Boolean> customRetry = new HashMap<>();
customRetry.put(500, false);

BatchEmitter emitter = new BatchEmitter(
      new NetworkConfiguration("https://collector"),
      new EmitterConfiguration().customRetryForStatusCodes(customRetry));
```

### New method for `Emitter` interface

The `BatchEmitter` method `close()` has been added to the `Emitter` interface. This method stops the executor service and closes the threads, and may be required to cleanly exit your application. If you are using a custom `Emitter`, you'll need to implement the method (`@Override`). The `Tracker` class now has a `close()` method too, which calls `Emitter.close()`.

### Updated method for `EventStore` interface

The `EventStore` method `cleanupAfterSendingAttempt()` now returns `List<TrackerPayload>`, rather than `void`. This is to help with the new `EmitterCallback` implementation (see below). If you are using a custom `EventStore`, you'll need to update the method signature.

## Default configuration changes

The default `bufferCapacity`, the maximum number of events that can be buffered in memory, is now 10 000 (down from `Integer.MAX_VALUE`).

By default, the Java tracker now does not retry sending events if the request received the HTTP status code `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, or `422 Unprocessable Entity`.

When requests get a "unsuccessful" response status code - anything other than 2xx - a backoff wait time is added before the next request is processed. The wait time increases exponentially with every failure. There is now a maximum wait time of 10 minutes.

## Removed features

The `SimpleEmitter` class, deprecated in v0.12, has been removed. The default `Emitter` is the `BatchEmitter`, but you can also create your own `Emitter` using the provided interface. The `BatchEmitter` sends events in batches via POST. Deleting `SimpleEmitter` allowed us to tidy up behind-the-scenes, merging `AbstractEmitter` and `BatchEmitter` into one class.

We also removed the [Guava](https://github.com/google/guava) dependency.

## New features

New and improved callbacks! You can now add custom callbacks using the `EmitterCallback` interface, for example to create tracker metrics. The `EmitterCallback` will be called when events are successfully sent, and also in response to several different failures. It's described fully [here](/docs/sources/java-tracker/configuring-how-events-are-sent/#using-the-emitter-callback).

You can now add a `CookieJar` to persist the third-party event collector `sp` cookie across requests. This allows all events to have the same `network_userid`.

---

# Track events with the Java tracker
> Track page views, screen views, structured events, and custom self-describing events with the Java tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/tracking-events/

To track an event, pass an `Event` instance to the `Tracker`.

For example, tracking a ScreenView:

```java
Event event = new ScreenView.builder()
                .name("screen name")
                .build();
tracker.track(event);
```

The Java tracker makes it easy to track different kinds of data. We provide a range of `Event` classes for tracking out-of-the-box event types as well as fully custom events.

Every tracked event payload has a unique `event_id` UUID string. Other ubiquitous properties include the `name_tracker` (`trackerNamespace`) and `app_id` (`appId`) set when the `Tracker` was initialized. From version 0.12 onwards, `Tracker.track()` returns the payload's `eventId`.

Snowplow events have a defined structure and [protocol](/docs/events/) that is identical regardless of the tracker used. A minimal payload - the raw event - is sent from the tracker to your collector. The raw event is [enriched](/docs/pipeline/enrichments/) as it passes through your pipeline. By the time the event arrives in your data storage, depending which [enrichments](/docs/pipeline/enrichments/available-enrichments/) you have enabled, it will have gained different kinds of metadata, and have many more fields than it started with. The default Java tracker event fields are shown [here](/docs/sources/java-tracker/what-do-java-tracker-events-look-like/).

The [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker) includes a mini demo, "simple-console". The demo sends one event of each type to your event collector.

## Auto-tracked events

The Java tracker does not yet support automatic event tracking. All tracking must be implemented manually.

## Manually-tracked events summary

The Java tracker provides classes for tracking different types of events. They are listed below.

| `Event` class              | `e` in raw event | `eventType` in enriched event |
| -------------------------- | ---------------- | ----------------------------- |
| `SelfDescribing` (custom)  | ue               | unstruct                      |
| `ScreenView`               | ue               | unstruct                      |
| `Timing`                   | ue               | unstruct                      |
| `PageView`                 | pv               | page\_view                    |
| `Structured`               | se               | struct                        |
| `EcommerceTransaction`     | tr               | transaction                   |
| `EcommerceTransactionItem` | ti               | transaction\_item             |

> **Note:** `EcommerceTransaction`/`EcommerceTransactionItem` are a legacy design and will be deprecated soon.

`SelfDescribing` events (called `Unstructured` prior to v1) allow you to track anything that can be described by a [JSON schema](/docs/fundamentals/schemas/). The data you provide will be sent as a JSON inside the raw event payload. The specific type of JSON schema needed are described fully on the [next page](/docs/sources/java-tracker/custom-tracking-using-schemas/).

The `ScreenView` and `Timing` out-of-the-box event types are actually wrappers for `SelfDescribing` events: the methods for building those events represent fields in their hidden self-describing JSON schemas. This is why both `ScreenView` and `Timing` events are labelled "unstruct" in the data warehouse.

The `PageView` and `Structured` event types are processed differently from `SelfDescribing` events. Properties tracked using these events are not sent packaged as JSON, but as individual "atomic" fields in the raw event payload. We call these event types "canonical" or "primitive".

`SelfDescribing` events and canonical events are loaded into and modelled differently in your data warehouse. The "atomic" fields will always have individual columns.

`EcommerceTransaction` and `EcommerceTransactionItem` events are legacy primitive events. We recommend instead designing your own `SelfDescribing` events plus [context entities](/docs/sources/java-tracker/custom-tracking-using-schemas/) for eCommerce tracking.

### Tracking data that is not event-type specific

Some data, such as that relating to the user whose activity is being tracked, is relevant across all event types. The Java tracker provides two mechanisms for tracking this kind of data.

Certain properties, including `userId` or `ipAddress`, can be set as "atomic" properties in the raw event, using the `Subject` class. `Subject` properties relate mainly to client-side tracking. If you are using the Java tracker for server-side tracking, you may wish to pass client-side data for tracking server-side. This is discussed [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/).

A more general and powerful method is to attach self-describing JSON "context entities" to your events - the same JSON schemas as used for `SelfDescribing` events. This means that any data that can be described by a JSON schema can be added to any or all of your events. Read more on the [next page](/docs/sources/java-tracker/custom-tracking-using-schemas/).

All events also provide the option for setting a custom timestamp, called `trueTimestamp`. See below for details.

### Creating a custom event (SelfDescribing)

To track data using an `SelfDescribing` event, the data must be structured as a `SelfDescribingJson` object, discussed fully on the next page. These require two fields. The first is a URI for a self-describing JSON schema. The second is a data map, and the data must be valid against the schema. `SelfDescribing` events can be considered wrappers for sending `SelfDescribingJson`.

The simplest initialisation looks like this:

```java
SelfDescribing selfDescribing = SelfDescribing.builder()
            .eventData(dataAsSelfDescribingJson)
            .build();
```

See the API docs for the full [SelfDescribing.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/SelfDescribing.Builder.html) options.

### Creating a `ScreenView` event

Track screen views with the `ScreenView` event. It's a wrapper around a `SelfDescribing` event, using [this schema](https://github.com/snowplow/iglu-central/blob/324cfe9a5390174a56084722a545287d01a5a060/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

A simple initialisation looks like this:

```java
ScreenView screenView = ScreenView.builder()
            .name("human readable screen name")
            .id("unique screen ID")
            .build();
```

At least one of `name` or `id` must be set. See the API docs for the full [ScreenView.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/ScreenView.Builder.html) options.

### Creating a `Timing` event

Track how long something took with the `Timing` event. It's a wrapper around a `SelfDescribing` event, using [this schema](https://github.com/snowplow/iglu-central/blob/324cfe9a5390174a56084722a545287d01a5a060/schemas/com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0).

A simple initialisation looks like this:

```java
Timing timing = Timing.builder()
            .category("category of the timed event")
            .variable("name of the timed event")
            .timing(10) // in milliseconds
            .label("optional label")
            .build();
```

Provide your `timing` value in milliseconds. The `label` property is optional. See the API docs for the full [Timing.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/Timing.Builder.html) options.

### Creating a PageView event

Track page views with the `PageView` event. This is a "canonical" event type; data will end up in individual "atomic" columns in the data warehouse.

| Property     | Field in raw event | Column in enriched event |
| ------------ | ------------------ | ------------------------ |
| page URL     | url                | page\_url                |
| page title   | page               | page\_title              |
| referrer URL | refr               | page\_referrer           |

The provided URLs will also be decomposed into other columns, such as `page_urlscheme`, during event [enrichment](/docs/pipeline/enrichments/).

A simple initialisation looks like this:

```java
PageView pageViewEvent = PageView.builder()
            .pageUrl("https://www.snowplowanalytics.com")
            .pagetitle("Snowplow")
            .referrer("https://github.com/snowplow/snowplow-java-tracker")
            .build();
```

Only `pageUrl` is required. See the API docs for the full [PageView.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/PageView.Builder.html) options.

### Creating a `Structured` event

To track custom data without schemas, use `Structured` events. They are the "canonical" equivalent of `SelfDescribing` events. The provided data will end up in individual "atomic" columns in the data warehouse. Because of this, it's not possible to fully customise a `Structured` event: the fields cannot be renamed, nor new fields added. `Structured` events are designed to be similar to Google-style events.

The `Structured` event fields have flexible definitions, and what you put into each field is up to you. This is a double-edged sword. It's highly advisable to agree business-wide on definitions for each of these fields, before implementing tracking.

| Property | Often contains data about     | Field in raw event | Column in enriched event |
| -------- | ----------------------------- | ------------------ | ------------------------ |
| category | Grouping for the action       | se\_ca             | se\_category             |
| action   | Type of user activity         | se\_ac             | se\_action               |
| label    | Additional event data         | se\_la             | se\_label                |
| property | The action or object acted on | se\_pr             | se\_property             |
| value    | Numerical event data          | se\_va             | se\_value                |

A simple initialisation looks like this:

```java
Structured structured = Structured.builder()
                .category("category e.g. auth")
                .action("action e.g. logout")
                .label("optional label")
                .property("optional property")
                .value(12.34)
                .build();
```

Both `category` and `action` are required. See the API docs for the full [Structured.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/Structured.Builder.html) options.

### Creating `EcommerceTransaction` and `EcommerceTransactionItem` events

To track eCommerce data, we recommend designing your own schemas for an `SelfDescribing` event. We suggest creating one schema for the overall transaction, and another schema for individual items. The transaction schema can be used for the `SelfDescribing` event. Items in the transaction can be added as [context entities](/docs/sources/java-tracker/custom-tracking-using-schemas/).

The `EcommerceTransaction` and `EcommerceTransactionItem` events are legacy events, and it's highly likely they will be deprecated in the future. They are designed to be sent together, with one `EcommerceTransaction` containing an `EcommerceTransactionItem` for every item in the transaction. When the `EcommerceTransaction` event is tracked, the `EcommerceTransactionItem` events are extracted and sent separately. This means that although you have only tracked one event (using `Tracker.track()`), multple events are generated.

`EcommerceTransaction` and `EcommerceTransactionItem` are "canonical" events. The data will end up in individual "atomic" columns in the data warehouse.

| `EcommerceTransaction` property | Field in raw event | Column in enriched event |
| ------------------------------- | ------------------ | ------------------------ |
| orderId                         | tr\_id             | tr\_orderid              |
| totalValue                      | tr\_tt             | tr\_total                |
| affiliation                     | tr\_af             | tr\_affiliation          |
| taxValue                        | tr\_tx             | tr\_tax                  |
| shipping                        | tr\_sh             | tr\_shipping             |
| city                            | tr\_ci             | tr\_city                 |
| state                           | tr\_st             | tr\_state                |
| country                         | tr\_co             | tr\_country              |
| currency                        | tr\_cu             | tr\_currency             |

| `EcommerceTransactionItem` property | Field in raw event | Column in enriched event |
| ----------------------------------- | ------------------ | ------------------------ |
| itemId                              | ti\_id             | ti\_orderid              |
| sku                                 | ti\_sk             | ti\_sku                  |
| price                               | ti\_pr             | ti\_price                |
| quantity                            | ti\_qu             | ti\_quantity             |
| name                                | ti\_nm             | ti\_name                 |
| category                            | ti\_ca             | ti\_category             |
| currency                            | ti\_cu             | ti\_currency             |

The `orderId` and `itemId` should be the same, as it's the most direct way to associate the two events once they are in the warehouse.

A simple initialisation looks like this:

```java
// Create events for each item in the transaction
EcommerceTransactionItem item = EcommerceTransactionItem.builder()
        .itemId("should be the same as order_id")
        .sku("sku")
        .price(1.0)
        .quantity(2)
        .name("item name")
        .category("category")
        .currency("currency")
        .build();

// Then make the EcommerceTransaction event
EcommerceTransaction ecommerceTransaction = EcommerceTransaction.builder()
        .items(item) // Add the EcommerceTransactionItem events
        .orderId("should be the same as item_id")
        .totalValue(2.0)
        .affiliation("affiliation")
        .taxValue(2.0)
        .shipping(3.0)
        .city("city")
        .state("state")
        .country("country")
        .currency("currency")
        .build();
```

For `EcommerceTransactionItem` events, `itemId`, `sku`, `price` and `quantity` are required. For `EcommerceTransaction` events, only `orderId` and `totalValue` are required. See the API docs for the full [EcommerceTransaction.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/EcommerceTransaction.Builder.html) and [EcommerceTransactionItem.Builder](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/events/EcommerceTransactionItem.Builder.html) options.

### Adding custom timestamps to events

Snowplow events have several timestamps. The raw event payload always contains a `deviceCreatedTimestamp` (`dtm`) and a `deviceSentTimestamp` (`stm`). Other timestamps are added as the event moves through the pipeline.

Every `Event.Builder` in the Java tracker allows for a custom timestamp, called `trueTimestamp` to be set. A `trueTimestamp` can be added to any event using the `trueTimestamp()` Builder method:

```java
// This example shows an Unstructured event, but all events can have a trueTimestamp
SelfDescribing selfDescribing = SelfDescribing.builder()
            .eventData(dataAsSelfDescribingJson)
            .trueTimestamp(1647616394785L)
            .build();
```

`trueTimestamp` should be a `Long` representing milliseconds since the Unix epoch.

---

# Track client side properties with the Java tracker
> Add user properties like user ID, IP address, and device information to events using the Subject class for client-side tracking.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/tracking-specific-client-side-properties/

An event describes a single, transient activity. The context in which that event occurs - the relatively persistent environment - is also incredibly valuable data. As discussed on the previous page, the most powerful way to track such data is to use [custom self-describing JSON entities](/docs/sources/java-tracker/custom-tracking-using-schemas/). These are sent as JSON in the raw event payload. Depending on your data warehouse, event context entities are loaded into their own tables, or remain as JSON in the loaded event.

The Java tracker provides an additional "canonical" method to track a small subset of contextual data, using the `Subject` class. As with the "canonical" [Event types](/docs/sources/java-tracker/tracking-events/), e.g. `PageView`, `Subject` properties directly populate the "atomic" fields of the raw event payload. These data will always end up in individual columns of the main event table in the data warehouse.

The fields tracked using `Subject` tend to be most relevant in client-side tracking. Some are set automatically in all events during enrichment, even when no `Subject` is added. These properties are marked with `*` below, and discussed below. Timezone, marked with `**`, is set when a `Subject` is tracked with the event.

Add these fields to an event using `Subject`:

| Property         | Field in raw event | Column(s) in enriched event           |
| ---------------- | ------------------ | ------------------------------------- |
| userId           | uid                | user\_id                              |
| ipAddress\*      | ip                 | user\_ipaddress                       |
| timezone\*\*     | tz                 | os\_timezone                          |
| language         | lang               | br\_lang                              |
| useragent\*      | ua                 | useragent                             |
| viewport         | vp                 | br\_viewheight, br\_viewwidth         |
| screenResolution | res                | dvce\_screenheight, dvce\_screenwidth |
| colorDepth       | cd                 | br\_colordepth                        |
| networkUserId\*  | tnuid              | network\_userid                       |
| domainUserId     | duid               | domain\_userid                        |
| domainSessionId  | sid                | domain\_sessionid                     |

Note: the ability to set `domainSessionId` was added in version 0.11.

These properties are especially useful for client-side tracking, or for linking client-side and server-side tracking. We highly recommend tracking in both client-side and server-side, if it's relevant to your application. Read more about this in these [blog](https://snowplowanalytics.com/blog/2019/02/05/how-server-side-tracking-fills-holes-in-your-data-and-improves-your-analytics/) [posts](https://snowplowanalytics.com/blog/2021/11/09/the-unrivaled-power-of-joining-client-and-server-side-tracking/).

If you are also using the Javascript tracker, it will set [cookies](/docs/sources/web-trackers/cookies-and-local-storage/) in the browser. The `Subject` properties `domainUserId`, and `domainSessionId` are intended to be used for extracted cookie values passed to the Java tracker. If you want to track other identification tokens, we recommend creating schemas and using context entities.

As always, be aware of privacy when tracking [personal identifiable information](https://snowplowanalytics.com/blog/2020/09/06/user-identification-and-privacy/) such as email addresses or IP addresses.

### Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

The `user_ipaddress` is automatically added to all enriched events. To manually override this, use a `Subject` and set a `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added during enrichment. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, use a `Subject` and set a `useragent` string. For the default Tracker configuration with OkHttp, the default `useragent` will be "okhttp/4.2.2".

The `network_userid` is the cookie value for the event collector’s third-party cookie. It is the server-side user identifier. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). The default behavior is for the collector to provide a new cookie/`network_userid` for each event it receives. To override the collector cookie’s value with your own generated ID, use a `Subject` object and set `networkUserId`. Alternatively, configure a `CookieJar` to persist the `sp` cookie, and the `network_userid`, across requests (see [here](/docs/sources/java-tracker/configuring-how-events-are-sent/#persisting-cookies-using-a-cookiejar)).

A further property, `timezone`, is generated automatically during `Subject` initialization, based on `Calendar.getInstance().getTimeZone()`. Therefore, this will be added to all events with a `Subject` attached. The default will be overriden if `timezone` is provided explicitly.

### Adding `Subject` properties to events

There are two ways to track the `Subject` "atomic" properties in your events: on an event-by-event basis; or globally, to affect all events. A combination of both methods is also possible.

A simple `Subject` initialisation looks like this:

```java
SubjectConfiguration subjectConfig = new SubjectConfiguration()
        .userId("java@snowplowanalytics.com");
Subject subject = new Subject(subjectConfig);
```

In this case only the `userId` (and `timezone`) were set. There are no required properties for the `SubjectConfiguration`.

A `Subject` can be added to any event using the `subject()` `Event.Builder` method:

```java
// This example shows an SelfDescribing event, but all events can have a Subject
SelfDescribing selfDescribing = SelfDescribing.builder()
            .eventData(dataAsSelfDescribingJson)
            .subject(subject)
            .build();
```

To set `Subject` properties in all subsequent events, add a `Subject` to your `Tracker` object:

```java
// A Subject can be provided at Tracker initialisation
// Indirectly, using the Snowplow interface and a SubjectConfiguration object
SubjectConfiguration subjectConfig = new SubjectConfiguration().useragent("useragent");
Tracker tracker = Snowplow.createTracker(
        new TrackerConfiguration("trackerNamespace", "appId"),
        new NetworkConfiguration("http://collector-url"),
        subjectConfig);

// Creating a Tracker directly using Subject and Emitter objects
Subject subject = new Subject();
BatchEmitter emitter = new BatchEmitter(new NetworkConfiguration("http://endpoint"));
Tracker tracker = new Tracker(new TrackerConfiguration("trackerNamespace", "appId"), emitter, subject);

// Alternatively, a Subject can be set later
Tracker tracker = Snowplow.createTracker("trackerNamespace", "appId", "http://collectorEndpoint");
Subject subject = new Subject();
tracker.setSubject(subject);
```

Subject properties can be updated or added to after initialization, using setter methods. See the [API docs](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Subject.html) for full details.

It's possible to use both `Event`-specific and `Tracker`-associated `Subject` objects simultaneously. Fields from both `Subject` objects are added to the payload, with the `Event`-specific `Subject` having priority.

```java
// Adding a global, Tracker-associated Subject
SubjectConfiguration trackerSubjectConfig = new SubjectConfiguration()
        .language("EN")
        .useragent("Snowplow");

Tracker tracker = Snowplow.createTracker(
        new TrackerConfiguration("trackerNamespace", "appId"),
        new NetworkConfiguration("http://collector-url"),
        trackerSubjectConfig);

// Adding an Event-specific Subject
SubjectConfiguration eventSubjectConfig = new SubjectConfiguration()
        .userId("java@snowplowanalytics.com")
        .useragent("Mozilla/5.0");

SelfDescribing selfDescribing = SelfDescribing.builder()
        .eventData(dataAsSelfDescribingJson)
        .subject(new Subject(eventSubjectConfig))
        .build();

// Tracking the event
tracker.track(selfDescribing);
```

The resulting enriched event would have these `Subject` atomic columns populated:

| Column in enriched event | Value / example value                       | Source         |
| ------------------------ | ------------------------------------------- | -------------- |
| user\_id                 | "<java@snowplowanalytics.com>"              | eventSubject   |
| os\_timezone             | e.g. "Europe/London"                        | eventSubject   |
| br\_lang                 | "EN"                                        | trackerSubject |
| useragent                | "Mozilla/5.0"                               | eventSubject   |
| network\_userid          | e.g. "8383057f-2769-4321-ad72-58fa1b22e4b3" | pipeline       |

---

# Using multiple trackers in one Java app
> Instantiate and manage multiple Java tracker instances with different configurations using the Snowplow interface.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/using-multiple-trackers/

Rarely, a tracking implementation may benefit from instantiating multiple Java trackers, each with a different configuration. This will not be appropriate for most Snowplow users.

Creating and managing multiple trackers is easiest using the `Snowplow` interface (added in v1.0.0). All trackers created with `Snowplow.createTracker()` are added to a private static map. Different `Tracker` objects are distinguished using their namespaces.

In this example, the user wants to send some events to a different collector. They create two Trackers, one for each endpoint:

```java
Snowplow.createTracker("main_tracker", "appId", "http://endpoint");
Snowplow.createTracker("other_tracker", "appId", "http://other-endpoint");

Snowplow.getDefaultTracker().track(event);
Snowplow.getTracker("other_tracker").track(event2);
```

See the API docs for the full [Snowplow](https://snowplow.github.io/snowplow-java-tracker/index.html?com/snowplowanalytics/snowplow/tracker/Snowplow.html) details.

## Using different Snowplow trackers together

Because all Snowplow trackers create events with the same structure, it's easy to work with data from different Snowplow tracker SDKs. For example, you may wish to implement the Java tracker for back-end server tracking, in conjunction with a client-side tracker such as JavaScript or Flutter.

Read more about tracking client-side/server-side [here](/docs/sources/java-tracker/tracking-specific-client-side-properties/).

---

# What do Java tracker events look like?
> Understand the structure of enriched Java tracker events with example JSON payloads for different event types.
> Source: https://docs.snowplow.io/docs/sources/java-tracker/what-do-java-tracker-events-look-like/

See [Tracking events](/docs/sources/java-tracker/tracking-events/) to learn how to track events with the Java tracker.

All Snowplow events, regardless of the tracker used, look similar. They all currently contain 137 columns/properties. Some are populated by pipeline enrichments, e.g. the [campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) populates the `mkt_` columns. Snowplow event structure and columns are explained [here](/docs/fundamentals/canonical-event/).

This reference page shows two simple example events generated using the default tracker configuration and a [Snowplow Micro](/docs/testing/snowplow-micro/) testing pipeline. These events were tracked from a simplified version of the "simple-console" demo provided in the [Java tracker Github repository](https://github.com/snowplow/snowplow-java-tracker). While most Snowplow pipelines transform events into table rows, Micro has an API that provides events in JSON format.

The JSON below is part of the "event" value for a [ScreenView](/docs/sources/java-tracker/tracking-events/#creating-a-screenview-event) event, which is a type of [SelfDescribing](/docs/sources/java-tracker/tracking-events/#creating-a-custom-event-selfdescribing) event. The full Micro JSON output includes other data, such as headers. Any columns with `null` values have also been removed.

```java
// This event was tracked
ScreenView screenView = ScreenView.builder()
            .name("login")
            .id("auth123")
            .build();
```

```json
"ScreenView event (edited)": {
  "app_id": "java-tracker-sample-console-app",
  "platform": "srv",
  "etl_tstamp": "2022-03-22T15:49:56.999Z",
  "collector_tstamp": "2022-03-22T15:49:56.957Z",
  "dvce_created_tstamp": "2022-03-22T15:49:56.734Z",
  "event": "unstruct",
  "event_id": "f9a58d5d-cd13-4bfe-b9b2-5136ce1cb9fa",
  "name_tracker": "demo",
  "v_tracker": "java-0.12.0",
  "v_collector": "ssc-2.3.0-stdout$",
  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",
  "user_ipaddress": "manually removed for privacy",
  "network_userid": "666219d1-9b9c-4fe1-b694-c8e94dd5a706",
  "unstruct_event": {
    "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
    "data": {
      "schema": "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0",
      "data": { "id": "auth123", "name": "login" }
    }
  },
  "useragent": "okhttp/4.2.2",
  "dvce_sent_tstamp": "2022-03-22T15:49:56.757Z",
  "derived_tstamp": "2022-03-22T15:49:56.934Z",
  "event_vendor": "com.snowplowanalytics.snowplow",
  "event_name": "screen_view",
  "event_format": "jsonschema",
  "event_version": "1-0-0",
}
```

Most of these properties are the default Java tracker properties, which will be present in all Java tracker events. For example, `app_id`,`platform`, and `name_tracker` were set at `Tracker` initialization.

Since this is a `SelfDescribing` event (previously called `Unstructured`), the "unstruct\_event" field is also populated. The [self-describing JSON](/docs/sources/java-tracker/custom-tracking-using-schemas/) provided as part of the `SelfDescribing`/`ScreenView` event has been validated against the [Iglu](/docs/api-reference/iglu/) schema registry.

Below is an example `PageView` event using the default tracker configuration with Snowplow Micro. Again, the `null` columns and other sections of the payload have been removed.

```java
// This event was tracked
PageView pageViewEvent = PageView.builder()
            .pageUrl("https://www.snowplowanalytics.com")
            .build();
```

```json
"PageView event (edited)": {
  "app_id": "java-tracker-sample-console-app",
  "platform": "srv",
  "etl_tstamp": "2022-03-22T17:02:44.394Z",
  "collector_tstamp": "2022-03-22T17:02:44.268Z",
  "dvce_created_tstamp": "2022-03-22T17:02:43.913Z",
  "event": "page_view",
  "event_id": "64ae291c-2170-4643-a957-0b2f766c8ac8",
  "name_tracker": "demo",
  "v_tracker": "java-0.12.0",
  "v_collector": "ssc-2.3.0-stdout$",
  "v_etl": "snowplow-micro-1.1.2-common-2.0.1",
  "user_ipaddress": "manually removed for privacy",
  "network_userid": "bbf5eda0-f3f7-48b9-9068-b12e721a7b9c",
  "page_url": "https://www.snowplowanalytics.com",
  "page_urlscheme": "https",
  "page_urlhost": "www.snowplowanalytics.com",
  "page_urlport": 443,
  "useragent": "okhttp/4.2.2",
  "dvce_sent_tstamp": "2022-03-22T17:02:43.926Z",
  "derived_tstamp": "2022-03-22T17:02:44.255Z",
  "event_vendor": "com.snowplowanalytics.snowplow",
  "event_name": "page_view",
  "event_format": "jsonschema",
  "event_version": "1-0-0",
}
```

Because PageView is a "primitive" [event type](/docs/sources/java-tracker/tracking-events/), the provided URL has populated the "atomic" `page_url` column. The different parts of the URL - scheme and host - have been automatically extracted and added to their own columns.

---

# Lua tracker SDK
> Track events in Lua applications with the Snowplow Lua tracker SDK, compatible with Lua 5.1+ and LuaJIT.
> Source: https://docs.snowplow.io/docs/sources/lua-tracker/

The Snowplow Lua Tracker allows you to track Snowplow events in your Lua applications.

The current version is 0.2.0. It is compatible with Lua >= 5.1, along with LuaJIT.

This will help you set up the Lua tracker and get started tracking. For more technical details, you can visit the [API docs](https://snowplow.github.io/snowplow-lua-tracker/).

## Getting Started

### Requirements

Ensure you have the following installed on your system:

- [Lua](https://www.lua.org/) version >= 5.1
- [LuaRocks](https://luarocks.org/) (Lua's dependency manager)
- curl

If using `brew`, simply run:

```bash
brew install lua luarocks curl
```

### Installation

Once these are installed, you can install the tracker using LuaRocks:

```bash
luarocks install snowplowtracker
```

Note: You may find that you need to pass in the CURL\_DIR flag if Lua cannot find `curl` by itself. Below is an example if `curl` was installed using `brew` on an Intel Mac.

```bash
luarocks install snowplowtracker CURL_DIR=/usr/local/Cellar/curl/7.82.0/
```

Or, add the Snowplow Tracker to the dependencies section of your rockspec (Note: you may still have to pass `CURL_DIR` when installing):

```gradle
dependencies = {
  ...
  "SnowplowTracker"
}
```

### Tracking Events

To track an event, simply create a tracker instance and call one of the `track_*` methods. For example, simple tracking of a [structured event](/docs/events/custom-events/#structured-events) can be done as follows:

```lua
local snowplow = require("snowplow")
local tracker = snowplow.new_tracker("{{ collector_url }}")

tracker:track_struct_event("category", "action", "label", "property", 10)
```

Visit documentation about [tracking events](/docs/sources/lua-tracker/tracking-specific-events/) to learn about other supported event types.

---

# Track events with the Lua tracker
> Track screen views, structured events, and custom self-describing events with the Lua tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/lua-tracker/tracking-specific-events/

Creating a tracking instance is as simple as calling `snowplow.new_tracker` and providing a URL for your collector:

```lua
local snowplow = require("snowplow")
local tracker = snowplow.new_tracker("{{ collector_url }}")
```

Tracking methods supported by the Lua Tracker:

| Method                         | Event type tracked                                  |
| ------------------------------ | --------------------------------------------------- |
| track\_screen\_view            | View of screen                                      |
| track\_struct\_event           | Semi-custom structured event                        |
| track\_self\_describing\_event | Custom event based on “self-describing” JSON schema |

## Track structured events with `track_struct_event`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. “Struct” events closely mirror the structure of Google Analytics events, with “category”, “action”, “label”, and “value” properties.

| Parameter | Description                                                    | Required in event? |
| --------- | -------------------------------------------------------------- | ------------------ |
| category  | The grouping of structured events which this action belongs to | Yes                |
| action    | Defines the type of user interaction which this event involves | Yes                |
| label     | Often used to refer to the ‘object’ the action is performed on | No                 |
| property  | Describing the ‘object’, or the action performed on it         | No                 |
| value     | Provides numerical data about the event                        | No                 |

```lua
tracker:track_struct_event("shop", "add-to-basket", "book", "pcs", 2)
```

## Track self-describing events with `track_self_describing_event`

Use `track_self_describing_event` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure. A guide to understanding Self-Describing events is [available here](/docs/fundamentals/events/#self-describing-events).

| Parameter | Description                            | Required in event? |
| --------- | -------------------------------------- | ------------------ |
| schema    | The schema to use                      | Yes                |
| action    | The data to send along with the schema | Yes                |

```lua
tracker:track_self_describing_event(
  "iglu:com.snowplowanalytics.snowplow/add_to_cart/jsonschema/1-0-0",
  { sku = "ASO01043", unitPrice = 49.95, quantity = 1000 }
)
```

## Track screen views with `track_screen_view`

Use track\_screen\_view to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages.

This method uses a self-describing event with the [`com.snowplowanalytics.snowplow/screen_view` schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0).

```lua
tracker:track_screen_view("Character Creation - Step 1", "c1")
```

## Configuring the Tracker

The `new_tracker` method has a couple of other parameters that can be used to configure your tracker instance, along with the collector `url`.

| Parameter      | Description                                          | Required | Default |
| -------------- | ---------------------------------------------------- | -------- | ------- |
| url            | The Snowplow Collector URL                           | Yes      | -       |
| request\_type  | The request type to use ("GET" or "POST")            | No       | "POST"  |
| encode\_base64 | If self-describing event payloads are base64 encoded | No       | true    |

Note: It's generally recommended to use the default values as `encode_base64 = true` can reduce the size of payloads and `POST` will receive more future support (such as batched requests), but you can change these if your use case calls for it.

## Tracker Method Return Values

A call to a `track_*` method will return two values, a boolean if the request was successful, and an optional error message if the request was not successful.

An example of the values returned from a failed request:

```lua
local ok, err = tracker:track_screen_view("Character Configuration - Part 1", "c1")
-- false, Host [https://test.invalid/com.snowplowanalytics.snowplow/tp2] not found (possible connectivity error)
```

An example of the values returned from a successful request:

```lua
local ok, err = tracker:track_screen_view("Character Configuration - Part 1", "c1")
-- true, nil
```

## Adding user and platform data

The tracker can store information about the user associated with the event, such as their `user_id`, what type of device they used, or what size screen that device had. It also stores which platform the event occurred on – e.g. server-side app, mobile, games console, etc. This is done through the provided `set_*` methods available on a tracker instance. The stored information is attached to the tracked events using fields described in the [Tracker Protocol](/docs/events/).

For a full list of setters, check out the [Tracker API Documentation](https://snowplow.github.io/snowplow-lua-tracker/modules/Tracker.html).

---

# Google Play Data safety (Android)
> Guidance for declaring data collection and handling practices when publishing Android apps using the Snowplow tracker to Google Play.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/android-google-play-data-safety/

Google Play require developers to declare how they collect and handle user data within their app. The provided information is shown on the Google Play store listing.

This table provides guidance for publishing apps incorporating our Snowplow Android Tracker.

| Category                    | Data type                    | Description                                                                                                                                                                                     | Snowplow Android tracker feature                                                                                                                                                                                                        |
| --------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Location                    | Approximate location         | User or device physical location to an area greater than or equal to 3 square kilometers, such as the city a user is in, or location provided by Android's `ACCESS_COARSE_LOCATION` permission. | Geolocation context (optional and disabled by default in `TrackerConfiguration` `geoLocationContext`)                                                                                                                                   |
|                             | Precise location             | User or device physical location within an area less than 3 square kilometers, such as location provided by Android's `ACCESS_FINE_LOCATION` permission.                                        |                                                                                                                                                                                                                                         |
| App activity                | Page views and taps in app   | Information about how a user interacts with your app. For example, the number of page views or the screen coordinates of taps.                                                                  | ScreenView automatic tracking (optional but enabled by default in `TrackerConfiguration` `screenViewAutotracking`)                                                                                                                      |
|                             | In-app search history        | Information about what a user has searched for in your app.                                                                                                                                     | Not tracked                                                                                                                                                                                                                             |
|                             | Installed apps               | Information about the apps installed on a user's device.                                                                                                                                        | Not tracked                                                                                                                                                                                                                             |
|                             | Other user-generated content | Any other user-generated content not listed here, or in any other section. For example, user bios or notes.                                                                                     | Not tracked                                                                                                                                                                                                                             |
|                             | Other actions                | Any other user activity or actions in-app not listed here such as gameplay and likes.                                                                                                           | Session automatic tracking (optional but enabled by default in `TrackerConfiguration` `sessionContext`). Background–Foreground transition tracking (optional and disabled by default in `TrackerConfiguration` `lifecycleAutotracking`) |
| Web browsing                | Web browsing history         | Information about the websites a user has visited.                                                                                                                                              | Not tracked                                                                                                                                                                                                                             |
| App info and performance    | Crash logs                   | Crash log data from your app. For example, the number of times your app has crashed, stack traces, or other information directly related to a crash.                                            | Crash reporting automatic tracking (optional but enabled by default in `TrackerConfiguration` `exceptionAutotracking`)                                                                                                                  |
|                             | Diagnostics                  | Information about the performance of your app. For example battery life, loading time, latency, framerate, or any technical diagnostics.                                                        | Platform context (optional but enabled by default in `TrackerConfiguration` `platformContext`)                                                                                                                                          |
|                             | Other app performance data   | Any other app performance data not listed here.                                                                                                                                                 | Not tracked                                                                                                                                                                                                                             |
| Device or other identifiers | Device or other identifiers  | Identifiers that relate to an individual device, browser or app. For example, an IMEI number, MAC address, Widevine Device ID, Firebase installation ID, or advertising identifier.             | Advertising identifier (AAID, also called IDFA) (optional, enabled by default as part of `TrackerConfiguration` `platformContext` but also requires extra configuration)                                                                |

> **Note:** The IP address of the device can be tracked by the collector. Disable this using [server-side anonymisation](/docs/sources/mobile-trackers/anonymous-tracking/).

---

# Anonymous tracking in the native mobile trackers
> Enable anonymous tracking to mask user identifiers and session information in mobile tracker events for privacy compliance.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/anonymous-tracking/

> **Info:** This feature is available since version 4.

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

On mobile, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress`, set by you in `Subject`
  - `userId`, a device ID, set automatically in the [session](/docs/sources/web-trackers/tracking-events/session/) entity
  - IDFA identifiers (`appleIdfa` and `appleIdfv` for iOS, `androidIdfa` for Android) in the mobile entity

- Client-side session identifiers:
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property in the `Subject`, that value has priority.
>
> Similarly, if you set the `network_userid` property in the `Subject`, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

There are several levels to the anonymization depending on which of the categories are affected.

## 1. Full client-side anonymization

In this case, we want to anonymize both the client-side user identifiers as well as the client-side session identifiers. This means disabling the Session entity altogether and enabling user anonymization:

**iOS:**

```swift
let config = TrackerConfiguration()
    .sessionContext(false) // Session context entity won't be added to events
    .userAnonymisation(true) // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

**Android (Kotlin):**

```kotlin
val config = TrackerConfiguration("appId")
    .sessionContext(false) // Session context entity won't be added to events
    .userAnonymisation(true) // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

**Android (Java):**

```java
TrackerConfiguration config = new TrackerConfiguration("appId")
    .sessionContext(false) // Session context entity won't be added to events
    .userAnonymisation(true); // User identifiers in Platform context (IDFA and IDFV) will be anonymized
```

***

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ no session entity                           |
| `sessionId`         | Session entity           | ❌ no session entity                           |
| `previousSessionId` | Session entity           | ❌ no session entity                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

## 2. Client-side anonymisation with session tracking

This setting disables client-side user identifiers are but tracks session information. In practice, this means that events track the Session entity but the `userId` property is a null UUID (`00000000-0000-0000-0000-000000000000`). If the Mobile (platform) entity is enabled, the IDFA identifiers will not be present.

**iOS:**

```swift
let config = TrackerConfiguration()
    .sessionContext(true) // Session entity is tracked with the session ID
    .userAnonymisation(true) // User identifiers in Session and Mobile (platform) entity are anonymized
```

**Android (Kotlin):**

```kotlin
val config: TrackerConfiguration = TrackerConfiguration("appId")
    .sessionContext(true) // Session entity is tracked with the session ID
    .userAnonymisation(true) // User identifiers in Session and Mobile (platform) entity are anonymized
```

**Android (Java):**

```java
TrackerConfiguration config = new TrackerConfiguration("appId")
    .sessionContext(true) // Session entity is tracked with the session ID
    .userAnonymisation(true); // User identifiers in Session and Mobile (platform) entity are anonymized
```

***

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ null UUID                                   |
| `sessionId`         | Session entity           | ✅                                             |
| `previousSessionId` | Session entity           | ❌                                             |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

> **Note:** When anonymous tracking is enabled or disabled using `tracker.setUserAnonymisation(true | false)`, the tracker starts a new session. This results in a new `sessionId`.

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set server-side. In particular, these are the `network_userid` property set in server-side cookie and the user IP address. You can anonymize the properties using the `serverAnonymisation` flag in `EmitterConfiguration`:

**iOS:**

```swift
let config = EmitterConfiguration()
    .serverAnonymisation(true)
```

**Android (Kotlin):**

```kotlin
val config = EmitterConfiguration()
    .serverAnonymisation(true)
```

**Android (Java):**

```java
EmitterConfiguration config = new EmitterConfiguration()
    .serverAnonymisation(true);
```

***

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅                                                   |
| `domain_userid`     | Atomic                   | ✅                                                   |
| `userId`            | Session entity           | ✅                                                   |
| `sessionId`         | Session entity           | ✅                                                   |
| `previousSessionId` | Session entity           | ✅                                                   |
| `appleIdfa`         | Mobile (platform) entity | ✅                                                   |
| `appleIdfv`         | Mobile (platform) entity | ✅                                                   |
| `androidIdfa`       | Mobile (platform) entity | ✅                                                   |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

Setting the flag will add a `SP-Anonymous` HTTP header to requests sent to the Snowplow Collector. The Snowplow pipeline will take care of anonymizing the identifiers.

## 4. Full anonymization

Full anonymization combines client-side and server-side anonymization, to remove all user and session identifiers from events. This approach provides the highest level of privacy.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .sessionContext(false) // Session entity won't be added to events
    .userAnonymisation(true) // User identifiers in Platform context will be anonymized

let emitterConfig = EmitterConfiguration()
    .serverAnonymisation(true) // Collector won't track network_userid or user IP
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .sessionContext(false) // Session entity won't be added to events
    .userAnonymisation(true) // User identifiers in Platform context will be anonymized

val emitterConfig = EmitterConfiguration()
    .serverAnonymisation(true) // Collector won't track network_userid or user IP
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .sessionContext(false) // Session entity won't be added to events
    .userAnonymisation(true); // User identifiers in Platform context will be anonymized

EmitterConfiguration emitterConfig = new EmitterConfiguration()
    .serverAnonymisation(true); // Collector won't track network_userid or user IP
```

***

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

---

# Track client side properties in the native mobile trackers
> Configure user and device properties to automatically attach to all mobile tracker events using SubjectConfiguration.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/client-side-properties/

An event describes a single, transient activity. The context in which that event occurs - the relatively persistent environment - is also incredibly valuable data.

The tracker allows the addition of a persistent set of information through the `SubjectConfiguration`, which represents the basic information about the user and the app. This data is added to every event as part of the [canonical event properties](/docs/fundamentals/canonical-event/).

| Property           | Description                 | Automatically added?  | Column(s) in enriched event             |
| ------------------ | --------------------------- | --------------------- | --------------------------------------- |
| `userId`           | User identifier.            | ❌                     | `user_id`                               |
| `ipAddress`        | User IP address.            | ✅                     | `user_ipaddress`                        |
| `timezone`         | Current timezone label.     | ✅                     | `os_timezone`                           |
| `language`         | Language set in the device. | ✅                     | `br_lang`                               |
| `useragent`        | User-agent.                 | ✅                     | `useragent`                             |
| `viewport`         | Screen viewport.            | ✅ (iOS) / ❌ (Android) | `br_viewheight`, `br_viewwidth`         |
| `screenResolution` | Screen resolution.          | ✅                     | `dvce_screenheight`, `dvce_screenwidth` |
| `colorDepth`       | Screen color depth.         | ❌                     | `br_colordepth`                         |
| `networkUserId`    | Network user ID.            | ✅                     | `network_userid`                        |
| `domainUserId`     | Domain user ID.             | ❌                     | `domain_userid`                         |

As always, be aware of privacy when tracking [personal identifiable information](https://snowplow.io/blog/2020/09/06/user-identification-and-privacy/) such as email addresses or IP addresses. The tracker provides anonymous tracking functionality to mask certain user identifiers. Refer to the [section on anonymous tracking to learn more](/docs/sources/mobile-trackers/anonymous-tracking/).

> **Note:** The default screen resolution values are fetched from `WindowManager`, an older Android API. Screen resolution is also reported in the [platform context entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/), which obtains the screen size from the Android context resources. The height value will likely be higher in `dvce_screenheight` than in the context entity, as the `WindowManager` size includes the menu bar.
>
> To use the modern API throughout the tracker and standardize the screen resolution between event and entity properties, set `useContextResourcesScreenResolution(true)` flag in `SubjectConfiguration`. This flag is off by default, and available from Android tracker v6.0.3 onwards.

## Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

Unless [anonymous tracking with server anonymisation](/docs/sources/mobile-trackers/anonymous-tracking/) is enabled, the `user_ipaddress` is automatically added to all enriched events. To manually override this without using anonymous tracking, use a `SubjectConfiguration` and set an `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added but it can be overriden on configuration. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, set a `useragent` string.

The `network_userid` is the cookie value for the event collector's third-party cookie. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). To override the collector cookie’s value with your own generated ID, set `networkUserId`. Again, anonymous tracking with server anonymisation will prevent the addition of this property to events. The `network_userid` is stored in the tracker and is kept the same until the app is deleted or the collector endpoint is changed or the cookie is expired.

All other `Subject` properties can also be manually overriden via `SubjectConfiguration` and `SubjectController`.

## Set the `Subject` properties

Initialize the tracker with a `SubjectConfiguration` like this:

**iOS:**

```swift
let subjectConfig = SubjectConfiguration()
    .userId("username")

let networkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com")
Snowplow.createTracker(
    namespace: "appTracker",
    network: networkConfig,
    configurations: [subjectConfig]
)
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/subjectconfiguration).

**Android (Kotlin):**

```kotlin
val subjectConfig = SubjectConfiguration()
    .userId("username")

val networkConfig = NetworkConfiguration("[http:/endpoint](https://snowplow-collector-url.com)")
Snowplow.createTracker(
    applicationContext,
    "namespace",
    networkConfig,
    subjectConfig
)
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/-subject-configuration/index.html).

**Android (Java):**

```java
SubjectConfiguration subjectConfig = new SubjectConfiguration()
    .userId("username");

NetworkConfiguration networkConfig = new NetworkConfiguration(
    "https://snowplow-collector-url.com"
);
Snowplow.createTracker(
    getApplicationContext(),
    "appTracker",
    networkConfig,
    subjectConfig
);
```

See the API docs for the full [list of options](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/-subject-configuration/index.html).

***

Subject properties can be updated or added to after initialization using the `SubjectController` retrieved from the tracker:

**iOS:**

```swift
let tracker = Snowplow.defaultTracker()
tracker?.subject?.userId = ""
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.defaultTracker
tracker?.subject?.userId = ''
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.getDefaultTracker();
tracker.getSubject().setUserId("");
```

***

---

# Configure how events are sent in the native mobile trackers
> Configure network connections, event batching, and retry policies for sending events from mobile trackers to your collector.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/configuring-how-events-are-sent/

A user interacts with your app: an event is generated and tracked using `track` method of `TrackerController`. But the event must be sent to your event collector, to enter your pipeline, before it has any value.

The tracker allows the configuration of the network connection, event sending, and buffering of events. The default configurations will be sufficient for many Snowplow users but a better fine tuning can be set through the `EmitterConfiguration`.

## Configuring the network connection

The `NetworkConfiguration` is used to specify the collector endpoint:

**iOS:**

```swift
let networkConfig = NetworkConfiguration(endpoint: "http://collector-endpoint")
```

**Android (Kotlin):**

```kotlin
val networkConfig = NetworkConfiguration("http://collector-endpoint")
```

**Android (Java):**

```java
NetworkConfiguration networkConfig = new NetworkConfiguration("http://collector-endpoint");
```

***

The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

In particular cases it can be useful to have a full control of the component in charge to send the events. This can be achieved with a custom `NetworkConnection` that will take care to send the events to the collector:

**iOS:**

```swift
let network = DefaultNetworkConnection(urlString: url, httpMethod: method)
network.emitThreadPoolSize = 20
network.byteLimitPost = 52000

let networkConfig = NetworkConfiguration(networkConnection: network)
```

In the example above we used the `DefaultNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

**Android (Kotlin):**

```kotlin
val connection = OkHttpNetworkConnectionBuilder(url, applicationContext)
    .method(method)
    .emitTimeout(10)
    .build()

val networkConfig = NetworkConfiguration(connection)
```

In the example above we used the `OkHttpNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

**Android (Java):**

```java
OkHttpNetworkConnection connection =
      new OkHttpNetworkConnection.OkHttpNetworkConnectionBuilder(url, getApplicationContext())
            .method(method)
            .emitTimeout(10)
            .build();

NetworkConfiguration networkConfig = new NetworkConfiguration(connection);
```

In the example above we used the `OkHttpNetworkConnection` but it can be used any custom component that implements the `NetworkConnection` interface.

***

## Persisting events with a custom EventStore

The tracker sends events asynchrounously in batches using POST requests. In case the collector is not reachable, the events are stored in an internal component called `EventStore` based on a SQLite database. The `EventStore` can be overriden with a custom one in case the developer require a different solution to persist the events before sending.

**iOS:**

```swift
let eventStore = CustomEventStore(namespace: trackerNamespace);
let emitterConfig = EmitterConfiguration()
      .eventStore(eventStore)
```

**Android (Kotlin):**

```kotlin
val eventStore: EventStore = CustomEventStore(applicationContext, trackerNamespace)

val emitterConfiguration = EmitterConfiguration()
    .eventStore(eventStore)
```

**Android (Java):**

```java
EventStore eventStore = new CustomEventStore(getApplicationContext(), trackerNamespace);

EmitterConfiguration emitterConfiguration = new EmitterConfiguration()
      .eventStore(eventStore);
```

***

In the example above the `CustomEventStore` is your implementation of the `EventStore` interface.

## What happens if an event fails to send?

The tracker has an option for setting response codes not to retry after. The intended use is for codes such as `400 Bad Request`, `401 Unauthorised`, `403 Forbidden`, `410 Gone`, `422 Unprocessable Entity`. When received in response from the Collector, the tracker doesn't retry to send events. Requests with all other 3xx, 4xx, and 5xx status codes are retried. The set of status codes for which events should be retried or not is customizable in `EmitterConfiguration`.

By default, the tracker retries sending events on all 3xx, 4xx, and 5xx status codes except the status codes indicated above. You may override the default behavior using the `customRetryForStatusCodes`. Please note that not retrying sending events to the Collector means that the events will be dropped when they fail to be sent. The `customRetryForStatusCodes` needs a dictionary that maps integers (status codes) to booleans (true for retry and false for not retry).

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()
    .customRetryForStatusCodes({403, true}) // retry sending events even if collector returns 403 status
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()
    .customRetryForStatusCodes(mapOf(
        403 to true
    ))
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()
      .customRetryForStatusCodes(new HashMap<>() {{ put(403, true); }});
```

***

Starting from version 5.5.0, you can also completely disable retrying failed requests to the collector using the `EmitterConfiguration.retryFailedRequests` configuration option. If configured, events that fail to be sent in the first request to the collector will be dropped. This may be useful in situations where it's necessary to prevent traffic spikes with many events being sent at the same time.

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()
    .retryFailedRequests(false) // don't retry any failed requests to the collector (defaults to true)
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()
    .retryFailedRequests(false) // don't retry any failed requests to the collector (defaults to true)
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()
      .retryFailedRequests(false); // don't retry any failed requests to the collector (defaults to true)
```

***

## Configuring how many events to send in one request

There are two options in the `EmitterConfiguration` that are relevant for configuring how many events are sent per request to the collector:

1. `bufferOption` – How many events to wait for before making a request. Defaults to 1.
2. `emitRange` – The maximum amount of events to send in a request. Defaults to 25.

The `bufferOption` tells the tracker how many events have to accumulate in the event store before it should make a request to the collector. There are three options: 1 (`BufferOption.single`), 10 (`BufferOption.smallGroup`), or 25 (`BufferOption.largeGroup`) events. Choosing `BufferOption.smallGroup` means that 10 events need to be tracked before the first request to the collector is made.

With very high event volumes, or when many events are buffered in the event store (e.g. if the network had been down), the `emitRange` settings come into play. This setting specifies the maximum number of events that can be sent in one request to the collector. For instance, let's say that 100 events accumulate in the event store. In case the default emit range (25) is used, the tracker will make 4 requests serially, one after the other, each with 25 events.

If the event store is empty when the tracker tries to send events - because another thread has just sent them - the thread sleeps for 5 seconds before trying again. If this happens 5 times in a row in the same thread, event sending will be paused for the whole tracker. It is restarted when a new event arrives.

Configure the batch size and emit range like this:

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()
      .bufferOption(BufferOption.single)
      .emitRange(25)
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()
    .bufferOption(BufferOption.Single)
    .emitRange(25)
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()
      .bufferOption(BufferOption.Single)
      .emitRange(25);
```

***

> **Note:** Before version 6 of the iOS and Android tracker, the `bufferOption` and `emitRange` had a slightly different meaning. Events were sent right after they were tracked regardless of the `bufferOption` used. The `bufferOption` was used to specify the maximum number of events per request and the `emitRange` specified the maximum number of total events to make in parallel requests at once. For instance, if there were 100 events in the event store and `bufferOption` was set to 10 (called `defaultGroup` previously) and `emitRange` to 50, the tracker would make 5 parallel requests to the collector with 10 events each. After that it would make another 10 parallel requests to the collector with 10 events each.

## Automatic clean up of the event store

> **Note:** This feature was introduced in version 6.0.0 of the iOS and Android trackers.

Under some situations events may keep accumulating in the event store without the tracker being able to send them at all. For instance, this may happen in case the user has an ad blocker installed or in case they permanently use the app offline.

If events accumulated in the event store without any limits, the size of the event store would keep getting bigger, potentially causing performance issues or app crashes. To prevent this, the tracker automatically removes old events from the event store. It removes old events based on two criteria that are configurable using `EmitterConfiguration`:

1. Maximum event store size (`maxEventStoreSize`) – in case the number of events surpasses this threshold, the oldest events will be removed until the number of events is under the threshold. Defaults to 1000.
2. Maximum event age (`maxEventStoreAge`) – events older than this threshold are removed. Defaults to 30 days.

The clean up is triggered before each emit attempt – before sending events to the collector.

You can configure the properties as follows.

**iOS:**

```swift
let emitterConfig = EmitterConfiguration()
      .maxEventStoreSize(1000) // events
      .maxEventStoreAge(TimeInterval(60 * 60 * 24 * 30)) // 30 days
```

**Android (Kotlin):**

```kotlin
val emitterConfiguration = EmitterConfiguration()
    .maxEventStoreSize(1000)
    .maxEventStoreAge(30.toDuration(DurationUnit.DAYS))
```

**Android (Java):**

```java
EmitterConfiguration emitterConfiguration = new EmitterConfiguration()
      .maxEventStoreSize(1000)
      .maxEventStoreAge(Duration.ofDays(30).toKotlinDuration());
```

***

---

# Global context for mobile trackers
> Apply context entities declaratively to all or specific subsets of tracked events using global context generators.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/custom-tracking-using-schemas/global-context/

As well as [adding context entities](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) to each individual event, it is possible to add context entities in a declarative way, so that they are applied to all (or a subset of) events within an application.

This can be done at tracker setup by providing a `GlobalContextConfiguration`. The logic for each global context entity is held within a `GlobalContext` generator. Multiple `GlobalContext` can be provided to the `GlobalContextConfiguration`, along with an identifying name or tag.

This example code shows the simplest kind of `GlobalContext` configuration: the same (static) self-describing JSON entity will be attached to every event tracked.

**iOS:**

```swift
// context entity to add to all events
let staticContext = SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["key": "staticExample"])

// create a GlobalContext instance with the entity as a static context
let staticGlobalContext = GlobalContext(staticContexts: [staticContext])

// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier
let globalContextsConfig = GlobalContextsConfiguration()
    .contextGenerators(["staticExampleTag": staticGlobalContext])

// pass the configuration when creating a new tracker
let tracker = Snowplow.createTracker(namespace: ..., network: ..., configurations: [..., globalContextsConfig])
```

**Android (Kotlin):**

```kotlin
// context entity to add to all events
val staticContext = SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    mapOf(
        "key" to "staticExample"
    )
)
// create a GlobalContext instance with the entity as a static context
val staticGlobalContext = GlobalContext(listOf(staticContext))

// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier
val globalContextsConfig = GlobalContextsConfiguration(
    mutableMapOf(
        "staticExampleTag" to staticGlobalContext
    )
)

// pass the configuration when creating a new tracker
Snowplow.createTracker(applicationContext, "namespace", networkConfiguration, globalContextsConfig)
```

**Android (Java):**

```java
// context entity to add to all events
SelfDescribingJson staticContext = new SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    new HashMap<String, String>() {{ put("key", "staticExample"); }}
);
// create a GlobalContext instance with the entity as a static context
GlobalContext staticGlobalContext = new GlobalContext(Arrays.asList(staticContext));

// create a GlobalContextsConfiguration and assign the GlobalContext with a unique tag identifier
GlobalContextsConfiguration globalContextsConfig = new GlobalContextsConfiguration(
    new HashMap<String, GlobalContext>() {{ put("staticExampleTag", staticGlobalContext); }}
);

// pass the configuration when creating a new tracker
Snowplow.createTracker(getApplicationContext(), "namespace", networkConfiguration, globalContextsConfig);
```

***

A generator can also be added at run-time using the method `add` like in the following example. This is possible even if no `GlobalContextConfiguration` was originally provided when the tracker was created.

**iOS:**

```swift
tracker.globalContexts?.add(tag: "staticExampleTag", contextGenerator: staticGlobalContext)
```

**Android (Kotlin):**

```kotlin
tracker.globalContexts.add("staticExampleTag", staticGlobalContext);
```

**Android (Java):**

```java
tracker.getGlobalContexts().add("staticExampleTag", staticGlobalContext);
```

***

Each context generator is associated with a tag string. The tag can be used to remove a generator at runtime using the method `remove` like in the following example.

**iOS:**

```swift
tracker.globalContexts.remove(tag: "staticExampleTag")
```

**Android (Kotlin):**

```kotlin
tracker.globalContexts.remove("staticExampleTag")
```

**Android (Java):**

```java
tracker.getGlobalContexts().remove("staticExampleTag");
```

***

It returns `nil`/`null` in case there aren’t generators registered with the specified tag, otherwise it returns the removed `GlobalContext` instance.

## Configuring the Global Context logic

An entity can be an immutable static entity, or a dynamic entity based off the event received. Also, the entity can be added either to all events, or conditionally to a subset of events by type or schema. The specific objects passed to the `GlobalContext` generator set how the global context entity will be applied, as described in the table below:

**iOS:**

|             | All events             | By event type                       | By event schema                       |
| ----------- | ---------------------- | ----------------------------------- | ------------------------------------- |
| **Static**  | `[SelfDescribingJson]` | `[SelfDescribingJson], FilterBlock` | `[SelfDescribingJson], SchemaRuleSet` |
| **Dynamic** | `GeneratorBlock`       | `GeneratorBlock, FilterBlock`       | `GeneratorBlock, SchemaRuleSet`       |

**Android (Kotlin):**

|             | All events                 | By event type                                | By event schema                           |
| ----------- | -------------------------- | -------------------------------------------- | ----------------------------------------- |
| **Static**  | `List<SelfDescribingJson>` | `List<SelfDescribingJson>, FunctionalFilter` | `List<SelfDescribingJson>, SchemaRuleSet` |
| **Dynamic** | `FunctionalGenerator`      | `FunctionalGenerator, FunctionalFilter`      | `FunctionalGenerator, SchemaRuleSet`      |

**Android (Java):**

|             | All events                 | By event type                                | By event schema                           |
| ----------- | -------------------------- | -------------------------------------------- | ----------------------------------------- |
| **Static**  | `List<SelfDescribingJson>` | `List<SelfDescribingJson>, FunctionalFilter` | `List<SelfDescribingJson>, SchemaRuleSet` |
| **Dynamic** | `FunctionalGenerator`      | `FunctionalGenerator, FunctionalFilter`      | `FunctionalGenerator, SchemaRuleSet`      |

***

If the Global Context logic becomes too complex, consider using the `ContextGenerator` interface/protocol instead.

### Static entities

This is useful in cases where the entity is static and it's always the same. A classic case is a contextual information like a user identifier that doesn't change during the tracking.

**iOS:**

```swift
let staticContext = SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["key": "staticExample"])
```

**Android (Kotlin):**

```kotlin
val staticContext = SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    mapOf(
        "key" to "staticExample"
    )
)
```

**Android (Java):**

```java
SelfDescribingJson staticContext = new SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    new HashMap<String, String>() {{ put("key", "staticExample"); }}
);
```

***

### Dynamic entities

A context generator callback (`FunctionalGenerator` on Android) returns an array of self describing JSONs, representing entities. They are evaluated each time an event is sent, and so can be used to send an entity based off event information. The `InspectableEvent` is an interface that exposes internal data of the processed event: name, schema and payload.

**iOS:**

```swift
let contextGenerator = GlobalContext(generator: {  event in
    return [
        SelfDescribingJson(schema: "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0", andData: ["eventName": event.schema!])
    ]
})
```

**Android (Kotlin):**

```kotlin
val contextGenerator = GlobalContext(object : FunctionalGenerator() {
    override fun apply(event: InspectableEvent): List<SelfDescribingJson> {
        return listOf(SelfDescribingJson(
            "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
            mapOf(
                    "eventName" to event.schema
            )
        ))
    }
})
```

**Android (Java):**

```java
GlobalContext contextGenerator = new GlobalContext(new FunctionalGenerator() {
    @Nullable
    @Override
    public List<SelfDescribingJson> apply(@NonNull InspectableEvent event) {
        return Arrays.asList(new SelfDescribingJson(
                "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
                new HashMap<String, String>() {{ put("eventName", event.getSchema()); }}
        ));
    }
});
```

***

### Conditional entities

The previous examples described the generation of entities that are associated with every event. However, there are cases where the contexts should only be applied to certain events.

#### Filter Callback

A filter callback is used to discriminate between events so we can attach global contexts only to certain events.

**iOS:**

```swift
let filteredGC = GlobalContext(
    staticContexts: [
        SelfDescribingJson(schema: "iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData: [
            "key": "value"
        ])
    ],
    filter: { event in
        // this will add the entity to all Structured events
        return "se" == event?.name
    })
tracker.add(filteredGC, tag: "tag1")
```

**Android (Kotlin):**

```kotlin
val staticContext = SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    mapOf(
        "key" to "staticExample"
    )
)
val staticGlobalContext =
    GlobalContext(listOf(staticContext), object : FunctionalFilter() {
        override fun apply(event: InspectableEvent): Boolean {
            // this will add the entity to all Structured events
            return event.name === "se"
        }
    })
```

**Android (Java):**

```java
SelfDescribingJson staticContext = new SelfDescribingJson(
        "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
        new HashMap<String, String>() {{ put("key", "staticExample"); }}
);
GlobalContext staticGlobalContext = new GlobalContext(Arrays.asList(staticContext), new FunctionalFilter() {
    @Override
    public boolean apply(@NonNull InspectableEvent event) {
        // this will add the entity to all Structured events
        return event.getName() == "se";
    }
});
```

***

#### Ruleset Provider

A ruleset provider is used when you want to attach a global context entity to certain events based on the schema URI.

A ruleset provider has a ruleset which has a list of allowed schemas and a list of denied schemas. Both lists contain Iglu URIs which can be modified based on some syntactic rules.

In this example, the ruleset provider will attach the generated entities (as described in the previous section) to events with the schema `iglu:com.acme.*/*/jsonschema/*-*-*`, but not to `iglu:com.acme.marketing/*/jsonschema/*-*-*`.

**iOS:**

```swift
let allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*"
let denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*"

let ruleset = SchemaRuleset(allowedList: [allowed], andDeniedList: [denied])
let rulesetGC = GlobalContext(staticContexts:[
  SelfDescribingJson(schema:"iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData:["key": "value"])
], ruleset:ruleset)

tracker.globalContexts?.add(tag: "tag1", contextGenerator: rulesetGC)
```

**Android (Kotlin):**

```java
val allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*"
val denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*"

val staticContext = SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    mapOf(
        "key" to "staticExample"
    )
)
val staticGlobalContext = GlobalContext(
    listOf(staticContext),
    buildRuleSet(listOf(allowed), listOf(denied))
)
```

**Android (Java):**

```java
String allowed = "iglu:com.snowplowanalytics.*/*/jsonschema/*-*-*";
String denied = "iglu:com.snowplowanalytics.mobile/*/jsonschema/*-*-*";

SelfDescribingJson staticContext = new SelfDescribingJson(
    "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
    new HashMap<String, String>() {{ put("key", "staticExample"); }}
);
GlobalContext staticGlobalContext = new GlobalContext(
    Arrays.asList(staticContext),
    SchemaRuleSet.buildRuleSet(Arrays.asList(allowed), Arrays.asList(denied))
);
```

***

##### Ruleset Format

Ruleset rules are the strings used to match against certain schemas, such as `iglu:com.acme/*/jsonschema/*-*-*`.

They follow the same five-part format as an Iglu URI `protocol:vendor/event_name/format/version` with the exception that a wildcard can be used to refer to all cases.

The parts of a rule are wildcarded with certain guidelines:

- asterisks cannot be used for the protocol (i.e. schemas always start with `iglu:`);
- version matching must be specified like so: `*–*–*`, where any part of the versioning can be defined, e.g. `1-*–*`, but only sequential parts can be wildcarded, e.g. `1-*-1` is invalid but `1-1–*` is valid;
- at least two parts: `com.acme.*` is valid, while `com.*` is not;
- vendors cannot be defined with non-wildcarded parts between wildcarded parts: `com.acme.*.marketing.*` is invalid, while `com.acme.*.*` is valid.

### ContextGenerator for custom logic

In case the logic for filter and generator callbacks are too complex, it’s possible to specify them in a class that implements the `ContextGenerator` interface/protocol.

In this case the logic for filtering and generation is encapsulated behind a context generator class.

**iOS:**

```swift
class GlobalContextGenerator : ContextGenerator {

    func filter(from event: InspectableEvent) -> Bool {
        return true
    }

    func generator(from event: InspectableEvent) -> [SelfDescribingJson]? {
        return [
            SelfDescribingJson(schema:"iglu:com.snowplowanalytics.snowplow/test_sdj/jsonschema/1-0-1", andData:["key": "value"]),
        ]
    }
}
```

It can be passed to the tracker as usual:

```swift
let contextGeneratorGC = GlobalContext(contextGenerator: GlobalContextGenerator())
tracker.globalContexts?.add(tag: "tag1", contextGenerator: contextGeneratorGC)
```

**Android (Kotlin):**

```kotlin
val staticGlobalContext = GlobalContext(object : ContextGenerator {
    override fun generateContexts(event: InspectableEvent): List<SelfDescribingJson> {
        return listOf(SelfDescribingJson(
            "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
            mapOf(
                    "eventName" to event.schema
            )
        ))
    }

    override fun filterEvent(event: InspectableEvent): Boolean {
        return true
    }
})
```

**Android (Java):**

```java
GlobalContext staticGlobalContext = new GlobalContext(new ContextGenerator() {
    @NonNull
    @Override
    public List<SelfDescribingJson> generateContexts(@NonNull InspectableEvent event) {
        return Arrays.asList(new SelfDescribingJson(
                "iglu:com.snowplowanalytics.iglu/anything-a/jsonschema/1-0-0",
                new HashMap<String, String>() {{ put("eventName", event.getSchema()); }}
        ));
    }

    @Override
    public boolean filterEvent(@NonNull InspectableEvent event) {
        return true;
    }
});
```

***

---

# Track custom events with the native mobile trackers
> Track custom self-describing events and attach context entities using JSON schemas in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach entities to any tracked event. The entities can describe the context in which the event happen or provide extra information to better describe the event.

## Tracking a custom event (SelfDescribing)

You may wish to track events in your app which are not directly supported by Snowplow and which structured event tracking does not adequately capture. Your event may have more than the five fields offered by Structured events, or its fields may not fit into the category-action-label-property-value model. The solution is Snowplow’s self-describing events. Self-describing events are a [data structure based on JSON Schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

**iOS:**

```swift
let data = ["targetUrl": "http://a-target-url.com"];
let event = SelfDescribing(schema: "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", payload: data)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val data = mapOf(
    "targetUrl" to "http://a-target-url.com"
)
val json = SelfDescribingJson(
    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",
    data
)
val event = SelfDescribing(json)
```

**Android (Java):**

```java
Map<String, String> data = new HashMap<>();
data.put("targetUrl", "http://a-target-url.com");
SelfDescribingJson json = new SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data);
SelfDescribing event = new SelfDescribing(json);

tracker.track(event);
```

***

A Self Describing event is a [self-describing JSON](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/). It has two fields:

- A `data` field, containing the properties of the event
- A `schema` field, containing the location of the JSON schema against which the `data` field should be validated.

See the full configuration and parameter options for all these classes and methods in the API docs - [Android](https://snowplow.github.io/snowplow-android-tracker/index.html) and [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/).

## Tracking a custom entity

Custom entities can be used to augment any standard Snowplow event type with additional data. Each custom context is an array of self-describing JSON following the same pattern as a self describing event. As with self describing events, if you want to create your own custom entity, you must create a JSON schema.

Note: Even if only one custom entity is being attached to an event, it still needs to be wrapped in an array.

Here are two examples of schema for custom entities. One describes a screen:

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.example",
        "name": "screen",
        "format": "jsonschema",
        "version": "1-0-1"
    },
    "type": "object",
    "properties": {
        "screenType": {
            "type": "string"
        },
        "lastUpdated": {
            "type": "string"
        }
    }
    "required": ["screenType", "lastUpdated"],
    "additionalProperties": false
}
```

and the other describes a user on that screen:

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.example",
        "name": "user",
        "format": "jsonschema",
        "version": "2-0-0"
    },
    "type": "object",
    "properties": {
        "user": {
            "type": "string"
        }
    }
    "required": ["user"],
    "additionalProperties": false
}
```

They can be used in the tracker to provide more context to specific events (e.g. ScreenView event).

**iOS:**

```swift
let event = ScreenView(name: "DemoScreenName")
event.entities.add(
    SelfDescribingJson(schema: "iglu:com.example/screen/jsonschema/1-0-1",
        andDictionary: [
             "screenType": "test",
             "lastUpdated": "2021-06-11"
        ])!)
event.entities.add(
    SelfDescribingJson(schema: "iglu:com.example/user/jsonschema/2-0-0",
        andDictionary: [
             "userType": "tester"
        ])!)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ScreenView("screen")
event.entities.add(
    SelfDescribingJson(
        "iglu:com.example/screen/jsonschema/1-2-1",
        mapOf(
            "screenType" to "test",
            "lastUpdated" to "2021-06-11"
        )
    )
)
event.entities.add(
    SelfDescribingJson(
        "iglu:com.example/user/jsonschema/2-0-0",
        mapOf(
            "userType" to "tester"
        )
    )
)

tracker.track(event)
```

**Android (Java):**

```java
ScreenView event = new ScreenView("screen");
event.getEntities().add(
    new SelfDescribingJson("iglu:com.example/screen/jsonschema/1-2-1",
        new HashMap<String, String>() {{
            put("screenType", "test");
            put("lastUpdated", "2021-06-11");
        }})
);
event.getEntities().add(
    new SelfDescribingJson("iglu:com.example/user/jsonschema/2-0-0",
        new HashMap<String, String>() {{
            put("userType", "tester");
        }})
);

tracker.track(event);
```

***

---

# Example applications for the native mobile trackers
> Explore example Android and iOS applications demonstrating implementation of the Snowplow mobile trackers.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

## Android Tracker Demos

- [Java](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-java)
- [Kotlin](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-kotlin)
- [Jetpack Compose](https://github.com/snowplow/snowplow-android-tracker/tree/master/snowplow-demo-compose)

## iOS Tracker Demos

The [examples repo](https://github.com/snowplow-industry-solutions/snowplow-ios-tracker-examples) contains examples of some of the ways you can use the iOS Tracker, including:

- A Swift SPM app
- A Swift Cocoapods app
- An ObjC Cocoapods app
- Iglu Central app in SwiftUI

---

# Track events from hybrid apps with the native mobile trackers
> Track events from both native mobile code and embedded WebViews in hybrid apps using unified session tracking.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/hybrid-apps/

> **Info:** This feature is available since v4. To use the Web plugin, you will need v6.1+ of the iOS or Android tracker.

Hybrid apps are mobile apps that in addition to a native interface, provide part of the UI through an embedded WebView. Snowplow events are tracked from both the native code (e.g., written in Swift or Kotlin) as well as the WebView (in JavaScript). Our goal is to have both events tracked from the native code as well as the WebView share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the mobile tracker.

1. Implement the Snowplow iOS or Android tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages.

   **iOS:**

   ```swift
   // Pass in a WKWebViewConfiguration
   Snowplow.subscribeToWebViewEvents(with: webView.configuration)
   ```

   **Android (Kotlin):**

   ```kotlin
   // Pass in a WebView object
   Snowplow.subscribeToWebViewEvents(webView);
   ```

   **Android (Java):**

   ```java
   // Pass in a WebView object
   Snowplow.subscribeToWebViewEvents(webView);
   ```

   ***

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the mobile tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the native mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the Snowplow iOS or Android tracker.

2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.

3. Subscribe to WebView event messages.

   **iOS:**

   ```swift
   // Pass in a WKWebViewConfiguration
   Snowplow.subscribeToWebViewEvents(with: webView.configuration)
   ```

   **Android (Kotlin):**

   ```kotlin
   // Pass in a WebView object
   Snowplow.subscribeToWebViewEvents(webView);
   ```

   **Android (Java):**

   ```java
   // Pass in a WebView object
   Snowplow.subscribeToWebViewEvents(webView);
   ```

   ***

4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# Native mobile tracker SDKs
> Use Snowplow's native mobile trackers to collect behavioral data from Android, iOS, macOS, tvOS, and watchOS applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/

Use our Snowplow [Android](https://github.com/snowplow/snowplow-android-tracker) and [iOS](https://github.com/snowplow/snowplow-ios-tracker) trackers to collect data from Android, iOS, macOS, tvOS and watchOS apps.

---

# Install and initialize the native mobile trackers
> Install the Snowplow mobile tracker SDK for iOS or Android and configure it to track events from your mobile application.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/installation-and-set-up/

**iOS:**

The Snowplow iOS Tracker SDK supports iOS 11.0+, macOS 10.13+, tvOS 12.0+, watchOS 6.0+, and visionOS 1.0+. It can be used both in Swift as well as in Objective-C code.

## Installing

The iOS Tracker SDK can be installed using various dependency managers.

**Swift Package Manager** (Recommended)

To install Snowplow Tracker with SPM:

1. In Xcode, select File > Swift Packages > Add Package Dependency.
2. Add the url where to download the library: <https://github.com/snowplow/snowplow-ios-tracker>

**Cocoapods**

To install Snowplow Tracker with Cocoapods:

1. Make sure that Cocoapods is installed on your system and correctly configured for your app.

2. Add the iOS Tracker SDK among the dependencies of your `Podfile`:

   ```ruby
   pod 'SnowplowTracker', '~> 6.0'
   ```

3. Run the command `pod install` to add the tracker to your app project.

> **Note:** Support for installing the tracker using Carthage was dropped in version 5 of the tracker.

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your application delegate `AppDelegate.swift` add `import SnowplowTracker`.

2. In the `application(_:didFinishLaunchingWithOptions:)` method, set up the SDK as follows:

   ```swift
   let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

3. It creates a tracker instance which can be used to track events like this:

   ```swift
   let event = ScreenView(name: "screen_name")
   tracker.track(event)
   ```

   If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

   ```swift
   Snowplow.defaultTracker()?.track(event)
   ```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [API docs](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/trackerconfiguration/) for more details.

```swift
Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com") {
  TrackerConfiguration()
      .appId("appId")
      .devicePlatform(.mobile)
      .base64Encoding(true)
      .logLevel(.off)
      .loggerDelegate(nil)
      .applicationContext(true)
      .platformContext(true)
      .platformContextProperties(nil)
      .platformContextRetriever(nil)
      .geoLocationContext(false)
      .sessionContext(true)
      .deepLinkContext(true)
      .screenContext(true)
      .screenViewAutotracking(true)
      .screenEngagementAutotracking(true)
      .lifecycleAutotracking(true)
      .installAutotracking(true)
      .exceptionAutotracking(true)
      .diagnosticAutotracking(false)
      .userAnonymisation(false)
      .immersiveSpaceContext(true)
      .advertisingIdentifierRetriever(nil)
  SessionConfiguration(
      foregroundTimeout: Measurement(value: 30, unit: .minutes),
      backgroundTimeout: Measurement(value: 30, unit: .minutes)
  )
      .continueSessionOnRestart(false)
}
```

The `createTracker` method allows the creation of multiple trackers in the same app. The `namespace` field lets you distinguish events sent by a specific tracker instance. It is mandatory even when the app uses just a single tracker instance like in the example above.

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Examples Github repository](https://github.com/snowplow-industry-solutions/snowplow-ios-tracker-examples) includes demo apps for Swift and Objective-C covering the most popular dependencies managers. They are provided as simple reference apps to help you set up the tracker.

**Android (Kotlin):**

The Android Tracker SDK supports Android 5 (**API level 21+**).

## Installing

The Android Tracker SDK can be installed using Gradle.

**Gradle**

Add into your `build.gradle` file:

```gradle
dependencies {
  ...
  implementation 'com.snowplowanalytics:snowplow-android-tracker:6.+'
  ...
}
```

No other dependencies are required to track events. However, some **optional** dependencies can be added:

- `InstallReferrer` to enable the referrer context entity of the [`ApplicationInstall` event](/docs/sources/mobile-trackers/tracking-events/installation-tracking/).
- `Play Services` dependencies for [tracking the app set ID and AAID](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/).

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your `Application` subclass, set up the SDK as follows:

   ```kotlin
   val tracker = Snowplow.createTracker(
       applicationContext, // Android context (LocalContext.current in Compose apps)
       "appTracker", // namespace
       "https://snowplow-collector-url.com" // Event collector URL
   )
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

2. It creates a tracker instance which can be used to track events like this:

   ```kotlin
   val event = ScreenView("screen_name")
   tracker.track(event)
   ```

If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

```kotlin
Snowplow.defaultTracker?.track(event)
```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [Android API docs](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/index.html) for more details.

```kotlin
val networkConfig = NetworkConfiguration(
    "https://snowplow-collector-url.com",
    HttpMethod.POST
)
val trackerConfig = TrackerConfiguration("appId")
    .devicePlatform(DevicePlatform.Mobile)
    .base64encoding(true)
    .logLevel(LogLevel.OFF)
    .loggerDelegate(null)
    .applicationContext(true)
    .platformContext(true)
    .geoLocationContext(false)
    .sessionContext(true)
    .deepLinkContext(true)
    .screenContext(true)
    .screenViewAutotracking(true)
    .screenEngagementAutotracking(true)
    .lifecycleAutotracking(true)
    .installAutotracking(true)
    .exceptionAutotracking(true)
    .diagnosticAutotracking(false)
    .userAnonymisation(false)
    .platformContextProperties(null)
    .platformContextRetriever(null)
val sessionConfig = SessionConfiguration(
    TimeMeasure(30, TimeUnit.SECONDS),
    TimeMeasure(30, TimeUnit.SECONDS)
)
    .continueSessionOnRestart(false)
createTracker(
    applicationContext,
    "appTracker",
    networkConfig,
    trackerConfig,
    sessionConfig
)
```

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Android tracker Github repository](https://github.com/snowplow/snowplow-android-tracker) includes demo apps in Java, Kotlin, and Kotlin with Jetpack Compose. They are provided as simple reference apps to help you set up the tracker.

## Using the tracker with R8 optimization

Depending on your app configuration, you may need to add ProGuard rules to prevent the R8 compiler removing code needed for tracker function. Fetching certain [platform context](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/) properties - AAID and app set ID - uses reflection. To include the necessary classes, add the following rules to the app's `proguard-rules.pro` file.

```text
# Reflection for the appSetId
-keep class com.google.android.gms.appset.AppSet { *; }
-keep class com.google.android.gms.appset.AppSetIdInfo { *; }
-keep class com.google.android.gms.internal.appset.zzr { *; }
-keep class com.google.android.gms.tasks.Tasks { *; }

# Reflection for the AAID (AndroidIdfa)
-keep class com.google.android.gms.ads.identifier.** { *; }
```

**Android (Java):**

The Android Tracker SDK supports Android 5 (**API level 21+**).

## Installing

The Android Tracker SDK can be installed using Gradle.

**Gradle**

Add into your `build.gradle` file:

```gradle
dependencies {
  ...
  implementation 'com.snowplowanalytics:snowplow-android-tracker:6.+'
  ...
}
```

No other dependencies are required to track events. However, some **optional** dependencies can be added:

- `InstallReferrer` to enable the referrer context entity of the [`ApplicationInstall` event](/docs/sources/mobile-trackers/tracking-events/installation-tracking/).
- `Play Services` dependencies for [tracking the app set ID and AAID](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/).

## Setting up

Once the tracker SDK is correctly set as a dependency in your app project you have to instrument the tracker:

1. In your `Application` subclass, set up the SDK as follows:

   ```java
   TrackerController tracker = Snowplow.createTracker(
         getApplicationContext(), // Android context
         "appTracker", // namespace
         "https://snowplow-collector-url.com" // Event collector URL
   );
   ```

   The URL path for your collector endpoint should include the protocol, "http" or "https". If not included in the URL, "https" connection will be used by default.

2. It creates a tracker instance which can be used to track events like this:

   ```java
   Event event = new ScreenView("screen_name");
   tracker.track(event);
   ```

If you prefer to access the tracker when the reference is not directly accessible, you can use the `defaultTracker` :

```java
Snowplow.getDefaultTracker().track(event);
```

You can override the default configuration with a fine grained configuration when you create the tracker. See the [Android API docs](https://snowplow.github.io/snowplow-android-tracker/snowplow-android-tracker/com.snowplowanalytics.snowplow.configuration/index.html) for the `Configuration` classes to see all the options and defaults.

```java
NetworkConfiguration networkConfig = new NetworkConfiguration(
    "https://snowplow-collector-url.com",
    HttpMethod.POST
);
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .base64encoding(false)
    .sessionContext(true)
    .platformContext(true)
    .lifecycleAutotracking(true)
    .screenViewAutotracking(true)
    .screenContext(true)
    .applicationContext(true)
    .exceptionAutotracking(true)
    .installAutotracking(true)
    .userAnonymisation(false)
    .logLevel(LogLevel.OFF);
SessionConfiguration sessionConfig = new SessionConfiguration(
    new TimeMeasure(30, TimeUnit.SECONDS),
    new TimeMeasure(30, TimeUnit.SECONDS)
)
    .continueSessionOnRestart(false);
Snowplow.createTracker(
    getApplicationContext(),
    "appTracker",
    networkConfig,
    trackerConfig,
    sessionConfig
);
```

> **Note:** The trackers created with the above method are configured "locally" only. To create a tracker where the configuration can be updated through downloaded files, read [this page](/docs/sources/mobile-trackers/remote-configuration/) about remote configuration.

The [Android tracker Github repository](https://github.com/snowplow/snowplow-android-tracker) includes demo apps in Java, Kotlin, and Kotlin with Jetpack Compose. They are provided as simple reference apps to help you set up the tracker.

## Using the tracker with R8 optimization

Depending on your app configuration, you may need to add ProGuard rules to prevent the R8 compiler removing code needed for tracker function. Fetching certain [platform context](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/) properties - AAID and app set ID - uses reflection. To include the necessary classes, add the following rules to the app's `proguard-rules.pro` file.

```text
# Reflection for the appSetId
-keep class com.google.android.gms.appset.AppSet { *; }
-keep class com.google.android.gms.appset.AppSetIdInfo { *; }
-keep class com.google.android.gms.internal.appset.zzr { *; }
-keep class com.google.android.gms.tasks.Tasks { *; }

# Reflection for the AAID (AndroidIdfa)
-keep class com.google.android.gms.ads.identifier.** { *; }
```

***

---

# Migration guides for the native mobile trackers
> Upgrade to newer versions of the iOS and Android mobile trackers using version-specific migration guides.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/

This section contains information to help you upgrade to newer versions of the mobile trackers.

---

# Migration guide for Snowplow Android Tracker SDK from version 1.x to 2.0
> Migrate your Android tracker instrumentation from version 1.x to 2.0 with updated configuration and API changes.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-android-tracker-sdk-from-version-1-x-to-2-0/

This section describes the main changes when converting an applications instrumentation of the Snowplow Android tracker SDK from version 1.x to version 2.0.

## The new approach

The main objective of this new release is a revision of the public API. Currently, the tracker configuration requires the creation of the Tracker and Emitter components. Optionally, the Subject component can be configured as well. Then both need to be passed to the Tracker component. The version 2.0 simplifies this process, letting the developer to specify the configuration with Configuration objects and passing them to a `createTracker` method able to return the tracker ready to use, without any other extra steps.

### Setup of the tracker

The entry point to setup the tracker is now the `Snowplow` static class and the `createTracker` methods. They need a namespace string which is now mandatory. The namespace string uniquely identifies the tracker in the app. It's important to note that all the events not sent to the collector but stored in the tracker are attached to the namespace string. This means that if a new configuration uses a different namespace, all those unsent event will still stored in the tracker, in a sort of zombie state, with no way to send them to any collector. To send them you need to reuse the original namespace string or delete them.

Fine tuning of the tracker is now possible with Configuration classes.

These are the classes now available:

- `NetworkConfiguration`: to configure network connection with the Snowplow collector.
- `TrackerConfiguration`: to configure contexts and automatic events of the tracker, and general behavior.
- `SessionConfiguration`: to configure session behavior.
- `EmitterConfiguration`: to fine tune about how the tracker sends events to the collector.
- `SubjectConfiguration`: to specify details to send with events about the user and the platform.
- `GdprConfiguration`: to configure the GDPR context.
- `GlobalContextsConfiguration`: to configure the GlobalContexts feature to dynamically send created contexts with some selected events.

They replace many of the settings previously set on the builders for the Tracker, Subject and Emitter components.

An example configuration:

```java
TrackerController initAndroidTracker(Context context, String trackerNamespace) {
        NetworkConfiguration networkConfiguration = new NetworkConfiguration("https://snowplow-collector-url.com");
        TrackerConfiguration trackerConfiguration = new TrackerConfiguration("example-of-app-id")
                .sessionContext(true)
                .platformContext(true)
                .applicationContext(true)
                .geoLocationContext(true)
                .lifecycleAutotracking(true)
                .screenViewAutotracking(true)
                .screenContext(true)
                .exceptionAutotracking(true)
                .installAutotracking(true)
                .diagnosticAutotracking(true);
        SessionConfiguration sessionConfiguration = new SessionConfiguration(
                new TimeMeasure(60, TimeUnit.SECONDS),
                new TimeMeasure(30, TimeUnit.SECONDS)
        );
        GdprConfiguration gdprConfiguration = new GdprConfiguration(
                Basis.CONSENT,
                "someId",
                "0.1.0",
                "this is a demo document description"
        );

        return Snowplow.createTracker(context,
                trackerNamespace,
                networkConfiguration,
                trackerConfiguration,
                sessionConfiguration,
                gdprConfiguration
        );
    }
```

The tracker can be controlled by the `TrackerController` (the `Tracker` class is now deprecated). Through the `TrackerController` it will be possible to update settings at runtime and get information about the tracking state. From the `TrackerController` it's also possible to access all the other controllers. This allows you to access and update settings about communication with the collector, emission of the events, session of the tracker (if enabled) and so on. In general, there is a specific controller for each area covered by a configuration. All of them are accessible through the `TrackerController`.

**Note:** Please, don't retain a reference to the sub-controllers. If you invalidate or recreate the tracker, all the sub-controller instances will no longer be active.

## Low level migration from 1.x

- Minimum supported version is API 21 (Android 5).

- Migrated from Android Support Library to AndroidX.

- Minimum OkHttpClient dependency is 4.9.1.

- `lifecycleAutotracking` (background and foreground events and indexes on sessions) are optional and off by default.

- `lifecycleAutotracking` requires `androidx.lifecycle:lifecycle-extensions` among the dependencies of the app.

- `RequestSecurity` class has been renamed `Protocol`.

- `mobileContext` property has been renamed `platformContext`.

- `DevicePlatforms` has been renamed `DevicePlatform`.

- `ReadyRequest` has been replaced by `Request` (not needed in the new API).

- `namespace` on Tracker is now mandatory for the correct operativity.

- `name` field on `ScreenView` is now mandatory.

- `EcommerceItem`s aren't forced to have the same timestamp of the `EcommerceTransaction` event.

- `SPEcommerceItem` constructor doesn't require `itemId` as it's added by the tracker when the e-commerce event is sent.

- Deprecated classes of the previous API (Tracker, Emitter, Subject, ...) are now accessible from the `internal` sub-package.

- The `internal` package is considered private API so it doesn't follow the restrictions of the [Semantic Version](https://semver.org/) policy. Although we will keep the legacy deprecated API with minimum changes until the release of the version 3.0.

- Global Contexts implementation is changed and uniformed to the one on iOS tracker.

- **For the versions 2.0 and 2.1**, the application install events report the timestamp of when installation happened in the `true_timestamp` rather than the `device_timestamp`. It caused an [issue](https://github.com/snowplow/snowplow-android-tracker/issues/462) with the `derived_timestamp`. For this reason, since the version 2.2, the application install events report the timestamp of then installation happened in the `device_timestamp` like the previous 1.x versions.

## Migrate to new Global Contexts API

- The `ContextGenerator` no longer owns the `tag` string used to identify the Global Context generator in the tracker. The `ContextGenerator` replacement is simply called `GlobalContext` which is the new generator class for the Global Contexts feature.
- The generator `GlobalContext` is added to the tracker and associated to the specified tag through the method `add` of `GlobalContextsConfiguration`.
- The classes `FilterProvider` and `RulesetProvider` have been removed. `GlobalContext` class and `ContextGenerator` protocol can be used to filter events through specific event fields or rulesets.

---

# Migration guide for Snowplow Android Tracker SDK from version 2.x to 3.0
> Migrate your Android tracker from version 2.x to 3.0 which removes the deprecated v1 API.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-android-tracker-sdk-from-version-2-x-to-3-0/

This new release doesn't introduce breaking changes with the previous 2.x version. The version 2.0 can be considered a transitory release because two different APIs were available for the developer: v1 API (deprecated) and v2 API (the new one). The version 3.0 doesn't introduce changes to the v2 API but it completely remove the v1 API. Hence, this version introduces breking changes only for the developer that have instrumented the v2.0 using the old deprecated v1 API.

In this release we've marked as deprecated:

- `PageView` event: this event has been designed for web tracking. Its role can be replaced on mobile with two different events: `ScreenView` event and `DeepLinkReceived` event. The first covers the need of tracking the screen visualized to the user by the app. The latter covers the case where the shown content is referred to an external url referral and we want to track that down, exactly like the PageView does in the web tracker.

In this release there are some breaking changes:

- The v1 API has been removed from this new version of the trackers, which means the v1 components (Tracker, Emitter, Subject, Session) used to set up the tracker are no longer available. If you have already migrated your instrumentation to the v2 API there aren't breaking changes.

- The events can be built using the constructor and builder methods. The builder classes available with v1 are no longer available. If you already build events using events' constructor suggested with the v2 API, there aren't breaking changes.

- Session callbacks have been removed. They will be reintroduced soon in one of the next minor versions. More details will be provided at the release.

- Utilities or Utils methods, available in v1 API, have been removed.

---

# Migration guide for Snowplow iOS Tracker SDK from version 1.x to 2.0
> Migrate your iOS tracker instrumentation from version 1.x to 2.0 with updated configuration and improved Swift compatibility.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-ios-tracker-sdk-from-version-1-x-to-2-0/

This section describes the main changes when converting an applications instrumentation of the Snowplow iOS tracker SDK from version 1.x to version 2.0.

## The new approach

The main objective of this new release is a revision of the public API. Currently, the tracker configuration requires the creation of the Tracker and Emitter components. Optionally, the Subject component can be configured as well. Then both need to be passed to the Tracker component. The version 2.0 simplifies this process, letting the developer to specify the configuration with Configuration objects and passing them to a `createTracker` method able to return the tracker ready to use, without any other extra steps.

The tracker has been also updated improving the compatibility with Swift. Now, the classes of the public API don't need the `SP` prefix required on Objective-C.

### Setup of the tracker

The entry point to setup the tracker is now the `Snowplow` class and the `createTracker` methods. They need a namespace string which is now mandatory. The namespace string uniquely identifies the tracker in the app. It's important to note that all the events not sent to the collector but stored in the tracker are attached to the namespace string. This means that if a new configuration uses a different namespace, all those unsent event will still stored in the tracker, in a sort of zombie state, with no way to send them to any collector. To send them you need to reuse the original namespace string or delete them.

Fine tuning of the tracker is now possible with Configuration classes.

These are the classes now available:

- `NetworkConfiguration`: to configure network connection with the Snowplow collector.
- `TrackerConfiguration`: to configure contexts and automatic events of the tracker, and general behavior.
- `SessionConfiguration`: to configure session behavior.
- `EmitterConfiguration`: to fine tune about how the tracker sends events to the collector.
- `SubjectConfiguration`: to specify details to send with events about the user and the platform.
- `GdprConfiguration`: to configure the GDPR context.
- `GlobalContextsConfiguration`: to configure the GlobalContexts feature to dynamically send created contexts with some selected events.

They replace many of the settings previously set on the builders for the Tracker, Subject and Emitter components.

An example configuration:

```swift
func initTracker(trackerNamespace: String) -> TrackerController {
        let networkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com")
        let trackerConfig = TrackerConfiguration()
            .sessionContext(true)
            .platformContext(true)
            .applicationContext(true)
            .lifecycleAutotracking(true)
            .screenViewAutotracking(true)
            .screenContext(true)
            .exceptionAutotracking(true)
            .installAutotracking(true)
            .diagnosticAutotracking(true)
        let sessionConfig = SessionConfiguration(
            foregroundTimeout: Measurement(value: 60, unit: .seconds),
            backgroundTimeout: Measurement(value: 30, unit: .seconds)
        )
        let gdprConfig = GDPRConfiguration(
            basis: .consent,
            documentId: "id",
            documentVersion: "1.0",
            documentDescription: "description"
        )

        return Snowplow.createTracker(
            namespace: trackerNamespace,
            network: networkConfig,
            configurations: [trackerConfig, emitterConfig, sessionConfig, gdprConfig]);
}
```

The tracker can be controlled by the `TrackerController` (the `Tracker` class is now deprecated). Through the `TrackerController` it will be possible to update settings at runtime and get information about the tracking state. From the `TrackerController` it's also possible to access all the other controllers. This allows you to access and update settings about communication with the collector, emission of the events, session of the tracker (if enabled) and so on. In general, there is a specific controller for each area covered by a configuration. All of them are accessible through the `TrackerController`.

**Note:** Please, don't retain a reference to the sub-controllers. If you invalidate or recreate the tracker, all the sub-controller instances will no longer be active.

## Low level migration from 1.x

- iOS minimum supported version is bumped to iOS 9.0.

- `lifecycleAutotracking` (background and foreground events and indexes on sessions) are optional and off by default.

- Enum `SPProtocol` items have been renamed to `SPProtocolHttp` and `SPProtocolHttps`.

- Enum `SPRequestOptions` has been renamed `SPHttpMethod` and its items are `SPHttpMethodGet` and `SPHttpMethodPost`.

- `Snowplow` class is now the entry point to create trackers. The old `Snowplow` class containing the constants has been renamed `SPTrackerConstants`.

- `SPPrimitive` and `SPSelfDescribing` have been renamed as `SPPrimitiveAbstract` and `SPSelfDescribingAbstract`.

- `SPUnstructured` class used for self-describing events has been renamed `SPSelfDescribing`.

- The property `trueTimestamp` changed the type from `NSNumber` to `NSDate`.

- `namespace` on Tracker is now mandatory for the correct operativity.

- `name` field on `ScreenView` is now mandatory.

- `EcommerceItem`s aren't forced to have the same timestamp of the `EcommerceTransaction` event.

- `SPEcommerceItem` constructor doesn't require `itemId` as it's added by the tracker when the e-commerce event is sent.

- Deprecated classes of the previous API (SPTracker, SPEmitter, SPSubject, ...).

- **For the versions 2.0 and 2.1**, the application install events report the timestamp of when installation happened in the `true_timestamp` rather than the `device_timestamp`. It caused an [issue](https://github.com/snowplow/snowplow-ios-tracker/issues/625) with the `derived_timestamp`. For this reason, since the version 2.2, the application install events report the timestamp of then installation happened in the `device_timestamp` like the previous 1.x versions.

---

# Migration guide for Snowplow iOS Tracker SDK from version 2.x to 3.0
> Migrate your iOS tracker from version 2.x to 3.0 which removes the deprecated v1 API.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-for-snowplow-ios-tracker-sdk-from-version-2-x-to-3-0/

This new release doesn't introduce breaking changes with the previous 2.x version. The version 2.0 can be considered a transitory release because two different APIs were available for the developer: v1 API (deprecated) and v2 API (the new one). The version 3.0 doesn't introduce changes to the v2 API but it completely remove the v1 API. Hence, this version introduces breking changes only for the developer that have instrumented the v2.0 using the old deprecated v1 API.

In this release we've marked as deprecated:

- `PageView` event: this event has been designed for web tracking. Its role can be replaced on mobile with two different events: `ScreenView` event and `DeepLinkReceived` event. The first covers the need of tracking the screen visualized to the user by the app. The latter covers the case where the shown content is referred to an external url referral and we want to track that down, exactly like the PageView does in the web tracker.

- `PushNotification` event: this event is available only on iOS tracker and it's strongly influenced by the format of push notifications in the iOS platform. It has been replaced with `MessageNotification` which is usable in both iOS and Android trackers v3.0.

In this release there are some breaking changes:

- The v1 API has been removed from this new version of the trackers, which means the v1 components (Tracker, Emitter, Subject, Session) used to set up the tracker are no longer available. If you have already migrated your instrumentation to the v2 API there aren't breaking changes.

- The events can be built using the constructor and builder methods. The builder classes available with v1 are no longer available. If you already build events using events' constructor suggested with the v2 API, there aren't breaking changes.

- Session callbacks have been removed. They will be reintroduced soon in one of the next minor versions. More details will be provided at the release.

- Utilities or Utils methods, available in v1 API, have been removed.

---

# Migration guide from version 3.x to 4.0
> Migrate to mobile tracker version 4.0 with updated remote configuration callbacks and event ID returns.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-3-x-to-4-0/

A breaking change in this release is the updated callback for remote configuration. The callback now in addition to the list of namespaces also receives the configuration state enum as an argument. Read more about the [new API here](/docs/sources/mobile-trackers/remote-configuration/).

The tracker now also returns the event ID from the `track(event)` method in `TrackerController`.

---

# Migration guide from version 4.x to 5.0
> Migrate to mobile tracker version 5.0 with internal rewrite to Swift and Kotlin while maintaining minimal API changes.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-4-x-to-5-0/

Although the trackers underwent a huge internal rewrite (from Objective-C to Swift and from Java to Kotlin), we tried to keep the API with as little breaking changes as possible.

## iOS tracker

The supported platforms have changed on the iOS tracker:

- Minimum iOS deployment target changed from 9.0 to 11.0.
- On macOS, from 10.10 to 10.13.
- On tvOS, from 9.0 to 12.0.
- On watchOS, from 2.0 to 6.0.

Additionally, there is a new way to enable tracking the IDFA identifier – instead of adding the `SNOWPLOW_IDFA_ENABLED` compiler flag, one now needs to implement a callback (`TrackerConfiguration.advertisingIdentifierRetriever`) that retrieves the value. See the [documentation for more information](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#identifier-for-advertisers-idfaaaid).

If using `EmitterConfiguration` when creating a new tracker, the default buffer option configuration changed from `single` (max 1 event per batch) to `default` (max 10 events per batch). If not using an `EmitterConfiguration`, the buffer option was set to `default` also in tracker v4.

## Android tracker

We adopted using Kotlin properties instead of Java fields for public properties of the event classes. This doesn't change the API in Kotlin. On the other hand, in Java, the properties are now accessible through getter and setter methods instead of directly as Java fields.

For example, to set the true timestamp of events in Java the API changes as follows:

```java
// v4 API:
event.trueTimestamp = 123456789L; // doesn't work anymore
// v5 API:
event.setTrueTimestamp(123456789L);
```

However, the API hasn't changed if you use the builder methods to set the properties. For example, this approach works in both the v4 and v5 tracker:

```java
// works both in v4 and v5 API:
event.trueTimestamp(123456789L);
```

If using `EmitterConfiguration` when creating a new tracker, the default buffer option configuration changed from `single` (max 1 event per batch) to `default` (max 10 events per batch). If not using an `EmitterConfiguration`, the buffer option was set to `default` also in tracker v4.

## Renaming contexts to entities

The `contexts` property in events used to assign custom context entities to events has been renamed to `entities`. The previous naming is still available but it is deprecated.

In the Android tracker, the `customContexts` property in events has been deprecated in favor of the `entities` property.

---

# Migration guide from version 5.x to 6.0
> Migrate to mobile tracker version 6.0 with changes to lifecycle tracking, event batching, and screen engagement features.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-from-version-5-x-to-6-0/

There were a few breaking changes within v6, for both iOS and Android. These are described here, with a quick summary of the rest of the changelog at the end.

## Changes to events

These changes probably won't break your app code, but they will affect the events generated.

### Lifecycle autotracking

Affects: iOS ✅ Android ✅

[Lifecycle autotracking](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/) is now on by default, for both trackers. This is because it's a prerequisite for the new [screen engagement tracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking), which is also on by default.

### Removed properties from platform context entity

Affects: iOS ✅ Android ❌

To comply with Apple's Privacy Manifest rules, we have removed the automatic tracking of `totalStorage` and `availableStorage` metrics from the [platform context entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/). We've also added a Privacy Manifest for the SDK.

To track `totalStorage` or `availableStorage`, use the new `PlatformContextRetriever` callbacks class to configure the platform context entity.

### Preventing unnecessary ScreenView event

Affects: iOS ❌ Android ✅

Previously a screen view event was tracked again by the [screen view autotracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) feature when the app moved to foreground. This is not expected because the screen doesn't change when the app is in background, and it is not consistent with how the screen view autotracking works on iOS. The extra event will no longer be tracked.

### Event entities API

Affects: iOS ✅ Android ❌

To standardise the behavior between trackers, we've changed how the `entities()` method of all events works on iOS. Previously, calling `event.entities(newListEntities)` replaced all the context entities currently attached to the event. Now, the new entities are appended instead.

There's no change to the behavior of the variable `entities`, so you could still replace them all using `event.entities = newListEntities`.

## Event batching

Affects: iOS ✅ Android ✅

In both trackers, we have changed how the `EmitterConfiguration` options `bufferOption` and `emitRange` are used, as well as changing the defaults. Read more about that [here](/docs/sources/mobile-trackers/configuring-how-events-are-sent/#configuring-how-many-events-to-send-in-one-request). The `BufferOption.defaultGroup` has been renamed to `BufferOption.SmallGroup`.

Network requests are now made serially. If you are using a custom `EmitterConfiguration.emitRange`, you may wish to set it to a lower value. The new default `emitRange` is 25 (down from 150).

## Non-optional tracker

Affects: iOS ✅ Android ❌

In the v5.x of the iOS tracker, `createTracker` returned an optional `TrackerController?`. This was an oversight. In v6, the iOS tracker again returns a non-optional `TrackerController`.

## `Tracker.track()` return type

Affects: iOS ✅ Android ❌

The return type of the `Tracker.track()` method has changed from `UUID?` to `UUID` on iOS. Previously, `nil` could be returned if the tracker was paused, or if the event was filtered out and not actually tracked. From v6 onwards it will always return a UUID. If the event is tracked, this will be the `eventId`.

This change was necessary as part of the comprehensive refactoring of the tracker thread model. The iOS tracker now uses a global dispatch queue. This single queue makes the tracker much safer. Almost all actions are now performed concurrently. Network requests in Emitter are still asynchronous.

## Changes to the `EventStore` interface

Affects: iOS ✅ Android ✅

A new method, `removeOldEvents()` has been added to the `EventStore` protocol on both trackers. This method is used in the new feature that deletes events from storage if they get too old (by default, 30 days). Also, if too many events collect in the `EventStore`, the older ones will be deleted (default 1000 events).

For the Android tracker, we have updated the `EventStore` interface to remove the optional types.

## Other changes

See the full changelog on Github, for [iOS](https://github.com/snowplow/snowplow-ios-tracker/releases/tag/6.0.0) and [Android](https://github.com/snowplow/snowplow-android-tracker/releases/tag/6.0.0).

### New events

The [screen engagement](/docs/sources/mobile-trackers/tracking-events/screen-tracking/#screen-engagement-tracking) feature adds new events for both iOS and Android. For iOS only, there are new events (and a new demo app) for [visionOS](/docs/sources/mobile-trackers/tracking-events/visionos/). For Android, the `PageView` event has been restored, after accidental deprecation in v5.

### Cross-navigation tracking

Decorate URIs with [user and session information](/docs/sources/mobile-trackers/tracking-events/session-tracking/#decorating-outgoing-links-using-cross-navigation-tracking) in both trackers using the new `CrossDeviceParameterConfiguration`. This is the equivalent of cross-domain tracking for web.

### Access to `EventStore` in iOS tracker

The `EventStore` is now exposed as part of the `EmitterController`, allowing access for e.g. deleting all stored events like this `tracker?.emitter?.eventStore?.removeAllEvents()`. This was already possible on Android.

### Codable structs in the iOS tracker

The custom event and entity classes `SelfDescribing` and `SelfDescribingJson` now accept data represented using `Encodable` structs. This alllows you to define the data using typed structs, and track that directly instead of using untyped dictionaries.

### Provide custom values to platform context entity

As mentioned above, the new `PlatformContextRetriever` callbacks class allows you to override any [platform entity](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/#overriding-platform-context-properties) properties. The `PlatformContextRetriever` is available for both iOS and Android.

### Emitter and network connection behavior

The Android tracker default emit timeout has been increased to 30 seconds, from 5 seconds. This setting is configured using `NetworkConfiguration`. The `timeout` configuration option has been added to the iOS tracker.

In both trackers, the internal `Emitter` constructor has been updated. The change moves the namespace and event store into the constructor, enabling removing some optionals, and makes the properties immutable and safer.

### Removed FMDB dependency for iOS

The FMDB dependency has been removed from `SQLiteEventStore`. The built-in `sqlite` methods are used instead.

---

# Migrating to the v5.4 Ecommerce package
> Migrate from deprecated ecommerce events to the new v5.4 ecommerce tracking package with structured entities and event types.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/migration-guides/migration-guide-to-new-ecommerce/

The [Ecommerce tracking package](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) was introduced in Android and iOS version 5.4.0. It provides 11 out-of-the-box event types to make it easier to thoroughly track user activity in an ecommerce store.

> **Note:** Migrating to the new ecommerce events is a breaking change for any relevant data models.

## Replacing the deprecated events

To update your existing tracking based on the now deprecated `Ecommerce` (iOS)/`EcommerceTransaction` (Android) and `EcommerceItem`/`EcommerceTransactionItem` event types, we recommend using `TransactionEvent`.

**iOS:**

Old API:

```swift
let product = EcommerceItem(sku: "sku1", price: 100, quantity: 1)
    .name("itemName")
    .category("category")
    .currency("GBP")
    .orderId("id-123")

let event = Ecommerce(orderId: "id-123", totalValue: 350, items: [product])
    .currency("GBP")
    .taxValue(10)
    .shipping(15)
    .city("London")
    .state("N/A")
    .country("UK")
    .affiliation("affiliation")

tracker.track(event)
```

New API:

```swift
let product = ProductEntity(
  id: "sku1",
  price: 100,
  quantity: 1,
  name: "itemName",
  category: "category",
  currency: "GBP"
)

let transaction = TransactionEntity(
  transactionId: "id-123",
  revenue: 350,
  products: [product],
  currency: "GBP",
  tax: 10,
  shipping: 15
)

let event = TransactionEvent(transaction: transaction, products: [product])

tracker.track(event)
```

**Android (Kotlin):**

Old API:

```kotlin
val product = EcommerceTransactionItem(sku = "sku1", price = 100.0, quantity = 1)
    .name("itemName")
    .category("category")
    .currency("GBP")
    .orderId("id-123")

val event = EcommerceTransaction(orderId = "id-123", totalValue = 350.0, items = listOf(product))
    .currency("GBP")
    .taxValue(10.0)
    .shipping(15.0)
    .city("London")
    .state("N/A")
    .country("UK")
    .affiliation("affiliation")

tracker.track(event)
```

New API:

```kotlin
val product = ProductEntity(
  id = "sku1",
  price = 100.0,
  quantity = 1,
  name = "itemName",
  category = "category",
  currency = "GBP"
)

val transaction = TransactionEntity(
  transactionId = "id-123",
  revenue = 350.0,
  products = [product],
  currency = "GBP",
  tax = 10.0,
  shipping = 15.0
)

val event = TransactionEvent(transaction, listOf(product))

tracker.track(event)
```

**Android (Java):**

Old API:

```java
EcommerceTransactionItem product = new EcommerceTransactionItem(
    "sku1", // sku
    100.0, // price
    1 // quantity
  ).name("itemName")
  .category("category")
  .currency("GBP")
  .orderId("id-123")

List<Product> products = new ArrayList<>();
products.add(product);

EcommerceTransaction event = EcommerceTransaction(
      "id-123", // orderId
      350.0, // totalValue
      products // items
  ).currency("GBP")
  .taxValue(10.0)
  .shipping(15.0)
  .city("London")
  .state("N/A")
  .country("UK")
  .affiliation("affiliation")

tracker.track(event)
```

New API:

```java
ProductEntity product = new ProductEntity(
  "sku1", // id
  100.0, // price
  1, // quantity
  "itemName", // name
  "category", // category
  "GBP", // currency
);
List<Product> products = new ArrayList<>();
products.add(product);

TransactionEntity transaction = new TransactionEntity(
  "id-123", // id
  350.0, // revenue
  products, // products
  "GBP", // currency
  10.0, // tax
  15.0 // shipping
);

TransactionEvent event = new TransactionEvent(transaction, products);

tracker.track(event)
```

***

These code snippets show only the direct API replacement. Check out the [docs page](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) for information about expanding your tracking with the other new event types.

Note that address details and affiliation are not captured in `TransactionEvent`. To track these so that they are attached to the `TransactionEvent`, we suggest creating [custom schemas](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) and adding the data as an entity. Alternatively for addresses, you could use the `CheckoutStepEvent` which has address and postcode as optional parameters.

## How the tracked events are different

### Old events

Using the old event types, an event is generated for each `EcommerceTransaction` event as well as each `EcommerceTransactionItem` attached. Therefore, tracking an `EcommerceTransaction` event with 3 items would result in 4 events being tracked. These events are not linked at all, unless the same `orderId` is provided to all the objects, adding potential challenges in data modeling.

In the tracked data, the events have `event_name` `transaction` or `transaction_item`. The data is sent within the legacy event properties `tr_x` and `ti_x`, e.g. `tr_id` for the `EcommerceTransaction` `orderId` property. All of these properties are listed [here](/docs/events/). Each one maps to a column in the warehouse table.

### New events

The new Ecommerce event types take advantage of the Snowplow entities/event context feature. Tracking `TransactionEvent` with 3 items results in a single event being tracked. The items, and the details of the transaction, are attached as entities. That single event would have one `TransactionEntity`, three `ProductEntity`, plus whichever out-of-the-box and/or custom entities are configured.

All of the new events are `SelfDescribingEvent` with `event_name` `snowplow_ecommerce_action`. None of the data is sent as legacy properties; it is all within entities, which end up in their own table columns.

Entities such as `ProductEntity` can be added to several of the new Ecommerce events, e.g. `ProductViewEvent` or `AddToCartEvent`. Reusing the same entities makes modeling easier and keeps the code cleaner.

## Using the dbt data model

Since the structure of the events is so different, migrating to the new ecommerce events is a breaking change for any relevant data models.

We provide a [dbt package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) that creates a set of derived tables including carts, transactions, and products from the raw ecommerce event data.

---

# Send user identifiers to Kantar Focal Meter
> Integrate mobile trackers with Kantar Focal Meter to send user identifiers for audience measurement.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/plugins/focal-meter/

The mobile trackers provide integration with [Focal Meter by Kantar](https://www.virtualmeter.co.uk/focalmeter). Focal Meter is a box that connects directly to the broadband router and collects viewing information for the devices on your network.

This integration enables measuring the audience of content through the Focal Meter router meter. The tracker has the ability to send the user identifier (`userId` present in the session context) to a [Kantar Focal Meter](https://www.virtualmeter.co.uk/focalmeter) endpoint.

To enable this feature, you can pass the `FocalMeterConfiguration` configuration with the URL of the Kantar endpoint. For example:

**iOS:**

```swift
let focalMeterConfig = FocalMeterConfiguration(kantarEndpoint: "https://thekantarendpoint.com")
let tracker = Snowplow.createTracker(namespace: namespace, network: networkConfig, configurations: [focalMeterConfig])
```

**Android (Kotlin):**

```kotlin
val focalMeterConfig = FocalMeterConfiguration("https://thekantarendpoint.com")
val tracker = Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig)
```

**Android (Java):**

```java
FocalMeterConfiguration focalMeterConfig = new FocalMeterConfiguration("https://thekantarendpoint.com");
Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig);
```

***

Once configured, the tracker will make a request to the endpoint whenever the user identifier changes or each time the app starts.

> **Note:** Session context needs to be enabled (default) in order for the integration to work.

> **Note:** The feature is available from version 5.6.0 of the iOS and Android trackers.

## Processing the user ID sent in requests to Kantar

By default, the plugin sends the user ID as a GET parameter in requests to Kantar without modifying it. In case you want to apply some transformation on the value, such as hashing, you can provide the `processUserId` callback when instantiating the plugin:

**iOS:**

```swift
let focalMeterConfig = FocalMeterConfiguration(kantarEndpoint: "https://thekantarendpoint.com") { userId in
  return hash(userId) // apply custom hashing function
}
let tracker = Snowplow.createTracker(namespace: namespace, network: networkConfig, configurations: [focalMeterConfig])
```

**Android (Kotlin):**

```kotlin
val focalMeterConfig = FocalMeterConfiguration("https://thekantarendpoint.com") { userId ->
  hash(userId) // apply custom hashing function
}
val tracker = Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig)
```

**Android (Java):**

```java
FocalMeterConfiguration focalMeterConfig = new FocalMeterConfiguration(
  "https://thekantarendpoint.com",
  userId -> hash(userId) // apply custom hashing function
);
Snowplow.createTracker(applicationContext, namespace, networkConfiguration, focalMeterConfig);
```

***

---

# Native mobile tracker plugins
> Extend mobile tracker functionality using plugins to enrich events, filter tracking, or inspect tracked data.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/plugins/

> **Note:** Tracker plugins are only available since version 5 of the mobile trackers.

The mobile trackers provide a plugin architecture that allows new functionality to be added to the trackers. There are several Snowplow maintained plugins, however you are also free to build your own or leverage community plugins too.

Plugins provide the ability to intercept tracked events in order to enrich them with additional entities, filter them or just to inspect them. To implement this, they contain three extension points:

1. `entities` callback which can return context entities to enrich events before they are tracked. The callback is passed the tracked event object to be enriched.
2. `filter` callback that returns true or false in order to decide whether to track a given event or not. The callback is passed the tracked event object.
3. `afterTrack` callback which can be used to inspect final tracked events. The passed event object contains all its properties as well as context entities.

## Creating a custom plugin

In order to build your own plugin, you can instantiate the `PluginConfiguration` class. It requires you to pass a unique identifier. There can only be a single plugin for a given identifier registered with the tracker.

**iOS:**

```swift
let plugin = PluginConfiguration(identifier: "myPlugin")
```

**Android (Kotlin):**

```kotlin
val plugin = PluginConfiguration("myPlugin")
```

**Android (Java):**

```java
PluginConfiguration plugin = new PluginConfiguration("myPlugin");
```

***

### Adding context entities

To enrich events with additional context entities, you can make use of the `entities` callback. This function accepts an optional list of schemas of self-describing events to subscribe to. To filter primitive (not self-describing events) such as structured events, you can pass their event names in the schemas list (`se` for structured events). When no list of schemas is passed, the callback is called for all tracked events.

The following code will add a specific entity, with URI "iglu:xx" (not a valid URI, for example only), to all Structured and ScreenView events.

**iOS:**

```swift
plugin.entities(schemas: [
    // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.
    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events
    "se" // structured events
]) { event in
    return [
        SelfDescribingJson(schema: "iglu:xx", andData: ["yy": true])
    ]
}
```

**Android (Kotlin):**

```kotlin
plugin.entities(
    schemas = listOf(
        // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.
        "iglu:com.google.analytics.measurement-protocol/screen_view/jsonschema/1-0-0", // screen view events
        "se" // structured events
    )
) {
    listOf(SelfDescribingJson("iglu:xx", hashMapOf("yy" to true)))
}
```

**Android (Java):**

```java
plugin.entities(
        Arrays.asList(
                // The list of schemas to call the closure for is optional. If not passed, the callback is called for all events.
                "iglu:com.google.analytics.measurement-protocol/screen_view/jsonschema/1-0-0", // screen view events
                "se" // structured events
        ),
        event -> Collections.singletonList(new SelfDescribingJson("iglu:xx", Collections.singletonMap("yy", true)))
);
```

***

### Filtering events

The `filter` callback enables you to add custom logic that decides whether a given event should be tracked or not. This can for example enable you to intercept events automatically tracked by the tracker and skip some of them. The callback returns true in case the event should be tracked and false otherwise. It also accepts the same optional `schemas` array as the `entities` callback to only be applied to events with those schemas.

The following code will apply to all screen view events and only accept ones with the name "Home Screen", other screen view events will be discarded:

**iOS:**

```swift
plugin.filter(schemas: [
    "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events
]) { event in
    return event.payload["name"] as? String == "Home Screen"
}
```

**Android (Kotlin):**

```kotlin
plugin.filter(
    schemas = listOf(
        "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events
    )
) { event ->
    event.payload["name"] == "Home Screen"
}
```

**Android (Java):**

```java
plugin.filter(
        Collections.singletonList(
                "iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0", // screen view events
        ),
        event -> event.getPayload().get("name") == "Home Screen"
);
```

***

### Inspecting events after they are tracked

To inspect events after they are tracked, you can make use of the `afterTrack` callback. It also accepts the same optional `schemas` array as the `entities` and `filter` callbacks to filter events.

**iOS:**

```swift
plugin.afterTrack { event in
    print("Tracked event with \(event.entities.count) entities")
}
```

**Android (Kotlin):**

```kotlin
plugin.afterTrack {
    println("Tracked event with ${it.entities.count()} entities")
}
```

**Android (Java):**

```java
plugin.afterTrack(
        null,
        event -> System.out.printf("Tracked event with %d entities%n", event.getEntities().size())
);
```

***

## Registering a plugin in the tracker

Once you have a plugin configuration, you can register it when creating a new tracker:

**iOS:**

```swift
// the plugin is supplied to the tracker as a configuration
let tracker = Snowplow.createTracker(namespace: "namespace",
                                     network: networkConfig,
                                     configurations: [plugin])
```

**Android (Kotlin):**

```kotlin
// the plugin is supplied to the tracker as a configuration
val tracker = Snowplow.createTracker(
    applicationContext,
    "namespace",
    networkConfiguration,
    plugin
)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(
        getApplicationContext(),
        "namespace",
        networkConfiguration,
        plugin
);
```

***

You will be able to list the registered plugin identifiers:

**iOS:**

```swift
// one can inspect the enabled plugins and get the list of their identifiers
let pluginIdentifiers = tracker?.plugins.identifiers
```

**Android (Kotlin):**

```kotlin
// one can inspect the enabled plugins and get the list of their identifiers
val pluginIdentifiers = tracker.plugins.identifiers
```

**Android (Java):**

```java
// one can inspect the enabled plugins and get the list of their identifiers
List<String> identifiers = tracker.getPlugins().getIdentifiers();
```

***

You can also add new plugins to an already instantiated tracker object:

**iOS:**

```swift
tracker?.plugins.add(plugin: otherPlugin)
```

**Android (Kotlin):**

```kotlin
tracker.plugins.addPlugin(otherPlugin)
```

**Android (Java):**

```java
tracker.getPlugins().addPlugin(otherPlugin);
```

***

And remove a plugin from a tracker instance:

**iOS:**

```swift
tracker?.plugins.remove(identifier: "myPlugin")
```

**Android (Kotlin):**

```kotlin
tracker.plugins.removePlugin("myPlugin")
```

**Android (Java):**

```java
tracker.getPlugins().removePlugin("myPlugin");
```

***

---

# Update the native mobile trackers configuration remotely
> Configure mobile trackers to download and apply configuration updates remotely without requiring app resubmission.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/remote-configuration/

Remote configuration allows for the configuration of the tracker without distributing an app update. When remote configuration is not used, changing the tracking configuration can require the resubmission of the whole app bundle.

With remote configuration, the tracker can download a configuration file and automatically reconfigure itself based on the downloaded file.

A tracker that will be remotely configured is created slightly differently from a locally-configured tracker, which is created using `Snowplow.createTracker()`. Instead, `Snowplow.setup()` is used. This example code will create a tracker with namespace "defaultNamespace" that can be configured remotely. Access the tracker using `Snowplow.defaultTracker` or `Snowplow.getTracker("defaultNamespace")`.

**iOS:**

1. In your application delegate `AppDelegate.swift` add `import SnowplowTracker`.

2. In the `application(_:didFinishLaunchingWithOptions:)` method, set up the SDK as follows:

```swift
// Indicate the URL where to download the config file
let remoteConfig = RemoteConfiguration(
  endpoint: "https://remote-config.com",
  method: .get
)

// Default configuration used when the remote config is not accessible
// Network, tracker, subject and sessionConfigurations can be provided
let defaultNetworkConfig = NetworkConfiguration(endpoint: "https://snowplow-collector-url.com", method: .post)
let bundle = ConfigurationBundle(namespace: "defaultNamespace", networkConfiguration: defaultNetworkConfig)
bundle.trackerConfiguration = TrackerConfiguration(appId: "app-id")
  .screenViewAutotracking(true)
let defaultConfig = [bundle]

// Optional callback for when the tracker reconfigures itself passing a list of active namespaces and the state of the configuration describing where it was fetched from
let successCallback: ([String]?) -> Void = { namespaces, state in
  // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.
}

// Set up tracker with remote configuration
Snowplow.setup(
  remoteConfiguration: remoteConfig,
  defaultConfiguration: defaultConfig,
  defaultConfigurationVersion: 1,
  onSuccess: successCallback
)
```

**Android (Kotlin):**

```kotlin
// Indicate the URL where to download the config file
val remoteConfig = RemoteConfiguration("https://remote-config.com", HttpMethod.GET)

// Default configuration used in case the remote config is not accessible
// Network, tracker, subject and sessionConfigurations can be provided
val defaultNetworkConfig = NetworkConfiguration("https://snowplow-collector-url.com", HttpMethod.POST)
val defaultConfig: List<ConfigurationBundle> =
    listOf(ConfigurationBundle("defaultNamespace", defaultNetworkConfig))

// Set up tracker with remote configuration
Snowplow.setup(
  context = applicationContext,
  remoteConfiguration = remoteConfig,
  defaultBundles = defaultConfig,
  defaultBundleVersion = 1
) {
  // This optional callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.
  configurationPair: Pair<List<String>, ConfigurationState?>? ->
    val namespaces = configurationPair?.first
    val configurationState =
        configurationPair?.second // either cached, default or fetched from the remote endpoint
}
```

**Android (Java):**

```java
// Indicate the URL where to download the config file
RemoteConfiguration remoteConfig = new RemoteConfiguration("https://remote-config.com", HttpMethod.get);

// Default configuration used in case the remote config is not accessible
// Network, tracker, subject and sessionConfigurations can be provided
NetworkConfiguration defaultNetworkConfig = new NetworkConfiguration("https://snowplow-collector-url.com", HttpMethod.post);
List<ConfigurationBundle> defaultConfig = Lists.listOf(new ConfigurationBundle("defaultNamespace", defaultNetworkConfig));

// Set up tracker with remote configuration
Snowplow.setup(getApplicationContext(),
  remoteConfig,
  defaultConfig,
  1, // version of the default configuration
  configurationPair -> {
  // This optional callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.
  List<String> namespaces = configurationPair.first;
  ConfigurationState configurationState = configurationPair.second; // either cached, default or fetched from the remote endpoint
});
```

***

On app startup, the Snowplow tracker initializer will attempt to download the remote configuration file. Meanwhile, it will initialize the tracker instance (or multiple tracker instances) based on the last cached configuration. The cached configuration is the last configuration downloaded remotely. If it's not available, the tracker initializer will spin up the default configuration passed as a parameter.

Every time the initializer successfully initializes the tracker instances it calls a callback passing the list of activated namespaces and the state of the configuration which represents the source where the configuration was retrieved from (using default values, cache, or fetched from the remote endpoint). The callback can be used for last minute settings at runtime once the tracker has been instanced.

## Default configuration

The default configuration will be used right after calling the `setup()` method in case there is no cached configuration (one that was fetched from the remote endpoint and stored in a cache file).

The `defaultConfiguration` parameter takes a list of `ConfigurationBundle` objects. Each one represents a tracker instance and it's own configuration. Each tracker instance is identified by a `namespace` which is a parameter required in each `ConfigurationBundle`.

The optional `defaultConfigurationVersion` parameter specifies the version of the default configuration. If the fetched remote configuration has version lower or equal to the default configuration, it will not be used.

## On success callback

The optional `onSuccess` callback is called for each successful configuration.

For example, if the tracker has a cached configuration and it's downloading a new configuration, first the `onSuccess` callback will be called for the cached configuration and then it will be called when the downloaded configuration is applied. When it happens the cached configuration is substituted with the downloaded one.

## The remote configuration file

The URL where the configuration file is hosted must be provided in the `RemoteConfiguration` object passed to the `setup()` method. There aren't restrictions about where to host the file but possible solutions may be S3 with Cloudfront or a Google Cloud Storage bucket.

The configuration file is simply a JSON file compliant with the [Remote Config JSONSchema](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/remote_config/jsonschema/1-0-0).

An example of remote config specification:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.mobile/remote_config/jsonschema/1-0-0",
  "configurationVersion": 1,
  "configurationBundle": [
    {
      "namespace": "testTracker1",
      "networkConfiguration": {
        "endpoint": "http://<collector.url>",
        "method": "get"
      },
      "trackerConfiguration": {
        "appId": "app-identifier",
        "devicePlatform": "mob",
        "logLevel": "off",
        "sessionContext": false,
        "applicationContext": false,
        "platformContext": false,
        "geoLocationContext": false,
        "screenContext": false,
        "screenViewAutotracking": false,
        "lifecycleAutotracking": false,
        "installAutotracking": false,
        "exceptionAutotracking": false,
        "diagnosticAutotracking": false
      },
      "subjectConfiguration": {
        "userId": "example",
        "networkUserId": "example",
        "domainUserId": "example",
        "useragent": "example",
        "ipAddress": "example",
        "timezone": "example",
        "language": "example"
      },
      "sessionConfiguration": {
        "backgroundTimeout": 1800,
        "foregroundTimeout": 1800
      }
    }
  ]
}
```

The required fields are:

- `$schema`: It specifies the format of the configuration file and it's used by the tracker to check if the format is compatible with that version of the tracker.

- `configurationVersion`: It's an incremental version number that identifies the current configuration. The tracker compares this value with the configurationVersion of the cached configuration to decide whether to update or not the tracker configuration.

  > **Warning:** The version MUST be increased on each update.

- `configurationBundle`: This is a list of configurations for the various tracker instances the developer wants to activate in the app. Usually there is a unique tracker instance for the app so the configurationBundle will likely be an array of a single object like in the example above.

The elements of the `configurationBundle` list are just a subset of the common configuration settings described in the "Introduction" section. At the moment the configuration objects configurable remotely are:

- `networkConfiguration`
- `trackerConfiguration`
- `subjectConfiguration`
- `sessionConfiguration`

Note: The `networkConfiguration` is fundamental in order to set the collector endpoint, the URL where the tracked events will be sent. If the ConfigurationBundle has a `namespace` but not the `networkConfiguration`, the tracker initializer will remove the tracker instance with the corresponding namespace.

## Refresh the configuration

By default the tracker will attempt to download the configuration file at the start up of the app. To check for configuration updates more often, for example at runtime or when the app comes back from background state, it's possible to use the `refresh()` method.

**iOS:**

```swift
let successCallback: ([String]?) -> Void = { namespaces, state in
    // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.
}

Snowplow.refresh(onSuccess: successCallback)
```

**Android (Kotlin):**

```kotlin
Snowplow.refresh(
    applicationContext
) { configurationPair: Pair<List<String>, ConfigurationState>? ->
    val namespaces = configurationPair?.first
    val configurationState = configurationPair?.second
}
```

**Android (Java):**

```java
Snowplow.refresh(getApplicationContext(), configurationPair -> {
  List<String> namespaces = configurationPair.first;
  ConfigurationState configurationState = configurationPair.second;

  // This callback can be used for last minute, post-configuration, updates once the tracker instance is enabled and configured.
});
```

***

The method needs only the callback parameter, without any remote configuration url or default configuration, because this is intended just for the configuration updates, not for the initial setup. The tracker will be configured only if a new configuration (with a higher `configurationVersion` value) is available at the url indicated in the `RemoteConfiguration` passed earlier at start up of the app in the `setup` method call.

## Policy for runtime settings and remote configuration updates

When the tracker initializer updates the tracker configuration, it would reset all the previous configuration with the new settings. Obviously, this can cause issues in case some runtime configuration has been applied meanwhile. To avoid this the tracker keeps track of runtime changes in the configuration settings and when a new remote configuration is downloaded and applied, it doesn't override the settings already changed at runtime.

A clear example is a runtime change on `userId` on `SubjectController`:

**iOS:**

```swift
// Load configuration with userId = nil
Snowplow.setup(remoteConfiguration: remoteConfig, defaultConfiguration: defaultConfig, onSuccess: successCallback)

// userId is set to nil

// ...later...

tracker.subject.userId = "my-runtime-updated-userId"

// userId is is set to "my-runtime-updated-userId"

// ...later...

// Later refreshing the configuration with userId = nil
Snowplow.refresh(onSuccess: successCallback)

// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

**Android (Kotlin):**

```kotlin
// Load configuration with userId = nil
Snowplow.setup(applicationContext, remoteConfig, defaultConfig, successCallback)

// userId is set to nil

// ...later...

tracker.subject.userId = "my-runtime-updated-userId"

// userId is is set to "my-runtime-updated-userId"

// ...later...

// Later refreshing the configuration with userId = nil
Snowplow.refresh(applicationContext, successCallback)

// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

**Android (Java):**

```java
// Load configuration with userId = nil
Snowplow.setup(getApplicationContext(), remoteConfig, defaultConfig, successCallback);

// userId is set to nil

// ...later...

tracker.getSubject().userId = "my-runtime-updated-userId"

// userId is is set to "my-runtime-updated-userId"

// ...later...

// Later refreshing the configuration with userId = nil
Snowplow.refresh(getApplicationContext(), successCallback);

// userId is still set to "my-runtime-updated-userId" because it was set at runtime
```

***

---

# Track ecommerce events with the native mobile trackers
> Track comprehensive ecommerce interactions including product views, cart actions, checkouts, and transactions in mobile apps.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/

> **Note:** Snowplow ecommerce tracking was added in version 5.4.0. With the addition of these new ecommerce events and entities, we have deprecated the old `EcommerceTransaction` and `EcommerceTransactionItem` events. [Migration guide](/docs/sources/mobile-trackers/migration-guides/migration-guide-to-new-ecommerce/).

The Snowplow ecommerce tracking APIs enable you to track events from your ecommerce store on the [web](/docs/sources/web-trackers/tracking-events/ecommerce/) as well as mobile apps.

All ecommerce events must be manually tracked; there is no ecommerce auto-tracking.

## Ecommerce events

Ecommerce events are tracked like normal Snowplow events. For example, tracking an ecommerce `ProductViewEvent` event:

**iOS:**

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")

let product = ProductEntity(
  id: "productId",
  category: "category",
  currency: "GBP",
  price: 100
)
let event = ProductViewEvent(product: product)
let eventId = tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.createTracker(
    context = applicationContext, // Android context
    namespace = "appTracker",
    endpoint = "https://snowplow-collector-url.com" // Event collector URL
)

val event = ProductViewEvent(ProductEntity(
  id = "productId",
  category = "category",
  currency = "GBP",
  price = 100
  )
)
tracker.track(event)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(
      getApplicationContext(), // Android context
      "appTracker", // namespace
      "https://snowplow-collector-url.com" // Event collector URL
);

ProductViewEvent event = new ProductViewEvent(new ProductEntity(
  "productId", // id
  "category",  // category
  "GBP", // currency
  100 // price
  )
);
tracker.track(event);
```

***

Add or update properties using setters. For example, adding data to a PromotionEntity:

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")
promotion.name = "bogof"
promotion.type = "popup"
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")
promotion.name = "bogof"
promotion.type = "popup"
```

**Android (Java):**

```java
PromotionEvent promotion = new PromotionEntity("promoId");
promotion.setName("bogof");
promotion.setType("popup");
```

***

This table lists all the ecommerce events.

| `Event`                 | Used for                                                                                                                                       |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `ProductViewEvent`      | Tracking a visit to a product detail screen. Also known as product detail view.                                                                |
| `AddToCartEvent`        | Track an addition to cart.                                                                                                                     |
| `RemoveFromCartEvent`   | Track a removal from cart.                                                                                                                     |
| `ProductListViewEvent`  | Track an impression of a product list. The list could be a search results page, recommended products, upsells etc.                             |
| `ProductListClickEvent` | Track the click/selection of a product from a product list.                                                                                    |
| `PromotionViewEvent`    | Track an impression for an internal promotion banner or slider or any other type of content that showcases internal products/categories.       |
| `PromotionClickEvent`   | Track the click/selection of an internal promotion.                                                                                            |
| `CheckoutStepEvent`     | Track a checkout step completion in the checkout process together with common step attributes for user choices throughout the checkout funnel. |
| `TransactionEvent`      | Track a transaction/purchase completion.                                                                                                       |
| `TransactionErrorEvent` | Track a failed transaction.                                                                                                                    |
| `RefundEvent`           | Track a transaction partial or complete refund.                                                                                                |

Each ecommerce event is a [self-describing](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) event using a single schema, `iglu:com.snowplowanalytics.snowplow.ecommerce/snowplow_ecommerce_action/jsonschema/1-0-2`.

**Ecommerce action event properties**

| Request Key | Required | Type/Format | Description                                                                   |
| ----------- | -------- | ----------- | ----------------------------------------------------------------------------- |
| type        | Y        | string      | Specific ecommerce event type (automatically tracked).                        |
| name        | N        | string      | For `ProductListView` and `ProductListClick` events: human-readable list name |

The events are distinguished by their `type` property, which is different for each `Event` class tracked. Aside from the optional list `name` in the `ProductListViewEvent` and `ProductListClickEvent` events, all tracked ecommerce properties are tracked as entities.

> **Note:** Check out the API docs ([Android](https://snowplow.github.io/snowplow-android-tracker/), [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/snowplow/)) for the full details of each Event and Entity.

### ProductView

Tracking a visit to a details screen for a specific product.

**iOS:**

```swift
let product = ProductEntity(
  id: "plow2",
  category: "snow.clearance.ploughs.large",
  currency: "NOK",
  price: 5000
)
let event = ProductViewEvent(product: product)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(
  id = "plow2",
  category = "snow.clearance.ploughs.large",
  currency = "NOK",
  price = 5000
)
val event = ProductViewEvent(product)

tracker.track(event)
```

**Android (Java):**

```java
ProductViewEvent event = new ProductViewEvent(new ProductEntity(
    "plow2", // id
    "snow.clearance.ploughs.large",  // category
    "NOK", // currency
    5000 // price
  )
);

tracker.track(event);
```

***

### AddToCart

Tracking one or more products being added to a cart.

**iOS:**

```swift
let product = ProductEntity(
  id: "productId",
  category: "clothes/shirts",
  currency: "EUR",
  price: 100.50
)
let cart = CartEntity(totalValue: 200, currency: "EUR")
let event = AddToCartEvent(products: [product], cart: cart)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(
  id = "productId",
  category = "clothes/shirts",
  currency = "EUR",
  price = 100.50
)
val cart = CartEntity(totalValue = 200, currency = "EUR")
val event = AddToCartEvent(listOf(product), cart)

tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(
  "productId", // id
  "clothes/shirts", // category
  "EUR", // currency
  100.50 // price
);
List<Product> products = new ArrayList<>();
products.add(product);

CartEntity cart = new CartEntity(
  200, // totalValue
  "EUR" // currency
);
AddToCartEvent event = new AddToCartEvent(products, cart);

tracker.track(event);
```

***

### RemoveFromCart

Tracking one or more products being removed from a cart.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "clothes/shirts", currency: "EUR", price: 100.50)
let cart = CartEntity(totalValue: 200, currency: "EUR")
let event = RemoveFromCartEvent(products: [product], cart: cart)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(
  id = "productId",
  category = "clothes/shirts",
  currency = "EUR",
  price = 100.50
)
val cart = CartEntity(totalValue = 200, currency = "EUR")
val event = RemoveFromCartEvent(listOf(product), cart)

tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(
  "productId", // id
  "clothes/shirts", // category
  "EUR", // currency
  100.50 // price
);
List<Product> products = new ArrayList<>();
products.add(product);

CartEntity cart = new CartEntity(
  200, // totalValue
  "EUR" // currency
);
RemoveFromCartEvent event = new RemoveFromCartEvent(products, cart);

tracker.track(event);
```

***

### ProductListView

Track an impression of a list of products. You can optionally provide a name for the list.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "software", currency: "USD", price: 99.99)
let event = ProductListViewEvent(products: [product], name: "snowplowProducts")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(
  id = "productId",
  category = "software",
  currency = "USD",
  price = 99.99
)
val event = ProductListViewEvent(listOf(product), name = "snowplowProducts")

tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(
  "productId", // id
  "software", // category
  "USD", // currency
  99.99 // price
);
List<Product> products = new ArrayList<>();
products.add(product);
ProductListViewEvent event = new ProductListViewEvent(
  products, // products
  "snowplowProducts" // name
);

tracker.track(event);
```

***

### ProductListClick

Track a specific product being selected from a list. You can optionally provide a name for the list.

**iOS:**

```swift
let product = ProductEntity(id: "productId", category: "software", currency: "USD", price: 99.99)
let event = ProductListClickEvent(product: product, name: "snowplowProducts")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val product = ProductEntity(
  id = "productId",
  category = "software",
  currency = "USD",
  price = 99.99
)
val event = ProductListClickEvent(product, name = "snowplowProducts")

tracker.track(event)
```

**Android (Java):**

```java
ProductEntity product = new ProductEntity(
  "productId", // id
  "software", // category
  "USD", // currency
  99.99 // price
);
ProductListClickEvent event = new ProductListClickEvent(
  product, // product
  "snowplowProducts" // name
);

tracker.track(event);
```

***

### CheckoutStep

Track the completion of a step in the checkout funnel, along with common checkout properties such as payment method or coupon code. See the API docs for the full details ([Android](https://snowplow.github.io/snowplow-android-tracker/), [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/snowplow/)).

**iOS:**

```swift
let event = CheckoutStepEvent(step: 3, deliveryMethod: "next_day")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = CheckoutStepEvent(step = 3, deliveryMethod = "next_day")

tracker.track(event)
```

**Android (Java):**

```java
CheckoutStepEvent event = new CheckoutStepEvent(
  3, // step
  null, // shippingPostcode
  null, // billingPostcode
  null, // shippingFullAddress
  null, // billingFullAddress
  "next_day" // deliveryMethod
);

tracker.track(event);
```

***

### Transaction

Track the completion of a purchase or transaction, along with common transaction properties such as shipping cost or discount amount.

**iOS:**

```swift
let transaction = TransactionEntity(
  transactionId: "id-123",
  revenue: 50000,
  currency: "JPY",
  paymentMethod: "debit",
  totalQuantity: 2
)
let event = TransactionEvent(transaction)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val transaction = TransactionEntity(
  id = "id-123",
  revenue = 50000,
  currency = "JPY",
  paymentMethod = "debit",
  totalQuantity = 2
)
val event = TransactionEvent(transaction)

tracker.track(event)
```

**Android (Java):**

```java
TransactionEntity transaction = new TransactionEntity(
  "id-123", // id
  50000, // revenue
  "JPY", // currency
  "debit", // paymentMethod
  2 // totalQuantity
);
TransactionEvent event = new TransactionEvent(transaction);

tracker.track(event);
```

***

### TransactionError

Track an error occurring during a transaction.

**iOS:**

```swift
let transaction = TransactionEntity(
  transactionId: "id-123",
  revenue: 50000,
  currency: "JPY",
  paymentMethod: "debit",
  totalQuantity: 2
)
let event = TransactionErrorEvent(
  transaction: transaction,
  errorCode: "E05",
  errorDescription: "connection_failure"
)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val transaction = TransactionEntity(
  id = "id-123",
  revenue = 50000,
  currency = "JPY",
  paymentMethod = "debit",
  totalQuantity = 2
)
val event = TransactionErrorEvent(
    transaction = transaction,
    errorCode = "E05",
    errorDescription = "connection_failure"
)

tracker.track(event)
```

**Android (Java):**

```java
TransactionEntity transaction = new TransactionEntity(
  "id-123", // id
  50000, // revenue
  "JPY", // currency
  "debit", // paymentMethod
  2 // totalQuantity
);
TransactionErrorEvent event = new TransactionErrorEvent(
    transaction, // transaction
    "E05", // errorCode
    null, // errorShortcode
    "connection_failure" // errorDescription
);

tracker.track(event);
```

***

### Refund

Track a refund being requested for part of, or the entirety of, a transaction.

**iOS:**

```swift
let event = RefundEvent(
  transactionId: "id-123", // use the transaction ID from the original Transaction event
  refundAmount: 20000,
  currency: "JPY"
)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Refund(
    id = "id-123", // use the transaction ID from the original Transaction event
    refundAmount = 20000,
    currency = "JPY"
)

tracker.track(event)
```

**Android (Java):**

```java
RefundEvent event = new RefundEvent(
    "id-123", // id; use the transaction ID from the original Transaction event
    20000, // refundAmount
    "JPY" // currency
);

tracker.track(event);
```

***

### PromotionView

Track an impression for any kind of internal promotion.

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")
let event = PromotionViewEvent(promotion: promotion)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")
val event = PromotionViewEvent(promotion)

tracker.track(event)
```

**Android (Java):**

```java
PromotionEntity promotion = new PromotionEntity("promoId");
PromotionViewEvent event = new PromotionViewEvent(promotion);

tracker.track(event);
```

***

### PromotionClick

Track an internal promotion being selected.

**iOS:**

```swift
let promotion = PromotionEntity(id: "promoId")
let event = PromotionViewEvent(promotion: promotion)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val promotion = PromotionEntity(id = "promoId")
val event = PromotionClick(promotion)

tracker.track(event)
```

**Android (Java):**

```java
PromotionEntity promotion = new PromotionEntity("promoId");
PromotionClickEvent event = new PromotionClickEvent(promotion);

tracker.track(event);
```

***

## Ecommerce entities

### Product entity

The product entity is used in several ecommerce events. It contains information about the product the user is interacting with.

**Product entity properties**

| Request Key     | Required | Type/Format | Description                                |
| --------------- | -------- | ----------- | ------------------------------------------ |
| id              | Y        | string      | The SKU or product ID.                     |
| category        | Y        | string      | Category the product belongs to.           |
| currency        | Y        | string      | Currency in which the product is priced.   |
| price           | Y        | number      | Current price.                             |
| listPrice       | N        | boolean     | Recommended or list price.                 |
| name            | N        | string      | Product name or title.                     |
| quantity        | N        | integer     | Quantity of the product (for cart events). |
| size            | N        | string      | Product size.                              |
| variant         | N        | string      | Product variant.                           |
| brand           | N        | string      | Product brand.                             |
| inventoryStatus | N        | string      | Inventory status, e.g. in stock.           |
| position        | N        | integer     | Position in a list of products.            |
| creativeId      | N        | string      | Identifier for the creative shown.         |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/product/jsonschema/1-0-0`.

### Cart entity

The cart entity is used for `AddToCart` and `RemoveFromCart` events.

**Cart entity properties**

| Request Key | Required | Type/Format | Description                         |
| ----------- | -------- | ----------- | ----------------------------------- |
| totalValue  | Y        | number      | Total cart value after this action. |
| currency    | Y        | string      | Currency used for the cart.         |
| cartId      | N        | string      | Unique cart identifier.             |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/cart/jsonschema/1-0-0`.

### Checkout step entity

The checkout step entity is used for `CheckoutStep` events.

**Checkout step entity properties**

| Request Key         | Required | Type/Format | Description                                                       |
| ------------------- | -------- | ----------- | ----------------------------------------------------------------- |
| step                | Y        | integer     | Checkout step index.                                              |
| shippingPostcode    | N        | string      | Shipping address postcode.                                        |
| billingPostcode     | N        | string      | Billing address postcode.                                         |
| shippingFullAddress | N        | string      | Shipping address.                                                 |
| billingFullAddress  | N        | string      | Billing address.                                                  |
| deliveryProvider    | N        | string      | Delivery provider e.g. DHL, Royal Mail.                           |
| deliveryMethod      | N        | string      | How the customer will get the product.                            |
| couponCode          | N        | string      | An applied coupon.                                                |
| accountType         | N        | string      | Customer account type, e.g. guest                                 |
| paymentMethod       | N        | string      | Payment method selected.                                          |
| proofOfPayment      | N        | string      | e.g. invoice, receipt                                             |
| marketingOptIn      | N        | boolean     | Whether the user email address is opted into marketing campaigns. |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/checkout_step/jsonschema/1-0-0`.

### Transaction entity

The transaction entity is used for `Transaction` and `TransactionError` events.

**Transaction entity properties**

| Request Key    | Required | Type/Format | Description                                |
| -------------- | -------- | ----------- | ------------------------------------------ |
| transactionId  | Y        | string      | Unique transaction identifier.             |
| revenue        | Y        | number      | Value of transaction.                      |
| currency       | Y        | string      | Currency used in transaction.              |
| paymentMethod  | Y        | string      | Payment method used.                       |
| totalQuantity  | Y        | integer     | Number of items in the transaction.        |
| tax            | N        | number      | Tax value in transaction.                  |
| shipping       | N        | number      | Shipping costs in transaction.             |
| discountCode   | N        | string      | An applied discount code.                  |
| discountAmount | N        | string      | Amount discounted from transaction.        |
| creditOrder    | N        | boolean     | Whether the transaction is a credit order. |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction/jsonschema/1-0-0`.

### Transaction error entity

The transaction error entity is used for `TransactionError` events.

**Transaction error entity properties**

| Request Key      | Required | Type/Format | Description                    |
| ---------------- | -------- | ----------- | ------------------------------ |
| errorCode        | N        | string      | Error-identifying code.        |
| errorShortcode   | N        | string      | Shortcode for the error.       |
| errorDescription | N        | string      | Longer description.            |
| errorType        | N        | string enum | Is the error "hard" or "soft". |
| resolution       | N        | string      | The chosen error resolution.   |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/transaction_error/jsonschema/1-0-0`.

### Refund entity

The refund entity is used for `Refund` events.

**Refund entity properties**

| Request Key   | Required | Type/Format | Description                                   |
| ------------- | -------- | ----------- | --------------------------------------------- |
| transactionId | Y        | string      | The identifier from the original transaction. |
| refundAmount  | Y        | number      | Amount to be refunded.                        |
| currency      | Y        | string      | Currency used in transaction.                 |
| refundReason  | N        | string      | Reason for the refund.                        |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/refund/jsonschema/1-0-0`.

### Promotion entity

The promotion entity is used for `PromotionView` and `PromotionClick` events.

**Promotion entity properties**

| Request Key | Required | Type/Format     | Description                                        |
| ----------- | -------- | --------------- | -------------------------------------------------- |
| id          | Y        | string          | The unique promotion identifier.                   |
| name        | N        | string          | Promotion name.                                    |
| productIds  | N        | list of strings | IDs of products in the promotion.                  |
| position    | N        | integer         | Index of the promotion in a list such as a banner. |
| creativeId  | N        | string          | Identifier for the promotion creative.             |
| type        | N        | string          | Type of promotion.                                 |
| slot        | N        | string          | Where the promotion was displayed.                 |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/promotion/jsonschema/1-0-0`.

## Global ecommerce entities Screen/Page and User

Use these APIs to add ecommerce context information to every subsequent event tracked.

> **Note:** These entities will be added to **all** events, not just ecommerce ones. This is for consistency with entities in the JavaScript tracker.

### Ecommerce Screen (Page) entity

> **Note:** The `setEcommerceScreen` method adds a `Page` (rather than `Screen`) entity to all events, for consistency with web tracking.

The Ecommerce Screen/Page entity helps in grouping insights by screen/page type, e.g. Product description, Product list, Home screen.

To set a Screen/Page entity you can use the `setEcommerceScreen` method:

**iOS:**

```swift
let entity = EcommerceScreenEntity(type: "demo_app_screen")
Snowplow.defaultTracker()?.ecommerce.setEcommerceScreen(entity)

// setting EcommerceScreen again will replace the original entity
let newEntity = EcommerceScreenEntity(type: "product_list", language: "EN-GB", locale: "UK")
Snowplow.defaultTracker()?.ecommerce.setEcommerceScreen(newEntity)

// remove the saved properties and stop the Page entity being added
Snowplow.defaultTracker()?.ecommerce.removeEcommerceScreen()
```

**Android (Kotlin):**

```kotlin
val entity = EcommerceScreenEntity(type = "demo_app_screen")
Snowplow.defaultTracker?.ecommerce.setEcommerceScreen(entity)

// setting EcommerceScreen again will replace the original entity
val newEntity = EcommerceScreenEntity(type = "product_list", language = "EN-GB", locale = "UK")
Snowplow.defaultTracker?.ecommerce.setEcommerceScreen(newEntity)

// remove the saved properties and stop the Page entity being added
Snowplow.defaultTracker?.ecommerce.removeEcommerceScreen()
```

**Android (Java):**

```java
EcommerceScreenEntity entity = new EcommerceScreenEntity(
  "demo_app_screen" // type
)
Snowplow.getDefaultTracker().getEcommerce().setEcommerceScreen(entity);

// setting EcommerceScreen again will replace the original entity
EcommerceScreenEntity newEntity = new EcommerceScreenEntity(
  "product_list", // type
  "EN-GB", // language
  "UK" // locale
)
Snowplow.getDefaultTracker().getEcommerce().setEcommerceScreen(newEntity);

// remove the saved properties and stop the Page entity being added
Snowplow.getDefaultTracker().getEcommerce().removeEcommerceScreen();
```

***

**Screen/Page entity properties**

| Request Key | Required | Type/Format | Description                   |
| ----------- | -------- | ----------- | ----------------------------- |
| type        | Y        | string      | Type of screen.               |
| language    | N        | string      | Language used for the screen. |
| locale      | N        | string      | Locale version.               |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/page/jsonschema/1-0-0`.

### Ecommerce User entity

The Ecommerce User entity helps in modeling guest/non-guest account interactions.

To set an Ecommerce User entity you can use the `setEcommerceUser` method with the following attributes:

**iOS:**

```swift
let entity = EcommerceUserEntity(id: "userId12345", isGuest: true)
Snowplow.defaultTracker()?.ecommerce.setEcommerceUser(entity)

// setting EcommerceUser again will replace the original entity
let newEntity = EcommerceUserEntity(id: "userId67890", isGuest: false, email: "email@email.com")
Snowplow.defaultTracker()?.ecommerce.setEcommerceUser(newEntity)

// remove the saved properties and stop the User entity being added
Snowplow.defaultTracker()?.ecommerce.removeEcommerceUser()
```

**Android (Kotlin):**

```kotlin
val entity = EcommerceUserEntity(id = "userId12345", isGuest = true)
Snowplow.defaultTracker?.ecommerce.setEcommerceUser(entity)

// setting EcommerceUser again will replace the original entity
val newEntity = EcommerceUserEntity(id = "userId67890", isGuest = false, email = "email@email.com")
Snowplow.defaultTracker?.ecommerce.setEcommerceUser(newEntity)

// remove the saved properties and stop the User entity being added
Snowplow.defaultTracker?.ecommerce.removeEcommerceUser()
```

**Android (Java):**

```java
EcommerceUserEntity entity = new EcommerceUserEntity(
  "userId12345", // id
  true // isGuest
)
Snowplow.getDefaultTracker().getEcommerce().setEcommerceUser(entity);

// setting ScreenType again will replace the original entity
EcommerceUserEntity newEntity = new EcommerceUserEntity(
  "userId67890", // id
  false, // isGuest
  "email@email.com" // email
)
Snowplow.getDefaultTracker().getEcommerce().setEcommerceUser(newEntity);

// remove the saved properties and stop the Page entity being added
Snowplow.getDefaultTracker().getEcommerce().removeEcommerceUser();
```

***

**User entity properties**

| Request Key | Required | Type/Format | Description                  |
| ----------- | -------- | ----------- | ---------------------------- |
| id          | Y        | string      | The unique user identifier.  |
| isGuest     | N        | boolean     | Whether the user is a guest. |
| email       | N        | string      | User email address.          |

_Schema:_ `iglu:com.snowplowanalytics.snowplow.ecommerce/user/jsonschema/1-0-0`.

---

# Track exceptions with the native mobile trackers
> Automatically capture and track unhandled exceptions and crashes within mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/exception-tracking/

Exception tracking captures any unhandled exceptions within the application.

The exception tracking is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .exceptionAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .exceptionAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .exceptionAutotracking(true);
```

***

It allows the tracker to intercept critical exceptions in the app. Exceptions can crash the app so it's likely that the event will be sent after the restart of the app. Being a critical situation we can't be 100% sure that all the exception stacktraces are reliably stored for sending before the crash of the app.

---

# Track consent and GDPR with the native mobile trackers
> Add GDPR consent context entities to all tracked events for legal compliance and privacy management.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/

You can add an [entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/gdpr/jsonschema/1-0-0) detailing the GDPR basis for processing to every event tracked. To configure this, provide a `GdprConfiguration` object when setting up a new tracker.

The only required argument is the `Basis` enum, which contains the six GDPR options. A document describing the legal basis for processing can also be provided (as a string), along with the document version and ID.

**iOS:**

```swift
let gdprConfig = GDPRConfiguration(
  basis: .consent, // basis for processing
  documentId: "id", // document ID
  documentVersion: "1.0", // document version
  documentDescription: "Legal justification" // document
)

let networkConfig = NetworkConfiguration(
  endpoint: "https://collector-url.com",
  method: .post
)

Snowplow.createTracker(
  namespace: "namespace",
  network: networkConfig,
  configurations: [gdprConfig]
)
```

**Android (Kotlin):**

```kotlin
val gdprConfiguration = GdprConfiguration(
    Basis.CONSENT, // basis for processing
    "someId", // document ID
    "0.1.0", // document version
    "Legal justification" // document
)

val networkConfiguration = NetworkConfiguration("https://collector-url.com")

createTracker(
    applicationContext,
    "namespace",
    networkConfiguration,
    gdprConfiguration
)
```

**Android (Java):**

```java
GdprConfiguration gdprConfiguration = new GdprConfiguration(
        Basis.CONSENT, // basis for processing
        "someId", // document ID
        "0.1.0", // document version
        "Legal justification" // document
);

NetworkConfiguration networkConfiguration = new NetworkConfiguration("https://collector-url.com");

Snowplow.createTracker(
        getApplicationContext(),
        "namespace",
        networkConfiguration,
        gdprConfiguration
);
```

***

---

# Track out-of-the-box events with the native mobile trackers
> Track pre-built event types including screen views, lifecycle events, sessions, and custom self-describing events in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/

To track an event, pass an event to the tracker instance.

For example, tracking a ScreenView:

**iOS:**

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")

let event = ScreenView(name: "screen name")
let eventId = tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val tracker = Snowplow.createTracker(
        applicationContext, // Android context
        "appTracker", // namespace
        "https://snowplow-collector-url.com" // Event collector URL
    )

val event = ScreenView("screen")
tracker.track(event)
```

**Android (Java):**

```java
TrackerController tracker = Snowplow.createTracker(
      getApplicationContext(), // Android context
      "appTracker", // namespace
      "https://snowplow-collector-url.com" // Event collector URL
);

ScreenView event = new ScreenView("screen");
tracker.track(event);
```

***

The tracker makes it easy to track different kinds of data. We provide a range of `Event` classes for tracking out-of-the-box event types, as well as fully custom events.

Each event has an associated context, which is composed of entities. The tracker attaches entities to the events based on the configuration, but you can attach your own [custom entities](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) as well.

Every tracked event payload has a unique `event_id` UUID string set by the tracker, a set of timestamps, and other ubiquitous properties such as the `namespace`. The `event_id` is returned from the `tracker.track(event)` method. You can know more about how events and entities are structured [here](/docs/events/).

See the full configuration and parameter options for all these classes and methods in the API docs - [Android](https://snowplow.github.io/snowplow-android-tracker/index.html) and [iOS](https://snowplow.github.io/snowplow-ios-tracker/documentation/snowplowtracker/).

## Auto-tracked events and entities

Automatically captured data are:

- [**Platform and Application Context Tracking**](/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/): Captures contextual information about the device and the app.
- [**Session Tracking**](/docs/sources/mobile-trackers/tracking-events/session-tracking/): Captures the session which helps to keep track of the user activity in the app.
- [**App Lifecycle Tracking**](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/): Captures application lifecycle state changes (foreground/background transitions).
- [**Screen View and Engagement Tracking**](/docs/sources/mobile-trackers/tracking-events/screen-tracking/): Captures each time a new “screen” is loaded.
- [**Exception Tracking**](/docs/sources/mobile-trackers/tracking-events/exception-tracking/): Captures any unhandled exceptions within the application.
- [**Installation Tracking**](/docs/sources/mobile-trackers/tracking-events/installation-tracking/): Captures an install event which occurs the first time an application is opened.
- [**Immersive Space Tracking**](/docs/sources/mobile-trackers/tracking-events/visionos/): iOS only. Captures the visionOS immersive space in which events occur.

Autotracking can be enabled in the tracker configuration. In this example, some helpful automatic entities and all autotracking is enabled:

**iOS:**

```swift
Snowplow.createTracker(namespace: "appTracker", network: networkConfig) {
    TrackerConfiguration()
        .platformContext(true)
        .applicationContext(true)
        .lifecycleAutotracking(true)
        .sessionContext(true)
        .screenViewAutotracking(true)
        .screenEngagementAutotracking(true) // available from v6 of the tracker
        .screenContext(true)
        .exceptionAutotracking(true)
        .installAutotracking(true)
        .immersiveSpaceContext(true)
}
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .platformContext(true)
    .applicationContext(true)
    .lifecycleAutotracking(true)
    .sessionContext(true)
    .screenViewAutotracking(true)
    .screenEngagementAutotracking(true) // available from v6 of the tracker
    .screenContext(true)
    .exceptionAutotracking(true)
    .installAutotracking(true)

Snowplow.createTracker(
    applicationContext,
    namespace,
    networkConfiguration,
    trackerConfig
)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .platformContext(true)
    .applicationContext(true)
    .lifecycleAutotracking(true)
    .sessionContext(true)
    .screenViewAutotracking(true)
    .screenEngagementAutotracking(true) // available from v6 of the tracker
    .screenContext(true)
    .exceptionAutotracking(true)
    .installAutotracking(true);

Snowplow.createTracker(getApplicationContext(),
                namespace,
                networkConfiguration,
                trackerConfig);
```

***

You can also configure your tracker to automatically add [GDPR information](/docs/sources/mobile-trackers/tracking-events/gdpr-tracking/) to every event.

## Manually-tracked events

The tracker provides classes for tracking different types of events. The events are divided into two groups: canonical events and self-describing events.

### Creating a Timing event

Use the `Timing` events to track user timing events such as how long resources take to load.

The event schema is `iglu:com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0`.

**iOS:**

```swift
let event = Timing(category: "timing-category", variable: "timing-variable", timing: 5)
    .label("optional-label")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Timing("timing-category", "timing-variable", 5)
    .label("optional-label")

tracker.track(event)
```

**Android:**

```java
Timing event = new Timing("timing-category", "timing-variable", 5)
    .label("optional-label");

tracker.track(event);
```

***

### Creating a ScreenView event

Track the user viewing a screen within the application. This type of tracking is typically used when automatic [screen view tracking](/docs/sources/mobile-trackers/tracking-events/screen-tracking/) is not suitable within your application.

The event schema is `iglu:com.snowplowanalytics.mobile/screen_view/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ScreenView(name: "DemoScreenName", screenId: UUID())

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ScreenView("DemoScreenName")

tracker.track(event)
```

**Android (Java):**

```java
ScreenView event = new ScreenView("DemoScreenName");

tracker.track(event);
```

***

### Creating a Consent event

#### Consent Granted

Use the `ConsentGranted` event to track a user opting into data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied.

The event schema is `iglu:com.snowplowanalytics.snowplow/consent_granted/jsonschema/1-0-0`. The consent document schema is `iglu:com.snowplowanalytics.snowplow/consent_document/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ConsentGranted(expiry: "2022-01-01T00:00:00Z", documentId: "1234abcd", version: "1.2")
    .name("document-name")
    .documentDescription("document-description")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ConsentGranted("2022-01-01T00:00:00Z", "1234abcd", "1.2")
    .documentDescription("document-description")
    .documentName("document-name")

tracker.track(event)
```

**Android (Java):**

```java
ConsentGranted event = new ConsentGranted("2022-01-01T00:00:00Z", "1234abcd", "1.2")
        .documentDescription("document-description")
        .documentName("document-name");

tracker.track(event);
```

***

#### Consent Withdrawn

Use the `ConsentWithdrawn` event to track a user withdrawing consent for data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied. To specify that a user opts out of all data collection, `all` should be set to `true`.

The event schema is `iglu:com.snowplowanalytics.snowplow/consent_withdrawn/jsonschema/1-0-0`. The consent document schema is `iglu:com.snowplowanalytics.snowplow/consent_document/jsonschema/1-0-0`.

**iOS:**

```swift
let event = ConsentWithdrawn()
    .all(true)
    .documentId("1234abcd")
    .version("1.2")
    .name("document-name")
    .documentDescription("document-description")

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = ConsentWithdrawn(false, "1234abcd", "1.2")
    .documentDescription("document-description")
    .documentName("document-name")

tracker.track(event)
```

**Android (Java):**

```java
ConsentWithdrawn event = new ConsentWithdrawn(false, "1234abcd", "1.2")
        .documentDescription("document-description")
        .documentName("document-name");

tracker.track(event);
```

***

### Tracking Push and Local Notifications

To track an event when a push (or local) notification is used, it is possible to use the `MessageNotification` event.

The event schema is `iglu:com.snowplowanalytics.mobile/message_notification/jsonschema/1-0-0`.

**iOS:**

```swift
let event = MessageNotification(title: "title", body: "body", trigger: .push)
    .notificationTimestamp("2020-12-31T15:59:60-08:00")
    .action("action")
    .bodyLocKey("loc key")
    .bodyLocArgs(["loc arg1", "loc arg2"])
    .sound("chime.mp3")
    .notificationCount(9)
    .category("category1")
    .attachments([
        MessageNotificationAttachment(identifier: "id", type: "type", url: "https://snowplow.io")
    ]);
tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = MessageNotification("title", "body", MessageNotificationTrigger.push)
    .notificationTimestamp("2021-10-18T10:16:08.008Z")
    .category("category")
    .action("action")
    .bodyLocKey("loc key")
    .bodyLocArgs(listOf("loc arg1", "loc arg2"))
    .sound("chime.mp3")
    .notificationCount(9)
    .category("category1")
tracker.track(event)
```

**Android (Java):**

```java
MessageNotification event = new MessageNotification("title", "body", MessageNotificationTrigger.push)
        .notificationTimestamp("2021-10-18T10:16:08.008Z")
        .category("category")
        .action("action")
        .bodyLocKey("loc key")
        .bodyLocArgs(Arrays.asList("loc arg1", "loc arg2"))
        .sound("chime.mp3")
        .notificationCount(9)
        .category("category1");
tracker.track(event);
```

***

### Tracking Deep Links

The Deep Link is received by the mobile operating system and passed to the related app. Our mobile tracker can't automatically track the Deep Link, but we provide an out-of-the-box event that can be used by the developer to manually track it as soon as the Deep Link is received in the app.

The event schema is `iglu:com.snowplowanalytics.mobile/deep_link_received/jsonschema/1-0-0`.

It will be the duty of the tracker to automatically attach the information of the Deep Link to the first ScreenView tracked.

In practice, when the app receives a Deep Link, the developer can track it through the DeepLinkReceived event:

**iOS:**

```swift
public func application(_ application: UIApplication,
                        continue userActivity: NSUserActivity,
                        restorationHandler: @escaping ([Any]?) -> Void) -> Bool
{
    ...
    if let url = userActivity.webpageURL {
        let deepLinkEvent = DeepLinkReceived(url: userActivity.webpageURL.absoluteString)
            .referrer(userActivity.referrerURL.absoluteString)
        tracker.track(deepLinkEvent)
    }
    ...
}
```

**Android (Kotlin):**

```kotlin
override fun onCreate(savedInstanceState: Bundle?) {
    ...

    // Extract info from Intent
    val deepLinkUrl = intent.data.toString()
    var referrer: String? = null
    val extras = intent.extras
    if (extras != null) {
        val referrerUri = extras[Intent.EXTRA_REFERRER] as? Uri
        if (referrerUri != null) {
            referrer = referrerUri.toString()
        }
    }
    // Create and track the event
    val event = DeepLinkReceived(deepLinkUrl).referrer(referrer)
    tracker.track(event)

    ...
}
```

**Android (Java):**

```java
@Override
public void onCreate(Bundle savedInstanceState) {
    ...

    // Extract info from Intent
    Intent intent = getIntent();
    String deepLinkUrl = intent.getData().toString();
    String referrer = null;
    Bundle extras = intent.getExtras();
    if (extras != null) {
        Uri referrerUri = extras.get(Intent.EXTRA_REFERRER);
        if (referrerUri != null) {
            referrer = referrerUri.toString();
        }
    }
    // Create and track the event
    DeepLinkReceived event = new DeepLinkReceived(deepLinkUrl).referrer(referrer);
    tracker.track(event);

    ...
}
```

***

The tracker keeps memory of the tracked Deep Link event and will attach a Deep Link entity to the first ScreenView tracked in the tracker. This is helpful during the analysis of the data because it will be clear the relation between the content visualized by the user (ScreenView event) and source (DeepLink entity) that originated that visualisation.

This behavior is enabled by default but it can be disabled from the `TrackerConfiguration`.

For example:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    ...
    .deepLinkContext(false)
    ...
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    ...
    .deepLinkContext(false)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    ...
    .deepLinkContext(false);
```

***

The `DeepLinkReceived` event can be used in pair with a `campaign-attribution-enrichment` appropriately enabled in the Snowplow pipeline. It works exactly like for `PageView` events in the web/JS tracker. When the user taps on an advertising banner or a marketing email or message, it can trigger the launch of the app through the Deep Linking feature. The referral from the advertising campaigns, websites, or other source can be composed by UTM parameters used to attribute the user activity back to the campaign. The Campaign Attribution Enrichment can parse the DeepLinkReceived event extracting the UTM parameters in the deep link url.

The tracked `DeepLinkReceived` event and each subsequent `ScreenView` event also have the URL and referrer information added in the `page_url` and `page_referrer` atomic properties. This makes them compatible with data models and enrichments built for events tracked on the Web.

### Creating a Structured event

Our philosophy in creating Snowplow is that users should design suitable data structures customised for their own consumer interactions data capture. You can read more about that philosophy [here](/docs/event-studio/).

However, as part of a Snowplow implementation there may be interactions where [custom `SelfDescribing` events](/docs/sources/mobile-trackers/custom-tracking-using-schemas/) are unwarranted. Those are candidates to track as `Structured` events. We do strongly recommend using fully custom `SelfDescribing` events.

**iOS:**

```swift
let event = Structured(category: "Example", action: "my-action")
    .label("my-label")
    .property("my-property")
    .value(5)

tracker.track(event)
```

**Android (Kotlin):**

```kotlin
val event = Structured("shop", "add-to-basket") // category and action
    .label("Add To Basket")
    .property("pcs")
    .value(2.0)
tracker.track(event)
```

**Android (Java):**

```java
Structured event = new Structured("shop", "add-to-basket") // category and action
    .label("Add To Basket")
    .property("pcs")
    .value(2);
tracker.track(event);
```

***

## Tracking data that is not event-type specific

Some data, such as that relating to the user whose activity is being tracked, is relevant across all event types. The tracker provides two mechanisms for tracking this kind of data.

Certain properties, including `userId` or `ipAddress`, can be set as "atomic" properties in the raw event, using the [`Subject` class](/docs/sources/mobile-trackers/client-side-properties/).

A more general and powerful method is to attach self-describing JSON "context entities" to your events - the same JSON schemas as used for self-describing events. This means that any data that can be described by a JSON schema can be added to any or all of your events. Read more about custom tracking [here](/docs/sources/mobile-trackers/custom-tracking-using-schemas/).

All events also provide the option for setting a custom timestamp, called `trueTimestamp`. See below for details.

### Adding custom timestamps to events

Snowplow events have several timestamps. The raw event payload always contains a `deviceCreatedTimestamp` (`dtm`) and a `deviceSentTimestamp` (`stm`). Other timestamps are added as the event moves through the pipeline.

Every `Event` in the tracker allows for a custom timestamp, called `trueTimestamp` to be set. A `trueTimestamp` can be added to any event using the `trueTimestamp()` method:

**iOS:**

```swift
// This example shows a self-describing event, but all events can have a trueTimestamp
let event = SelfDescribing(schema: "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", payload: data)
    .trueTimestamp(Date(timeIntervalSince1970: 166184300))
```

`trueTimestamp` should be a `Date` object.

**Android (Kotlin):**

```kotlin
// This example shows a self-describing event, but all events can have a trueTimestamp
val event = SelfDescribing(SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data))
    .trueTimestamp(166184300L)
```

**Android (Java):**

```java
// This example shows a self-describing event, but all events can have a trueTimestamp
SelfDescribing event = new SelfDescribing(new SelfDescribingJson("iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", data))
    .trueTimestamp(166184300L);
```

***

---

# Track application installs with the native mobile trackers
> Track first-time app installations and capture referrer information from Google Play Store for Android apps.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/installation-tracking/

Installation tracking tracks an install event which occurs the first time an application is opened. The tracker will record when it's first been installed, so deleting and reinstalling an app will trigger another install event.

If installation autotracking is not enabled, the tracker will still keep track of when the app was first installed, so that when enabled, the tracker will send the recorded install event with a timestamp reflecting when it was first installed.

The installation autotracking is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .installAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .installAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .installAutotracking(true);
```

***

## Android-only: Tracking referrer information from the Google Play Referrer library

The Google Play Referrer library is a tool provided by Google for Android developers to track and capture referral information when a user installs or updates an app from the Google Play Store. It allows developers to gather valuable insights about the sources that drive app installations, such as ad campaigns, referral links, or marketing efforts.

When an app is installed or updated, the Google Play Referrer library retrieves the referral information from the Google Play Store and provides it to the app. This information includes the referrer URL, which can contain parameters or tracking codes that identify the specific referral source.

The Android tracker can make use of the library to retrieve the referral information. It attaches the information in an entity attached to the install event. The entity uses the `iglu:com.android.installreferrer.api/referrer_details/jsonschema/1-0-0` schema with the following properties:

| Property                 | Type     | Description                                                                                        |
| ------------------------ | -------- | -------------------------------------------------------------------------------------------------- |
| `installReferrer`        | String   | The referrer URL of the installed package                                                          |
| `referrerClickTimestamp` | Datetime | The timestamp when referrer click happens                                                          |
| `installBeginTimestamp`  | Datetime | The timestamp when installation begins                                                             |
| `googlePlayInstantParam` | Boolean  | Boolean indicating if the user has interacted with the app's instant experience in the past 7 days |

To enable tracking the entity, you will need to have `installAutotracking` enabled (it's on by default) and add the following line to the dependencies section of the `build.gradle` file in your app:

```gradle
dependencies {
    ...
    implementation "com.android.installreferrer:installreferrer:2.2"
}
```

---

# Track application lifecycle changes with the native mobile trackers
> Automatically track application foreground and background transitions to understand app usage patterns.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/

The tracker can capture application lifecycle state changes. In particular, when the app changes state from foreground to background and vice versa.

The lifecycle tracking is enabled by default (since v6.0.0). It can be configured in `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .lifecycleAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .lifecycleAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .lifecycleAutotracking(true);
```

***

Once enabled, the tracker will automatically track a [`Background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event) when the app is moved to background and a [`Foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event) when the app moves back to foreground (becomes visible in the screen).

The tracker attaches a [`LifecycleEntity`](/docs/events/ootb-data/mobile-lifecycle-events/#lifecycle-entity) to all the events tracked by the tracker reporting if the app was visible (foreground state) when the event was tracked.

The `LifecycleEntity` value is conditioned by the internal state of the tracker only. To make an example, if the app is in foreground state but the developer tracks a `Background` event intentionally, it would force the generation of a `LifecycleEntity` that mark the app as non visible, even if it's actually visible in the device.

---

# Track media events with the native mobile trackers
> Track media playback events including play, pause, buffering, and completion using the mobile media tracking plugin.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/media-tracking/

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

**iOS:**

```swift
// 1. Start tracking when player loads
let id = UUID().uuidString
let player = MediaPlayerEntity()
  .duration(300)
  .label("My Video Title")
  .mediaType(.video)

let mediaTracking = tracker.media.startMediaTracking(id: id, player: player)

// 2. Update player state every second (or use AVPlayer auto-tracking)
mediaTracking.update(player: MediaPlayerEntity().currentTime(currentTime))

// 3. Track events as they occur
mediaTracking.track(MediaPlayEvent())
mediaTracking.track(MediaPauseEvent())
mediaTracking.track(MediaSeekStartEvent())
mediaTracking.track(MediaSeekEndEvent())
mediaTracking.track(MediaEndEvent())

// 4. End tracking when done
tracker.media.endMediaTracking(id: id)
```

**Android (Kotlin):**

```kotlin
// 1. Start tracking when player loads
val id = UUID.randomUUID().toString()
val player = MediaPlayerEntity(
  duration = 300.0,
  label = "My Video Title",
  mediaType = MediaType.Video
)
val mediaTracking = tracker?.media?.startMediaTracking(id, player)

// 2. Update player state every second
mediaTracking?.update(player = MediaPlayerEntity(currentTime = currentTime))

// 3. Track events as they occur
mediaTracking?.track(MediaPlayEvent())
mediaTracking?.track(MediaPauseEvent())
mediaTracking?.track(MediaSeekStartEvent())
mediaTracking?.track(MediaSeekEndEvent())
mediaTracking?.track(MediaEndEvent())

// 4. End tracking when done
tracker?.media?.endMediaTracking(id)
```

**Android (Java):**

```java
// 1. Start tracking when player loads
String id = UUID.randomUUID().toString();
MediaPlayerEntity player = new MediaPlayerEntity();
player.setDuration(300.0);
player.setLabel("My Video Title");
player.setMediaType(MediaType.Video);

TrackerController tracker = Snowplow.getDefaultTracker();
MediaTracking mediaTracking = tracker.getMedia().startMediaTracking(id, player);

// 2. Update player state every second
MediaPlayerEntity update = new MediaPlayerEntity();
update.setCurrentTime(currentTime);
mediaTracking.update(update, null, null);

// 3. Track events as they occur
mediaTracking.track(new MediaPlayEvent(), null, null, null);
mediaTracking.track(new MediaPauseEvent(), null, null, null);
mediaTracking.track(new MediaSeekStartEvent(), null, null, null);
mediaTracking.track(new MediaSeekEndEvent(), null, null, null);
mediaTracking.track(new MediaEndEvent(), null, null, null);

// 4. End tracking when done
tracker.getMedia().endMediaTracking(id);
```

***

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

**iOS:**

```swift
let player = MediaPlayerEntity()
  .currentTime(0)          // Current playback position in seconds
  .duration(300)           // Total duration in seconds
  .ended(false)            // Whether playback has ended
  .livestream(false)       // Whether this is a live stream
  .label("My Video")       // Human-readable title (recommended)
  .loop(false)             // Whether to restart after ending
  .mediaType(.video)       // .video or .audio
  .muted(false)            // Whether audio is muted
  .paused(true)            // Whether playback is paused
  .pictureInPicture(false) // Whether in picture-in-picture mode
  .playbackRate(1.0)       // Playback speed (1 = normal)
  .quality("1080p")        // Quality level
  .volume(100)             // Volume percentage (0-100)

let mediaTracking = tracker.media.startMediaTracking(id: id, player: player)
```

**Android (Kotlin):**

```kotlin
val player = MediaPlayerEntity(
  currentTime = 0.0,        // Current playback position in seconds
  duration = 300.0,         // Total duration in seconds
  ended = false,            // Whether playback has ended
  livestream = false,       // Whether this is a live stream
  label = "My Video",       // Human-readable title (recommended)
  loop = false,             // Whether to restart after ending
  mediaType = MediaType.Video, // MediaType.Video or MediaType.Audio
  muted = false,            // Whether audio is muted
  paused = true,            // Whether playback is paused
  pictureInPicture = false, // Whether in picture-in-picture mode
  playbackRate = 1.0,       // Playback speed (1 = normal)
  quality = "1080p",        // Quality level
  volume = 100              // Volume percentage (0-100)
)

val mediaTracking = tracker?.media?.startMediaTracking(id, player)
```

**Android (Java):**

```java
MediaPlayerEntity player = new MediaPlayerEntity();
player.setCurrentTime(0.0);          // Current playback position in seconds
player.setDuration(300.0);           // Total duration in seconds
player.setEnded(false);              // Whether playback has ended
player.setLivestream(false);         // Whether this is a live stream
player.setLabel("My Video");         // Human-readable title (recommended)
player.setLoop(false);               // Whether to restart after ending
player.setMediaType(MediaType.Video); // MediaType.Video or MediaType.Audio
player.setMuted(false);              // Whether audio is muted
player.setPaused(true);              // Whether playback is paused
player.setPictureInPicture(false);   // Whether in picture-in-picture mode
player.setPlaybackRate(1.0);         // Playback speed (1 = normal)
player.setQuality("1080p");          // Quality level
player.setVolume(100);               // Volume percentage (0-100)

MediaTracking mediaTracking = tracker.getMedia().startMediaTracking(id, player);
```

***

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)
  .pingInterval(30)   // Seconds between pings (default: 30)
  .maxPausedPings(1)  // Max pings while paused (default: 1)

// Or turn off pings entirely
let configuration = MediaTrackingConfiguration(id: id)
  .pings(false)
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(
  id = id,
  pingInterval = 30,   // Seconds between pings (default: 30)
  maxPausedPings = 1   // Max pings while paused (default: 1)
)

// Or turn off pings entirely
val configuration = MediaTrackingConfiguration(id = id, pings = false)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);
configuration.setPingInterval(30);   // Seconds between pings (default: 30)
configuration.setMaxPausedPings(1);  // Max pings while paused (default: 1)

// Or turn off pings entirely
configuration.setPings(false);
```

***

### Percentage progress

Track events when playback reaches specified percentage boundaries.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)
  .boundaries([10, 25, 50, 75, 90])  // Fire events at these percentages
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(
  id = id,
  boundaries = listOf(10, 25, 50, 75, 90)  // Fire events at these percentages
)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);
configuration.setBoundaries(Arrays.asList(10, 25, 50, 75, 90));  // Fire events at these percentages
```

***

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id).session(true)
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(id = id, session = true)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);
configuration.setSession(true);
```

***

### Custom entities

You can attach custom entities to all events for this media tracking instance.

**iOS:**

```swift
let configuration = MediaTrackingConfiguration(id: id)
  .entities([
      SelfDescribingJson(
          schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",
          andData: ["contentId": "abc123", "category": "tutorial"]
      )
  ])
```

**Android (Kotlin):**

```kotlin
val configuration = MediaTrackingConfiguration(
  id = id,
  entities = listOf(
      SelfDescribingJson(
          schema = "iglu:com.example/video_metadata/jsonschema/1-0-0",
          data = mapOf("contentId" to "abc123", "category" to "tutorial")
      )
  )
)
```

**Android (Java):**

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);
configuration.setEntities(Collections.singletonList(
  new SelfDescribingJson(
      "iglu:com.example/video_metadata/jsonschema/1-0-0",
      new HashMap<String, Object>() {{
          put("contentId", "abc123");
          put("category", "tutorial");
      }}
  )
));
```

***

### Filter events

**iOS:**

Only track specific event types by providing an allowlist:

```swift
let configuration = MediaTrackingConfiguration(id: id)
  .captureEvents([MediaPlayEvent.self, MediaPauseEvent.self, MediaEndEvent.self])
```

**Android (Kotlin):**

Only track specific event types by providing an allowlist:

```kotlin
val configuration = MediaTrackingConfiguration(
  id = id,
  captureEvents = listOf(MediaPlayEvent::class, MediaPauseEvent::class, MediaEndEvent::class)
)
```

**Android (Java):**

Only track specific event types by providing an allowlist:

```java
MediaTrackingConfiguration configuration = new MediaTrackingConfiguration(id, null);
configuration.setCaptureEvents(Arrays.asList(
  MediaPlayEvent.class, MediaPauseEvent.class, MediaEndEvent.class
));
```

***

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

**iOS:**

```swift
mediaTracking.track(MediaReadyEvent())        // Player is ready
mediaTracking.track(MediaPlayEvent())         // Playback started
mediaTracking.track(MediaPauseEvent())        // Playback paused
mediaTracking.track(MediaEndEvent())          // Playback ended
mediaTracking.track(MediaSeekStartEvent())    // Seek operation started
mediaTracking.track(MediaSeekEndEvent())      // Seek operation ended
mediaTracking.track(MediaBufferStartEvent())  // Buffering started
mediaTracking.track(MediaBufferEndEvent())    // Buffering ended
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(MediaReadyEvent())        // Player is ready
mediaTracking?.track(MediaPlayEvent())         // Playback started
mediaTracking?.track(MediaPauseEvent())        // Playback paused
mediaTracking?.track(MediaEndEvent())          // Playback ended
mediaTracking?.track(MediaSeekStartEvent())    // Seek operation started
mediaTracking?.track(MediaSeekEndEvent())      // Seek operation ended
mediaTracking?.track(MediaBufferStartEvent())  // Buffering started
mediaTracking?.track(MediaBufferEndEvent())    // Buffering ended
```

**Android (Java):**

```java
mediaTracking.track(new MediaReadyEvent(), null, null, null);        // Player is ready
mediaTracking.track(new MediaPlayEvent(), null, null, null);         // Playback started
mediaTracking.track(new MediaPauseEvent(), null, null, null);        // Playback paused
mediaTracking.track(new MediaEndEvent(), null, null, null);          // Playback ended
mediaTracking.track(new MediaSeekStartEvent(), null, null, null);    // Seek operation started
mediaTracking.track(new MediaSeekEndEvent(), null, null, null);      // Seek operation ended
mediaTracking.track(new MediaBufferStartEvent(), null, null, null);  // Buffering started
mediaTracking.track(new MediaBufferEndEvent(), null, null, null);    // Buffering ended
```

***

### Player state events

**iOS:**

```swift
mediaTracking.track(MediaPlaybackRateChangeEvent(newRate: 1.5))
mediaTracking.track(MediaVolumeChangeEvent(newVolume: 80))
mediaTracking.track(MediaFullscreenChangeEvent(fullscreen: true))
mediaTracking.track(MediaPictureInPictureChangeEvent(pictureInPicture: true))
mediaTracking.track(MediaQualityChangeEvent(
  newQuality: "1080p",
  bitrate: 5000,
  framesPerSecond: 30,
  automatic: true
))
mediaTracking.track(MediaErrorEvent(
  errorCode: "500",
  errorName: "NetworkError",
  errorDescription: "Failed to load media"
))
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(MediaPlaybackRateChangeEvent(newRate = 1.5))
mediaTracking?.track(MediaVolumeChangeEvent(newVolume = 80))
mediaTracking?.track(MediaFullscreenChangeEvent(fullscreen = true))
mediaTracking?.track(MediaPictureInPictureChangeEvent(pictureInPicture = true))
mediaTracking?.track(MediaQualityChangeEvent(
  newQuality = "1080p",
  bitrate = 5000,
  framesPerSecond = 30,
  automatic = true
))
mediaTracking?.track(MediaErrorEvent(
  errorCode = "500",
  errorName = "NetworkError",
  errorDescription = "Failed to load media"
))
```

**Android (Java):**

```java
mediaTracking.track(new MediaPlaybackRateChangeEvent(null, 1.5), null, null, null);
mediaTracking.track(new MediaVolumeChangeEvent(null, 80), null, null, null);
mediaTracking.track(new MediaFullscreenChangeEvent(true), null, null, null);
mediaTracking.track(new MediaPictureInPictureChangeEvent(true), null, null, null);

MediaQualityChangeEvent qualityEvent = new MediaQualityChangeEvent();
qualityEvent.setNewQuality("1080p");
qualityEvent.setBitrate(5000);
qualityEvent.setFramesPerSecond(30);
qualityEvent.setAutomatic(true);
mediaTracking.track(qualityEvent, null, null, null);

MediaErrorEvent errorEvent = new MediaErrorEvent();
errorEvent.setErrorCode("500");
errorEvent.setErrorName("NetworkError");
errorEvent.setErrorDescription("Failed to load media");
mediaTracking.track(errorEvent, null, null, null);
```

***

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

**iOS:**

```swift
// Ad break events
let adBreak = MediaAdBreakEntity(breakId: "break-1")
  .name("pre-roll")
  .breakType(.linear)
  .podSize(3)
mediaTracking.track(MediaAdBreakStartEvent(), adBreak: adBreak)
mediaTracking.track(MediaAdBreakEndEvent())

// Ad events
let ad = MediaAdEntity(adId: "ad-1")
  .name("Product Ad")
  .creativeId("creative-1")
  .duration(30)
  .skippable(true)
mediaTracking.track(MediaAdStartEvent(), ad: ad)
mediaTracking.track(MediaAdFirstQuartileEvent())
mediaTracking.track(MediaAdMidpointEvent())
mediaTracking.track(MediaAdThirdQuartileEvent())
mediaTracking.track(MediaAdCompleteEvent())
mediaTracking.track(MediaAdSkipEvent())
mediaTracking.track(MediaAdClickEvent())
mediaTracking.track(MediaAdPauseEvent())
mediaTracking.track(MediaAdResumeEvent())
```

**Android (Kotlin):**

```kotlin
// Ad break events
val adBreak = MediaAdBreakEntity(
  breakId = "break-1",
  name = "pre-roll",
  breakType = MediaAdBreakType.Linear,
  podSize = 3
)
mediaTracking?.track(MediaAdBreakStartEvent(), adBreak = adBreak)
mediaTracking?.track(MediaAdBreakEndEvent())

// Ad events
val ad = MediaAdEntity(
  adId = "ad-1",
  name = "Product Ad",
  creativeId = "creative-1",
  duration = 30.0,
  skippable = true
)
mediaTracking?.track(MediaAdStartEvent(), ad = ad)
mediaTracking?.track(MediaAdFirstQuartileEvent())
mediaTracking?.track(MediaAdMidpointEvent())
mediaTracking?.track(MediaAdThirdQuartileEvent())
mediaTracking?.track(MediaAdCompleteEvent())
mediaTracking?.track(MediaAdSkipEvent())
mediaTracking?.track(MediaAdClickEvent())
mediaTracking?.track(MediaAdPauseEvent())
mediaTracking?.track(MediaAdResumeEvent())
```

**Android (Java):**

```java
// Ad break events
MediaAdBreakEntity adBreak = new MediaAdBreakEntity("break-1");
adBreak.setName("pre-roll");
adBreak.setBreakType(MediaAdBreakType.Linear);
adBreak.setPodSize(3);
mediaTracking.track(new MediaAdBreakStartEvent(), null, null, adBreak);
mediaTracking.track(new MediaAdBreakEndEvent(), null, null, null);

// Ad events
MediaAdEntity ad = new MediaAdEntity("ad-1");
ad.setName("Product Ad");
ad.setCreativeId("creative-1");
ad.setDuration(30.0);
ad.setSkippable(true);
mediaTracking.track(new MediaAdStartEvent(), null, ad, null);
mediaTracking.track(new MediaAdFirstQuartileEvent(), null, null, null);
mediaTracking.track(new MediaAdMidpointEvent(), null, null, null);
mediaTracking.track(new MediaAdThirdQuartileEvent(), null, null, null);
mediaTracking.track(new MediaAdCompleteEvent(), null, null, null);
mediaTracking.track(new MediaAdSkipEvent(), null, null, null);
mediaTracking.track(new MediaAdClickEvent(), null, null, null);
mediaTracking.track(new MediaAdPauseEvent(), null, null, null);
mediaTracking.track(new MediaAdResumeEvent(), null, null, null);
```

***

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

**iOS:**

```swift
mediaTracking.track(SelfDescribing(
  schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",
  payload: ["action": "share", "platform": "twitter"]
))
```

**Android (Kotlin):**

```kotlin
mediaTracking?.track(SelfDescribing(
  "iglu:com.example/video_interaction/jsonschema/1-0-0",
  mapOf("action" to "share", "platform" to "twitter")
))
```

**Android (Java):**

```java
mediaTracking.track(new SelfDescribing(
  "iglu:com.example/video_interaction/jsonschema/1-0-0",
  new HashMap<String, Object>() {{
      put("action", "share");
      put("platform", "twitter");
  }}
), null, null, null);
```

***

## Update player state

**iOS:**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```swift
mediaTracking.update(player: MediaPlayerEntity().currentTime(currentTime))
```

**Android (Kotlin):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```kotlin
mediaTracking?.update(player = MediaPlayerEntity(currentTime = currentTime))
```

**Android (Java):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```java
MediaPlayerEntity player = new MediaPlayerEntity();
player.setCurrentTime(currentTime);
mediaTracking.update(player, null, null);
```

***

## AVPlayer auto-tracking (iOS)

The iOS tracker can automatically track events from [AVPlayer](https://developer.apple.com/documentation/avfoundation/avplayer). Pass the player instance to `startMediaTracking`:

```swift
let mediaTracking = tracker.media.startMediaTracking(
    player: avPlayer,
    configuration: MediaTrackingConfiguration(id: "my-video")
)
```

The following events are auto-tracked:

- Play events
- Pause events
- Seek end events
- Ping events
- Percent progress events (if configured)
- Buffer start events
- End events
- Error events

---

# Track platform and application data with the native mobile trackers
> Automatically capture device and application context including OS version, device model, battery level, and advertising identifiers.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/platform-and-application-context/

Platform and application data tracking features capture information about the device and the app.

They are enabled by default. But the setting can be changed through `TrackerConfiguration` like in the example below:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .platformContext(true)
    .applicationContext(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .platformContext(true)
    .applicationContext(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .platformContext(true)
    .applicationContext(true);
```

***

## Application context

The [application context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.mobile/application/jsonschema/1-0-0) contains two properties:

| Property  | Type   | Description                                             | Required in schema |
| --------- | ------ | ------------------------------------------------------- | ------------------ |
| `version` | String | Version number of the application e.g 1.1.0             | Yes                |
| `build`   | String | Build name of the application e.g s9f2k2d or 1.1.0 beta | Yes                |

## Platform (mobile) context

The [platform context entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-3) contains the following properties:

| Property                | Type                                 | Android | iOS | Description                                                                                                                                                             | Required in schema |
| ----------------------- | ------------------------------------ | ------- | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `osType`                | String                               | ✅       | ✅   | Type of the operating system (e.g., "ios", "tvos", "watchos", "osx", "android")                                                                                         | Yes                |
| `osVersion`             | String                               | ✅       | ✅   | Version of the mobile operating system                                                                                                                                  | Yes                |
| `deviceManufacturer`    | String                               | ✅       | ✅   | Device vendor                                                                                                                                                           | Yes                |
| `deviceModel`           | String                               | ✅       | ✅   | Model of the device                                                                                                                                                     | Yes                |
| `carrier`               | String                               | ✅       | ✅   | Carrier of the SIM inserted in the device                                                                                                                               | No                 |
| `networkType`           | String                               | ✅       | ✅   | One of: "mobile", "wifi", "offline"                                                                                                                                     | No                 |
| `networkTechnology`     | String                               | ✅       | ✅   | Radio access technology that the device is using                                                                                                                        | No                 |
| `openIdfa`              | String                               | ❌       | ❌   | Deprecated property                                                                                                                                                     | No                 |
| `appleIdfa`             | String                               | ❌       | ✅   | Advertising identifier on iOS                                                                                                                                           | No                 |
| `appleIdfv`             | String                               | ❌       | ✅   | UUID [identifier for vendors](https://developer.apple.com/documentation/uikit/uidevice/1620059-identifierforvendor) on iOS                                              | No                 |
| `androidIdfa`           | String                               | ✅       | ❌   | Advertising identifier on Android                                                                                                                                       | No                 |
| `physicalMemory`        | Integer                              | ✅       | ✅   | Total physical system memory in bytes                                                                                                                                   | No                 |
| `systemAvailableMemory` | Integer                              | ✅       | ❌   | Available memory on the system in bytes                                                                                                                                 | No                 |
| `appAvailableMemory`    | Integer                              | ❌       | ❌   | Amount of memory in bytes available to the current app                                                                                                                  | No                 |
| `batteryLevel`          | Integer                              | ✅       | ✅   | Remaining battery level as an integer percentage of total battery capacity                                                                                              | No                 |
| `batteryState`          | String                               | ✅       | ✅   | Battery state for the device. One of: "unplugged", "charging", "full"                                                                                                   | No                 |
| `lowPowerMode`          | Boolean                              | ❌       | ✅   | A Boolean indicating whether Low Power Mode is enabled                                                                                                                  | No                 |
| `availableStorage`      | Integer                              | ✅       | ❌   | Bytes of storage remaining                                                                                                                                              | No                 |
| `totalStorage`          | Integer                              | ✅       | ❌   | Total size of storage in bytes                                                                                                                                          | No                 |
| `isPortrait`            | Boolean                              | ✅       | ✅   | A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                         | No                 |
| `resolution`            | String                               | ✅       | ✅   | Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes. See note below.                      | No                 |
| `scale`                 | Number                               | ✅       | ✅   | Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android)                 | No                 |
| `language`              | String                               | ✅       | ✅   | System language currently used on the device (ISO 639)                                                                                                                  | No                 |
| `appSetId`              | String                               | ✅       | ❌   | Android vendor ID scoped to the set of apps published under the same Google Play developer account (see <https://developer.android.com/training/articles/app-set-id>)   | No                 |
| `appSetIdScope`         | String (either "app" or "developer") | ✅       | ❌   | Scope of the `appSetId`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) | No                 |

> **Note:** The screen resolution for the platform entity is obtained from the Android context resources. The height value will likely be lower than that reported in the canonical `dvce_screenheight` [event property](/docs/fundamentals/canonical-event/#device-and-operating-system-fields), which is fetched from `WindowManager`, an older API that still includes the menu bar.
>
> To standardize the screen resolution between event and entity properties, provide a `SubjectConfiguration` with `useContextResourcesScreenResolution(true)` flag at tracker initialization. This flag is false by default, and available from Android tracker v6.0.3 onwards. Read about configuring `Subject` properties [here](/docs/sources/mobile-trackers/client-side-properties/).

### Choosing which properties to track

You can choose which properties should be added to the platform context entity. By default, all available properties are tracked in the entity. In case you don't want certain properties to be tracked, you can choose the ones to track using `TrackerConfiguration.platformContextEntities`:

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .platformContextProperties([.batteryLevel, .isPortrait, .language])
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
     .platformContextProperties(listOf(PlatformContextProperty.BATTERY_LEVEL, PlatformContextProperty.IS_PORTRAIT))
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .platformContextProperties(Collections.singletonList(PlatformContextProperty.BATTERY_LEVEL, PlatformContextProperty.IS_PORTRAIT));
```

***

### Overriding platform context properties

In case you want to override the values for certain properties of the platform context (or provide ones that are not tracked by default, such as the `totalStorage` on iOS), you can set the `platformContextRetriever` in `TrackerConfiguration` to provide these values.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .platformContextRetriever(
        PlatformContextRetriever(deviceVendor: { "my-custom-vendor" })
    )
```

**Android (Kotlin):**

```kotlin
val trackerConfiguration = TrackerConfiguration("appId")
    .platformContextRetriever(
        PlatformContextRetriever(deviceVendor = { "my-custom-vendor" })
    )
```

**Android (Java):**

```java
PlatformContextRetriever platformContextRetriever = new PlatformContextRetriever();
platformContextRetriever.setDeviceVendor(() -> "my-custom-vendor");
TrackerConfiguration trackerConfiguration = new TrackerConfiguration("appId")
    .platformContextRetriever(platformContextRetriever);
```

***

### Identifier for Advertisers (IDFA/AAID)

The IDFA advertising identifiers are only added to the platform context if you fulfill the following requirements. Otherwise, their values will be NULL.

**iOS:**

The Apple advertising identifier is stored in the `appleIdfa` property.

Starting with iOS 14, one has to request user consent through the [App Tracking Transparency](https://developer.apple.com/documentation/apptrackingtransparency) framework in order to access the advertising identifier.

1. Add and follow the guidelines of [App Tracking Transparency Framework](https://developer.apple.com/documentation/apptrackingtransparency) in your app.
2. Pass a callback to your `TrackerConfiguration` that retrieves the identifier:

```swift
import AdSupport

let tracker = Snowplow.createTracker(namespace: "ns", network: networkConfig) {
    TrackerConfiguration()
        .advertisingIdentifierRetriever {
            ASIdentifierManager.shared().advertisingIdentifier
        }
}
```

> **Note:** The simulators can’t generate a proper IDFA, instead they generate a sequence of zeros. If you want to test IDFA with a real code, please use the physical device.
>
> The user has the ability to limit ad-tracking from the device’s Settings. If the user enable the limitations the tracker will not be able to track the IDFA.

**Android:**

The AAID (Android Advertising ID) is a unique user-resettable identifier, which uniquely identifies a particular user for advertising use cases, such as ad personalization. The tracker allows retrieval of the AAID, sending it as property `androidIdfa`.

For privacy purposes the user can reset the identifier at any moment. In that case the tracker will report a new AAID, despite the device and user being the same as before. Also, the user can "Opt out of Ads Personalisation" from the Android settings menu. In that case the tracker will report an empty string in place of the AAID.

If you want to track the AAID, you need to add the Google Mobile Ads library to your app. If it isn’t included, the tracker will not send the AAID with the events.

The Google Mobile Ads can be imported in the `dependencies` section of the `build.gradle` adding:

```gradle
dependencies {
    ...
    implementation 'com.google.android.gms:play-services-ads:22.6.0'
    ...
}
```

The Google Mobile Ads SDK v.17.0.0 introduced some [changes](https://ads-developers.googleblog.com/2018/10/announcing-v1700-of-android-google.html) requiring a tag in the `androidManifest.xml` explained below.

#### Manifest tag for AdMob publishers

AdMob publishers have to add the AdMob app ID in the `AndroidManifest.xml` file:

```xml
<manifest>
    <application>
        <!-- TODO: Replace with your real AdMob app ID -->
        <meta-data
            android:name="com.google.android.gms.ads.APPLICATION_ID"
            android:value="ca-app-pub-################~##########"/>
    </application>
</manifest>
```

Failure to add this tag will result in the app crashing at app launch with a message starting with "The Google Mobile Ads SDK was initialized incorrectly".

#### Manifest tag for Google Ad Manager publishers

Publishers using Google Ad Manager have to declare the app an "Ad Manager app" in the `AndroidManifest.xml` file:

```xml
<manifest>
    <application>
        <meta-data
            android:name="com.google.android.gms.ads.AD_MANAGER_APP"
            android:value="true"/>
    </application>
</manifest>
```

Failure to add this tag will result in the app crashing at app launch with a message starting with "The Google Mobile Ads SDK was initialized incorrectly".

Read more about [Google Play Data safety here.](/docs/sources/mobile-trackers/android-google-play-data-safety/)

***

### App set ID (Android only)

To identify a set of apps owned by an organization, Google provides the app set ID. Read more about it [here](https://developer.android.com/training/articles/app-set-id).

An extra dependency is required to populate the Android-specific properties `appSetId` and `appSetIdScope` within the platform context entity.

```gradle
dependencies {
    ...
    implementation 'com.google.android.gms:play-services-appset:16.0.2'
    ...
}
```

---

# Track screen views and engagement with the native mobile trackers
> Automatically track screen changes, time spent on screens, and user engagement metrics in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/screen-tracking/

Screen view tracking captures screen changes within the app.

**iOS:**

## Screen View Tracking in UIKit

The screen view tracking for UIKit views is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```swift
let trackerConfig = TrackerConfiguration()
    .screenViewAutotracking(true)
    .screenContext(true)
```

Using method swizzling in the `ViewController` class, the tracker automatically detects when screens are loaded (triggered by viewDidAppear in a ViewController) and tracks events that include information about the current and previous view controllers.

## Screen View tracking in SwiftUI

The tracker is able to track screen view events when selected view components appear in the SwiftUI lifecycle. To provide this functionality, it implements an extension over the `View` component which lets you annotate the components the events should be tracked for using the `snowplowScreen()` function:

```swift
import SwiftUI

struct ProductList: View {
    var body: some View {
        List {
            ...
        }
        .snowplowScreen(name: "ProductList") // this will ensure that screen view events are tracked for this view
    }
}
```

You can provide additional context entities to be tracked with the screen view events and also choose the tracker namespace to use:

```swift
import SwiftUI

struct ProductDetail: View {
    var product: Product

    var body: some View {
        List {
            ...
        }
        .snowplowScreen(
            name: "ProductDetail",
            entities: [ // list of context entities attached to the events
                (
                    schema: "iglu:com.acme_company/example/jsonschema/2-1-1",
                    data: [ "name": product.name ]
                )
            ],
            trackerNamespace: "ns" // namespace of tracker to track the event with
        )
    }
}
```

**Android (Kotlin):**

## Screen View tracking in traditional Activity-based apps

The screen view tracking for Activities (screens) is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .screenViewAutotracking(true)
    .screenContext(true)
```

Using the Android `Application.ActivityLifecycleCallbacks` interface, the tracker automatically detects when Activities are loaded and tracks events that include information about the current and previous Activity.

## Screen View tracking in Jetpack Compose apps

Apps built with Jetpack Compose are constructed from Composable functions rather than Activities, and so the built-in ScreenViewAutotracking will not work.

One possibility for an equivalent functionality is to add a navigation listener, which can track a screen view for each new destination.

```kotlin
fun autoTrackScreenView(navController: NavController) {
    navController.addOnDestinationChangedListener { _, destination, _ ->
        Snowplow.defaultTracker?.track(ScreenView(destination.route ?: "null"))
    }
}
```

Try out this example in the Compose demo inside the [Android codebase](https://github.com/snowplow/snowplow-android-tracker).

Depending how your app is configured, this listener may result in a `ScreenView` being tracked when the app returns from background. This is not how the tracker is intended to work; our screen engagement feature tracks the time spent on a screen whether in foreground or background. If you are experiencing this extra `ScreenView` (and `ScreenEnd`) being tracked, consider adding a check to the listener callback.

**Android (Java):**

## Screen View tracking in traditional Activity-based apps

The screen view tracking for Activities (screens) is enabled by default. It can be set in `TrackerConfiguration` like in the example below:

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .screenViewAutotracking(true)
    .screenContext(true);
```

Using the Android `Application.ActivityLifecycleCallbacks` interface, the tracker automatically detects when Activities are loaded and tracks events that include information about the current and previous Activity.

***

## Screen view event and screen context entity

Automatic screen view tracking tracks two pieces of information:

- The tracker automatically tracks each screen change using a [`ScreenView` event](/docs/events/ootb-data/page-and-screen-view-events/#screen-views).
- If the `TrackerConfiguration.screenContext` property is enabled, the tracker attaches a [`Screen` entity](/docs/events/ootb-data/page-and-screen-view-events/#screen-entity) to all the events tracked by the tracker reporting the last (and probably current) screen visible on device when the event was tracked.

The `Screen` entity is conditioned by the internal state of the tracker only. To make an example, if the developer manually tracks a `ScreenView` event, all the following events will have a `Screen` entity attached reporting the same information as the last tracked ScreenView event, even if it was manually tracked and the app is in a different screen.

Indeed, disabling the `screenViewAutotracking` only, the tracker can still attach `Screen` entities automatically based only to the manual tracking of `ScreenView` events, and vice versa.

## Screen engagement tracking

> **Note:** This feature has been added in the version 6.0.0 of the iOS and Android trackers.

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .screenEngagementAutotracking(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .screenEngagementAutotracking(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .screenEngagementAutotracking(true);
```

***

### Screen time

Screen time is tracked automatically by the tracker based on the screen view events.

Additionally, the tracker makes use of `application_background` and `application_foreground` events to distinguish between screen time in foreground (`foreground_sec` property in `screen_summary` entity) and screen time in background (`background_sec` property). Make sure that lifecycle autotracking is enabled (it is by default) in order for the tracker to be able to distinguish between these two states.

The foreground time is translated into the engaged time during modeling with the unified dbt package (see below). Foreground and background time together result in the absolute time on screen.

### List item view tracking

Part of screen engagement is tracking how much users saw on the screen. There are two ways that this information can be tracked:

1. By the number of items in a list viewed on the screen.
2. By the scroll depth on the screen in pixels.

The first approach assumes that there is a single list displayed on the screen, ideally with items of the same size. In this scenario, the track can track the index of the last viewed item and the total count of all items.

To provide the tracker with the information that a new list item was viewed by a user, the app will track a `ListItemView` event. Although it is tracked as an event, the tracker won't send it individually to the collector (as long as `screenEngagementAutotracking` is enabled) but will process the information into the next `screen_summary` entity and discard the list item view event.

**iOS:**

```swift
let event = ListItemView(
    index: 1, // Index of the item in the list.
    totalItems: 10 // Total number of items in the list.
)
Snowplow.defaultTracker()?.track(event)
```

#### Using the `snowplowListItem` modifier in SwiftUI

If you are using SwiftUI in your app, you can use the `snowplowListItem` view modifier to automatically track list item views. Annotating the items in your list will ensure that they are tracked as they become visible.

```swift
import SwiftUI

struct ProductList: View {
    var products: [Product]

    var body: some View {
        List {
            ForEach(Array(products.enumerated()), id: \.1.url) { offset, product in
                NavigationLink {
                    ProductDetail(url: product)
                } label: {
                    ...
                }
                .snowplowListItem(index: offset, itemsCount: products.count) // will track the ListItemView event automatically
            }
        }
        .snowplowScreen(...)
    }
}
```

**Android (Kotlin):**

```kt
val event = ListItemView(
    index = 10, // Index of the item in the list.
    itemsCount = 100 // Total number of items in the list.
)
Snowplow.defaultTracker?.track(event)
```

**Android (Java):**

```java
ListItemView event = new ListItemView(
    10, // Index of the item in the list.
    100 // Total number of items in the list.
);
Snowplow.defaultTracker.track(event);
```

***

### Scroll depth tracking

The second approach for tracking how much users saw on the screen is to track the scroll depth in pixels. This may be useful in case you have a single scroll view on the screen which can be scrolled horizontally, vertically or in both directions.

To give the tracker the information of the scroll depth, one can use the `ScrollChanged` event. Similarly like in the `ListItemView` event, this event won't be send to the collector as an individual event (as long as `screenEngagementAutotracking` is enabled) but will be processed into the `screen_summary` entity tracked with the next lifecycle or screen end event.

The `ScrollChanged` event contains information about the scroll offset as well as the content size (size of the content shown in the scroll view) and the dimensions of the scroll view on the screen. To track the event in UIKit on iOS, for instance, you can make use of the `scrollViewDidEndDecelerating` and `scrollViewDidEndDragging` callbacks in `UIScrollViewDelegate`.

**iOS:**

```swift
let event = ScrollChanged(xOffset: 15, yOffset: 30, viewWidth: 15, viewHeight: 20, contentWidth: 200, contentHeight: 100)
Snowplow.defaultTracker()?.track(event)
```

**Android (Kotlin):**

```kt
val event = ScrollChanged(
    xOffset = 15, // Horizontal scroll offset in pixels.
    yOffset = 30, // Vertical scroll offset in pixels.
    viewWidth = 15, // Width of the scroll view in pixels.
    viewHeight = 20, // Height of the scroll view in pixels.
    contentWidth = 150, // Height of the content of the scroll view in pixels
    contentHeight = 100 // Width of the content of the scroll view in pixels
)
Snowplow.defaultTracker?.track(event)
```

**Android (Java):**

```java
ScrollChanged event = new ScrollChanged();
event.setXOffset(15); // Horizontal scroll offset in pixels.
event.setYOffset(10); // Vertical scroll offset in pixels.
event.setViewHeight(20); // Height of the scroll view in pixels.
event.setViewWidth(30); // Width of the scroll view in pixels.
event.setContentHeight(100); // Height of the content of the scroll view in pixels
event.setContentWidth(150); // Width of the content of the scroll view in pixels
Snowplow.defaultTracker.track(event);
```

***

### Modeled data using the Snowplow Unified dbt package

Starting with version 0.2.0, the [Snowplow Unified Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) will process the screen engagement information in the `screen_summary` entity to provide the following metrics:

1. In the `snowplow_unified_views` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.
   - `last_list_item_index`, `list_items_count`, and `list_items_percentage_scrolled` will contain information about the number of viewed items in a list.
   - `horizontal_pixels_scrolled`, `vertical_pixels_scrolled`, `horizontal_percentage_scrolled`, `horizontal_percentage_scrolled` will contain information about the scroll depth in pixels.

2. In the `snowplow_unified_sessions` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.

3. In the `snowplow_unified_users` derived table:

   - `engaged_time_in_s` will contain the screen foreground time.
   - `absolute_time_in_s` will contain the screen foreground + background time.

---

# Track sessions with the native mobile trackers
> Track user sessions with configurable timeouts, session callbacks, and cross-navigation support in mobile applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/session-tracking/

Session tracking captures the session which helps to keep track of the user activity in the app.

Client session tracking is enabled by default. It can be set through the `TrackerConfiguration` as explained below.

**iOS:**

```swift
let trackerConfig = TrackerConfiguration()
    .sessionContext(true)
```

**Android (Kotlin):**

```kotlin
val trackerConfig = TrackerConfiguration("appId")
    .sessionContext(true)
```

**Android (Java):**

```java
TrackerConfiguration trackerConfig = new TrackerConfiguration("appId")
    .sessionContext(true);
```

***

When enabled, the tracker appends a [`client_session` entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) to each event it sends and it maintains this session information as long as the application is installed on the device.

Sessions correspond to tracked user activity. User information is set via the [session entity](/docs/events/ootb-data/user-and-session-identification/#session-entity).

A session expires when no tracking events have occurred for the amount of time defined in a timeout (by default 30 minutes). The session timeout check is executed for each event tracked. If the gap between two consecutive events is longer than the timeout the session is renewed. There are two timeouts since a session can timeout in the foreground (while the app is visible) or in the background (when the app has been suspended, but not closed).

The timeouts for the session can be configured in the `SessionConfiguration` like in the example below:

**iOS:**

```swift
let sessionConfig = SessionConfiguration(
    foregroundTimeout: Measurement(value: 360, unit: .seconds),
    backgroundTimeout: Measurement(value: 360, unit: .seconds)
)
Snowplow.createTracker(
    namespace: "appTracker",
    network: networkConfig,
    configurations: [trackerConfig, sessionConfig]
)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(
    TimeMeasure(6, TimeUnit.SECONDS),
    TimeMeasure(30, TimeUnit.SECONDS)
)
Snowplow.createTracker(applicationContext, namespace, networkConfig, sessionConfig)
```

**Android (Java):**

```java
SessionConfiguration sessionConfig = new SessionConfiguration(
    new TimeMeasure(6, TimeUnit.SECONDS),
    new TimeMeasure(30, TimeUnit.SECONDS)
);
Snowplow.createTracker(getApplicationContext(), namespace, networkConfig, sessionConfig);
```

***

The lifecycle events (`Foreground` and `Background` events) have a role in the session expiration. The lifecycle events can be enabled as explained in [App Lifecycle Tracking](/docs/sources/mobile-trackers/tracking-events/lifecycle-tracking/). Once enabled they will be fired automatically when the app moves from foreground state to background state and vice versa.

When the app moves from foreground to background, the `Background` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the foreground timeout. When the app moves from background to foreground, the `Foreground` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the background timeout.

For instance, with this configuration:

**iOS:**

```swift
SessionConfiguration(
    foregroundTimeout: Measurement(value: 360, unit: .seconds),
    backgroundTimeout: Measurement(value: 15, unit: .seconds)
)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(
    TimeMeasure(360, TimeUnit.SECONDS),
    TimeMeasure(15, TimeUnit.SECONDS)
)
```

**Android (Java):**

```java
SessionConfiguration sessionConfig = new SessionConfiguration(
    new TimeMeasure(360, TimeUnit.SECONDS),
    new TimeMeasure(15, TimeUnit.SECONDS)
);
```

***

the session would expire if the app is in background for more than 15 seconds, like in this example:

```text
time: 0s - ScreenView event - foreground timeout session check - session 1
time: 3s - Background event - foreground timeout session check (3 < 360) - session 1
time: 30s - Foreground event - background timeout session check (30 > 15) - session 2
```

In the above example, the `Foreground` event triggers a new session because the time spent in background (without tracked events) is bigger than the background timeout for the session.

## Session callback

> **Info:** This feature is available since v3.1.

The tracker allows the configuration of a callback to inform the app every time a new session is created (in correspondence of a session timeout check). This can be configured in the `SessionConfiguration` and it provides the `SessionState` where all the info already tracked in the session can be accessed.

Below is an example of where the session callback is used to print out the values of session every time a new session is generated by the tracker:

**iOS:**

```swift
...
let sessionConfig = SessionConfiguration()
    .onSessionStateUpdate { session in
        print("SessionState: id: \(session.sessionId) - index: \(session.sessionIndex) - userID: \(session.userId) - firstEventID: \(session.firstEventId)")
    }
...
let tracker = Snowplow.createTracker(namespace: kNamespace, network: networkConfig, configurations: [sessionConfig])
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration(
    TimeMeasure(6, TimeUnit.SECONDS),
    TimeMeasure(30, TimeUnit.SECONDS)
)
    .onSessionUpdate { state ->
        log(
            "Session: " + state.getSessionId()
                .toString() + "\r\nprevious: " + state.getPreviousSessionId()
                .toString() + "\r\neventId: " + state.getFirstEventId()
                .toString() + "\r\nindex: " + state.getSessionIndex()
                .toString() + "\r\nuserId: " + state.getUserId()
        )
    }

Snowplow.createTracker(applicationContext, namespace, networkConfig, sessionConfig)
```

**Android (Java):**

```java
...
SessionConfiguration sessionConfig = new SessionConfiguration(
    new TimeMeasure(6, TimeUnit.SECONDS),
    new TimeMeasure(30, TimeUnit.SECONDS)
)
    .onSessionUpdate(state -> log(
        "Session: " + state.getSessionId()
                + "\r\nprevious: " + state.getPreviousSessionId()
                + "\r\neventId: " + state.getFirstEventId()
                + "\r\nindex: " + state.getSessionIndex()
                + "\r\nuserId: " + state.getUserId()
    ));

...
Snowplow.createTracker(getApplicationContext(), namespace, networkConfig, sessionConfig);
```

***

## Decorating outgoing links using cross-navigation tracking

> **Note:** This feature was introduced in version 6.0.0 of the iOS and Android trackers.

The `decorateLink` function is part of Snowplow's [cross-navigation solution](/docs/events/cross-navigation/) for tracking the movement of users across different apps and platforms.

Choose which parameters to include using a `CrossDeviceParameterConfiguration` object. The `domainUserId` and `timestamp` are always included automatically. Use booleans to select which properties to include in the link decoration. The tracker will derive the required values from its configuration.

This table shows the options:

| Property         | Description                      | Type    | Included by default? | Value used                         |
| ---------------- | -------------------------------- | ------- | -------------------- | ---------------------------------- |
| `sessionId`      | Current session UUID identifier  | Boolean | ✅                    | `SessionController.sessionId`      |
| `subjectUserId`  | Custom business user identifier  | Boolean | ❌                    | `SubjectController.userId`         |
| `sourceId`       | Application identifier           | Boolean | ✅                    | `TrackerConfiguration.appId`       |
| `sourcePlatform` | Platform of the current device   | Boolean | ❌                    | `TrackerController.devicePlatform` |
| `reason`         | Custom information or identifier | String  | ❌                    | Custom string provided by you      |

The `subjectUserId` property is not included by default, in case it contains personal data.

**iOS:**

```swift
let link = URL(string: "https://example.com")!
let decoratedLink = Snowplow.defaultTracker()?.decorateLink(
  link,
  // optional configuration for which information to be added to the link
  extendedParameters: CrossDeviceParameterConfiguration(sessionId: false, subjectUserId: true)
)
```

**Android (Kotlin):**

```kotlin
val link = Uri.parse("http://example.com")
val decoratedLink = Snowplow.defaultTracker.decorateLink(
  link,
  // optional configuration for which information to be added to the link
  CrossDeviceParameterConfiguration(sessionId = false, subjectUserId = true)
)
```

**Android (Java):**

```java
Uri link = Uri.parse("http://example.com");
Uri decoratedLink = Snowplow.getDefaultTracker().decorateLink(link, new CrossDeviceParameterConfiguration(
    false, // sessionId
    true, // subjectUserId
    false, // sourceId
    false, // sourcePlatform
    null // reason
));
```

***

## Session behavior on app restarts

By default, each time the mobile app restarts (or when a new tracker instance is created), a new session is started regardless of whether the previous session timed out or not.

Since version 6.2 of the iOS and Android trackers, there is an option to continue the previously persisted session when the app restarts. The previous session is only continued in case the app is restarted within the configured session timeout interval. The option can be configured using `SessionConfiguration`:

**iOS:**

```swift
let sessionConfig = SessionConfiguration()
    .continueSessionOnRestart(true)
```

**Android (Kotlin):**

```kotlin
val sessionConfig = SessionConfiguration()
    .continueSessionOnRestart(true)
```

**Android (Java):**

```java
SessionConfiguration trackerConfig = new SessionConfiguration()
    .continueSessionOnRestart(true);
```

***

With the option enabled, every session update is persisted to UserDefaults or shared preferences storage. In case it's disabled, only session changes are persisted to UserDefaults. This means that there is more overhead when the option is enabled.

---

# Track visionOS events with the iOS tracker
> Track visionOS-specific events including window groups and immersive spaces in Apple Vision Pro applications.
> Source: https://docs.snowplow.io/docs/sources/mobile-trackers/tracking-events/visionos/

> **Note:** Snowplow visionOS tracking was added in version 6.0.0.

The Snowplow iOS tracker supports tracking within visionOS apps. All the usual events, including screen views, can be tracked as for any SwiftUI app. We've also provided additional events and entities to help you understand your visionOS users.

The immersive space context entity is semi-automatically added to all events.

The events are: `OpenWindowEvent`, `DismissWindowEvent`, `OpenImmersiveSpaceEvent`, and `DismissImmersiveSpaceEvent`. Find the schema details for the events and entities [here](/docs/events/ootb-data/visionos-swiftui/).

### Window events

Track the opening and dismissing of SwiftUI window groups using `OpenWindowEvent` and `DismissWindowEvent`. These events can be used in any SwiftUI app, not just visionOS. The event data is sent as a window group context entity (see below) attached to these events; the events themselves have no properties.

#### OpenWindowEvent

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")

let event = OpenWindowEvent(
    id: "group 1", // window group ID
    uuid: UUID(), // optional UUID
    titleKey: "window 1", // optional window title
    windowStyle: WindowStyle.automatic // optional style of the window group
)
tracker.track(event)
```

#### DismissWindowEvent

```swift
let tracker = Snowplow.createTracker(namespace: "appTracker", endpoint: "https://snowplow-collector-url.com")

let event = DismissWindowEvent(
    id: "group 1", // window group ID
    windowId: UUID(), // optional UUID
    titleKey: "window 1", // optional window title
    windowStyle: WindowStyle.automatic // optional style of the window group
)
tracker.track(event)
```

Determining which events occurred in which window group can be done during modelling using these two event types and timestamps. You could also manually add a window group context entity to tracked events.

In this example, a window group entity is added to an [Ecommerce](/docs/sources/mobile-trackers/tracking-events/ecommerce-tracking/) `ProductViewEvent`.

```swift
let product = ProductEntity(
  id: "productId",
  category: "category",
  currency: "GBP",
  price: 100
)
let event = ProductViewEvent(product: product)
let entity = WindowGroupEntity(id: "group_1")
event.entities.append(entity)
```

Read more about customized tracking [here](/docs/sources/mobile-trackers/custom-tracking-using-schemas/).

### Immersive space events

Use the `OpenImmersiveSpaceEvent` and `DismissImmersiveSpaceEvent` to automatically add an immersive space context entity to all events occurring within an immersive space. The entity will identify the immersive space in which the events occurred. This feature is enabled by default.

```swift
let tracker = Snowplow.createTracker(
  namespace: "appTracker",
  endpoint: "https://snowplow-collector-url.com"
)

let event = OpenImmersiveSpaceEvent(
    id: "group 1", // space ID
    viewId: UUID(), // optional UUID
    immersionStyle: ImmersionStyle.automatic, // optional
    upperLimbVisibility: UpperLimbVisibility.hidden // optional
)
// all subsequent events, including this one, will have the immersive space entity
// until DismissImmersiveSpaceEvent is tracked
tracker.track(event)

// the dismiss event will have the entity
// but subsequent events will not
tracker.track(DismissImmersiveSpaceEvent())
```

If the immersive space entity autotracking is turned off, the `Open` and `Dismiss` events can still be tracked. However, only the `OpenImmersiveSpaceEvent` will have the entity. The events themselves have no properties.

The tracker automatically generates a `viewId` each time a new immersive space entity is created. You could also pass in your own UUID.

The visionOS methods `openImmersiveSpace` and `dismissImmersiveSpace` are asynchronous. We advise that you write your tracking code such that the Snowplow immersive space events await those methods' completion, for accurate tracking.

---

# Configure emitters in the .NET tracker
> Configure the emitter for sending and storing events with async processing, persistent storage, and collector endpoints.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/emitter/

These instructions are for Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL).

The Emitter object is responsible for sending and storing all events.

We have one emitter available currently:

- `AsyncEmitter` : Fully asynchronous operation which uses threads to perform all of its operations.

The Emitter depends on four other objects being built:

- `IEndpoint`
- `IStorage`
- `IPersistentBlockingQueue`
- `IPayloadToString`

## Emitter Constructor

| **Argument Name**    | **Description**                                                                   | **Required?** | **Default** |
| -------------------- | --------------------------------------------------------------------------------- | ------------- | ----------- |
| `endpoint`           | The endpoint object configured for sending events                                 | Yes           | Null        |
| `queue`              | The queue to be used to push and pop events from                                  | Yes           | Null        |
| `sendLimit`          | The amount of events to get from the queue at a time                              | No            | 100         |
| `stopPollIntervalMs` | The amount of time to wait before checking for more events                        | No            | 300         |
| `sendSuccessMethod`  | An optional callback function which will report event success and failure counts  | No            | Null        |
| `deviceOnlineMethod` | An optional delegate function which will be used to check if the device is online | No            | Null        |
| `logger`             | The logger to use within the application                                          | No            | Null        |

A full Emitter construction should look like the following:

```csharp
var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);
var storage = new LiteDBStorage("events.db");
var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());

AsyncEmitter emitter = new AsyncEmitter(endpoint, queue, l: logger);
```

**NOTE**: The send limit can impact performance greatly as it determines how often we need to perform I/O to the disk and how big the POSTed event batches can be.

**WARNING**: If you are sending events via GET note that each event is sent as its own task, so this has the potential to launch 100 outbound tasks in parallel. It is recommended to lower this range if using GET to 10-15 as a maximum.

## Endpoint Constructor

This is a container for information about how to reach your collector.

| **Argument Name** | **Description**                                        | **Required?** | **Default**       |
| ----------------- | ------------------------------------------------------ | ------------- | ----------------- |
| `host`            | The collector uri to send events to                    | Yes           | Null              |
| `protocol`        | The protocol to use when sending events (HTTP / HTTPs) | No            | HttpProtocol.HTTP |
| `port`            | If the collector is not on port 80                     | No            | Null              |
| `method`          | The method to use when sending (GET / POST)            | No            | HttpMethod.GET    |
| `postMethod`      | Custom method for sending events via POST              | No            | Null              |
| `getMethod`       | Custom method for sending events via GET               | No            | Null              |
| `byteLimitPost`   | Maximum byte limit when sending a POST request         | No            | 40000             |
| `byteLimitGet`    | Maximum byte limit when sending a GET request          | No            | 40000             |
| `logger`          | The logger to use within the application               | No            | Null              |

We have one endpoint available currently:

- `SnowplowHttpCollectorEndpoint`

A full Endpoint construction should look like the following:

```csharp
SnowplowHttpCollectorEndpoint endpoint = new SnowplowHttpCollectorEndpoint("com.acme-collector", protocol: HttpProtocol.HTTPS, method: HttpMethod.GET, l: logger);
```

**NOTE**: If any individual event exceeds the byte limits set then this event will be sent - but it will be assumed to have succeeded. This is to prevent constanstly attempting to send overly large events.

## Storage Constructor

| **Argument Name** | **Description**                             | **Required?** | **Default** |
| ----------------- | ------------------------------------------- | ------------- | ----------- |
| `path`            | The file path to store the database file at | Yes           | Null        |

We have one storage target available currently:

- `LiteDBStorage`

A full Storage construction should look like the following:

```csharp
LiteDBStorage storage = new LiteDBStorage("events.db");
```

**NOTE**: When using the Tracker within Xamarin you will need to fetch a correct path for internal storage. Some example code for fetching this path:

```csharp
// Android
public string GetLocalFilePath(string filename)
{
  string path = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
  return Path.Combine(path, filename);
}

// iOS
public string GetLocalFilePath(string filename)
{
  string docFolder = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
  string libFolder = Path.Combine(docFolder, "..", "Library", "Databases");

  if (!Directory.Exists(libFolder))
  {
    Directory.CreateDirectory(libFolder);
  }

  return Path.Combine(libFolder, filename);
}
```

## Queue Constructor

| **Argument Name** | **Description**                          | **Required?** | **Default** |
| ----------------- | ---------------------------------------- | ------------- | ----------- |
| `storage`         | The storage object to use with the queue | Yes           | Null        |
| `payloadToString` | Serializer for Payload objects           | Yes           | Null        |

We have one queue available currently:

- `PersistentBlockingQueue`

A full queue construction should look like the following:

```csharp
PersistentBlockingQueue queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());
```

## Payload Serializer Constructor

We have one payload serializer available currently:

- `PayloadToJsonString`

A full queue construction should look like the following:

```csharp
PayloadToJsonString serializer = new PayloadToJsonString();
```

This controls how we queue information for internal use.

---

# Track events with the .NET tracker
> Track page views, structured events, screen views, and custom self-describing events with the .NET tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/event-tracking/

Events supported by the .NET Tracker at a glance:

| **Events**                    | **Description**                                        |
| ----------------------------- | ------------------------------------------------------ |
| `Track(PageView)`             | Track and record views of web pages                    |
| `Track(ScreenView)`           | Track the user viewing a screen within the application |
| `Track(Structured)`           | Track a Snowplow custom structured event               |
| `Track(Timing)`               | Track a Timing with Category event                     |
| `Track(Unstructured)`         | Track a Snowplow custom unstructured event             |
| `Track(EcommerceTransaction)` | Track an ecommerce transaction and its items           |

### Track page views with `Track(PageView)`

You can use `Track(PageView)` to track a user viewing a web page within your app.

Arguments are:

| **Argument**     | **Description**                      | **Required?** | **Type**         |
| ---------------- | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`        | The URL of the page                  | Yes           | `string`         |
| `pageTitle`      | The title of the page                | No            | `string`         |
| `referrer`       | The address which linked to the page | No            | `string`         |
| `customContexts` | Optional custom context              | No            | `List<IContext>` |
| `trueTimestamp`  | Optional true-timestamp              | No            | `long`           |
| `eventId`        | Optional custom event id             | No            | `string`         |

Examples:

```csharp
t1.Track(new PageView()
    .SetPageUrl("www.example.com")
    .SetPageTitle("example")
    .SetReferrer("www.referrer.com")
    .Build());

t1.Track(new PageView()
    .SetPageUrl("www.example.com")
    .SetPageTitle("example")
    .SetReferrer("www.referrer.com")
    .SetCustomContext(contextList)
    .SetTrueTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

### Track screen views with `Track(ScreenView)`

Use `Track(ScreenView)` to track a user viewing a screen (or equivalent) within your app. You **must** use either `name` or `id`. Arguments are:

| **Argument**             | **Description**                     | **Required?** | **Type**         |
| ------------------------ | ----------------------------------- | ------------- | ---------------- |
| `name`                   | Human-readable name for this screen | No            | `string`         |
| `id`                     | Unique identifier for this screen   | No            | `string`         |
| `customContexts`         | Optional custom context             | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp   | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp             | No            | `long`           |
| `eventId`                | Optional custom event id            | No            | `string`         |

Examples:

```csharp
t1.Track(new ScreenView()
    .SetName("HUD > Save Game")
    .SetId("screen23")
    .Build());

t1.Track(new ScreenView()
    .SetName("HUD > Save Game")
    .SetId("screen23")
    .SetCustomContext(contextList)
    .SetTrueTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

### Track structured events with `Track(Structured)`

Use `Track(Structured)` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**             | **Description**                                                  | **Required?** | **Type**         |
| ------------------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`               | The grouping of structured events which this `action` belongs to | Yes           | `string`         |
| `action`                 | Defines the type of user interaction which this event involves   | Yes           | `string`         |
| `label`                  | A string to provide additional dimensions to the event data      | No            | `string`         |
| `property`               | A string describing the object or the action performed on it     | No            | `string`         |
| `value`                  | A value to provide numerical data about the event                | No            | `double`         |
| `customContexts`         | Optional custom context                                          | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp                                | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp                                          | No            | `long`           |
| `eventId`                | Optional custom event id                                         | No            | `string`         |

Examples:

```csharp
t1.Track(new Structured()
    .SetCategory("shop")
    .SetAction("add-to-basket")
    .Build());

t1.Track(new Structured()
    .SetCategory("shop")
    .SetAction("add-to-basket")
    .SetLabel("Add To Basket")
    .SetProperty("pcs")
    .SetValue(2.00)
    .SetCustomContext(contextList)
    .SetTrueTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

### Track timing events with `Track(Timing)`

Use `Track(Timing)` to track an event related to a custom timing.

| **Argument**             | **Description**                        | **Required?** | **Type**         |
| ------------------------ | -------------------------------------- | ------------- | ---------------- |
| `category`               | The category of the timed event        | Yes           | `string`         |
| `label`                  | The label of the timed event           | No            | `string`         |
| `timing`                 | The timing measurement in milliseconds | Yes           | `int`            |
| `variable`               | The name of the timed event            | Yes           | `string`         |
| `customContexts`         | Optional custom context                | No            | `List<IContext>` |
| `deviceCreatedTimestamp` | Optional device-created-timestamp      | No            | `long`           |
| `trueTimestamp`          | Optional true-timestamp                | No            | `long`           |
| `eventId`                | Optional custom event id               | No            | `string`         |

Examples:

```csharp
t1.Track(new Timing()
    .SetCategory("category")
    .SetVariable("variable")
    .SetTiming(1)
    .Build());

t1.Track(new Timing()
    .SetCategory("category")
    .SetVariable("variable")
    .SetTiming(1)
    .SetLabel("label")
    .SetCustomContext(contextList)
    .SetTrueTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

#### Track unstructured events with `Track(SelfDescribing)`

Custom unstructured events are a flexible tool that enable Snowplow users to define their own event types and send them into Snowplow.

When a user sends in a custom unstructured event, they do so as a JSON of name-value properties, that conforms to a JSON schema defined for the event earlier.

Use `Track(SelfDescribing)` to track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**             | **Description**                   | **Required?** | **Type**             |
| ------------------------ | --------------------------------- | ------------- | -------------------- |
| `eventData`              | The properties of the event       | Yes           | `SelfDescribingJson` |
| `customContexts`         | Optional custom context           | No            | `List<IContext>`     |
| `deviceCreatedTimestamp` | Optional device-created-timestamp | No            | `long`               |
| `trueTimestamp`          | Optional true-timestamp           | No            | `long`               |
| `eventId`                | Optional custom event id          | No            | `string`             |

Example event json to track:

```csharp
{
  "schema": "iglu:com.acme/save_game/jsonschema/1-0-0",
  "data": {
    "levelName": "Barrels o' Fun",
    "levelIndex": 23
  }
}
```

How to set it up?

```csharp
// Create a Dictionary of your event data
Dictionary<string, object> eventDict = new Dictionary<string, object>();
eventDict.Add("levelName", "Barrels o' Fun");
eventDict.Add("levelIndex", 23);

// Create your event data
SelfDescribingJson eventData = new SelfDescribingJson("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict);

// Track your event with your custom event data
t1.Track(new SelfDescribing()
    .SetEventData(eventData)
    .Build());

// OR

t1.Track(new SelfDescribing()
    .SetEventData(eventData)
    .SetCustomContext(contextList)
    .SetTrueTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

### Track ecommerce transactions with `Track(EcommerceTransaction)`

Use `Track(EcommerceTransaction)` to track an ecommerce transaction.

Arguments:

| **Argument**             | **Description**                   | **Required?** | **Type**                         |
| ------------------------ | --------------------------------- | ------------- | -------------------------------- |
| `orderId`                | ID of the eCommerce transaction   | Yes           | `string`                         |
| `totalValue`             | Total transaction value           | Yes           | `double`                         |
| `affiliation`            | Transaction affiliation           | No            | `string`                         |
| `taxValue`               | Transaction tax value             | No            | `double`                         |
| `shipping`               | Delivery cost charged             | No            | `double`                         |
| `city`                   | Delivery address city             | No            | `string`                         |
| `state`                  | Delivery address state            | No            | `string`                         |
| `country`                | Delivery address country          | No            | `string`                         |
| `currency`               | Transaction currency              | No            | `string`                         |
| `items`                  | Items in the transaction          | Yes           | `List<EcommerceTransactionItem>` |
| `customContexts`         | Optional custom context           | No            | `List<IContext>`                 |
| `deviceCreatedTimestamp` | Optional device-created-timestamp | No            | `long`                           |
| `trueTimestamp`          | Optional true-timestamp           | No            | `long`                           |
| `eventId`                | Optional custom event id          | No            | `string`                         |

The `items` argument is a `List` of individual `EcommerceTransactionItem` elements representing the items in the e-commerce transaction. Note that `Track(EcommerceTransaction)` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `items` `List`.

Each transaction item event will have the same device created timestamp, true timestamp, orderId, and currency as the main transaction event.

#### `EcommerceTransactionItem`

#### Using Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL)

To instantiate a `EcommerceTransactionItem` in your code, simply use the following constructor signature:

```csharp
EcommerceTransactionItem item = new EcommerceTransactionItem ()
      .SetSku ("sku")
      .SetPrice (10.2)
      .SetQuantity (1)
      .SetName ("name")
      .SetCategory ("category")
      .Build ()
```

These are the fields that can appear as elements in each `EcommerceTransactionItem` element of the transaction item's `List`:

| **Field**        | **Description**          | **Required?** | **Type**         |
| ---------------- | ------------------------ | ------------- | ---------------- |
| `sku`            | Item SKU                 | Yes           | `string`         |
| `price`          | Item price               | Yes           | `double`         |
| `quantity`       | Item quantity            | Yes           | `int`            |
| `name`           | Item name                | No            | `string`         |
| `category`       | Item category            | No            | `string`         |
| `customContexts` | Optional custom context  | No            | `List<IContext>` |
| `eventId`        | Optional custom event id | No            | `string`         |

Example of tracking a transaction containing two items:

```csharp
// Create some Transaction Items
EcommerceTransactionItem item1 = new EcommerceTransactionItem ()
    .SetSku ("item_sku_1")
    .SetPrice (10.2)
    .SetQuantity (1)
    .SetName ("item_name_1")
    .SetCategory ("item_category")
    .Build ();

EcommerceTransactionItem item2 = new EcommerceTransactionItem()
    .SetSku("item_sku_2")
    .SetPrice(1.00)
    .SetQuantity(1)
    .SetName("item_name_2")
    .SetCategory("item_category")
    .Build();

// Add these items to a List
List<EcommerceTransactionItem> items = new List<EcommerceTransactionItem>();
items.Add(item1);
items.Add(item2);

// Now Track the Transaction by using this list of items as an argument
tracker.Track(new EcommerceTransaction()
    .SetOrderId("order_id_1")
    .SetTotalValue(300.00)
    .SetAffiliation("my_affiliate")
    .SetTaxValue(30.00)
    .SetShipping(10.00)
    .SetCity("Boston")
    .SetState("Massachusetts")
    .SetCountry("USA")
    .SetCurrency("USD")
    .SetItems(items)
    .Build());
```

### Custom Contexts

Custom contexts are Self Describing Jsons with extra descriptive information that can be optionally attached to any Snowplow event with `SetCustomContexts(...)`. We provide several builders for Snowplow custom contexts as well as a generic builder if you wish to define and send your own custom contexts!

For ease of development you are also able to extend the `IContext` interface or the `AbstractContext` class for your own contexts if you so wish.

All of these contexts will need to be combined into a `List<IContext>` before being attachable to Snowplow Events.

#### `DesktopContext`

The following arguments can be used in a DesktopContext:

| **Field**            | **Description**                     | **Required?** | **Type** |
| -------------------- | ----------------------------------- | ------------- | -------- |
| `osType`             | The Operating System Type           | Yes           | `string` |
| `osVersion`          | The Version of the Operating System | Yes           | `string` |
| `osServicePack`      | Service Pack information            | No            | `string` |
| `osIs64Bit`          | If the OS is 32 or 64 bit           | No            | `bool`   |
| `deviceManufacturer` | Who made the device                 | No            | `string` |
| `deviceModel`        | What is the device model            | No            | `string` |
| `processorCount`     | How many cores does the device have | No            | `int`    |

An example of a DesktopContext construction:

```csharp
DesktopContext context = new DesktopContext ()
    .SetOsType("OS-X")
    .SetOsVersion("10.10.5")
    .SetOsServicePack("Yosemite")
    .SetOsIs64Bit(true)
    .SetDeviceManufacturer("Apple")
    .SetDeviceModel("Macbook Pro")
    .SetDeviceProcessorCount(4)
    .Build ();
```

#### `MobileContext`

The following arguments can be used in a MobileContext:

| **Field**            | **Description**                     | **Required?** | **Type**      |
| -------------------- | ----------------------------------- | ------------- | ------------- |
| `osType`             | The Operating System Type           | Yes           | `string`      |
| `osVersion`          | The Version of the Operating System | Yes           | `string`      |
| `deviceManufacturer` | Who made the device                 | Yes           | `string`      |
| `deviceModel`        | What is the device model            | Yes           | `string`      |
| `carrier`            | The name of the carrier             | No            | `string`      |
| `networkType`        | The type of network                 | No            | `NetworkType` |
| `networkTechnology`  | The networks technlogy              | No            | `string`      |
| `openIdfa`           | An OpenIDFA UUID                    | No            | `string`      |
| `appleIdfa`          | An Apple IDFA UUID                  | No            | `string`      |
| `appleIdfv`          | An Apple IDFV UUID                  | No            | `string`      |
| `androidIdfa`        | An Android IDFA UUID                | No            | `string`      |

An example of a MobileContext construction:

#### `GeoLocationContext`

The following arguments can be used in a GeoLocationContext:

| **Field**                   | **Description**            | **Required?** | **Type** |
| --------------------------- | -------------------------- | ------------- | -------- |
| `latitude`                  | The user latitude          | Yes           | `double` |
| `longitude`                 | The user longitude         | Yes           | `double` |
| `latitudeLongitudeAccuracy` | The user lat-long accuracy | No            | `double` |
| `altitude`                  | The user altitude          | No            | `double` |
| `altitudeAccuracy`          | The user alt accuracy      | No            | `double` |
| `bearing`                   | The user bearing           | No            | `double` |
| `speed`                     | The user speed             | No            | `double` |
| `timestamp`                 | A timestamp in ms          | No            | `long`   |

An example of a GeoLocationContext construction:

```csharp
GeoLocationContext context = new GeoLocationContext ()
    .SetLatitude(123.564)
    .SetLongitude(-12.6)
    .SetLatitudeLongitudeAccuracy(5.6)
    .SetAltitude(5.5)
    .SetAltitudeAccuracy(2.1)
    .SetBearing(3.2)
    .SetSpeed(100.2)
    .SetTimestamp(1234567890000)
    .Build ();
```

#### `GenericContext`

The GenericContext is a simple builder with three functions:

- `SetSchema(string)` : Sets the Context Schema Path
- `Add(string, object)` : Adds a single key-pair value to the data packet of this context
- `AddDict(string, object)` : Adds a dictionary of key-pair values to the data packet

You must set a schema string or a RuntimeException will be thrown.

An example of a GenericContext construction:

```csharp
GenericContext context = new GenericContext()
    .SetSchema("iglu:com.acme/acme_context/jsonschema/1-0-0")
    .Add("context", "custom")
    .Build();
```

### SelfDescribingJson

A `SelfDescribingJson` is used as a wrapper around a `Dictionary<string, object>`. After creating the Dictionary you want to wrap you can create a `SelfDescribingJson` using the following:

```csharp
// Data as a Dictionary
Dictionary<string, object> data = new Dictionary<string, object>();
data.Add("Event", "Data")

// We then create a new SelfDescribingJson
SelfDescribingJson json = new SelfDescribingJson("iglu:com.acme/example/jsonschema/1-0-0", data);
```

This object is now ready to be Tracked within an SelfDescribing Event.

You can create a SelfDescribingJson with the following arguments:

| **Argument** | **Description**                           | **Required?** | **Type**                    |
| ------------ | ----------------------------------------- | ------------- | --------------------------- |
| `schema`     | JsonSchema that describes the data        | Yes           | `string`                    |
| `data`       | Data that will be validated by the schema | No            | `Dictionary<string,object`> |

---

# .NET tracker SDK
> Track events in .NET websites and desktop applications with the Snowplow .NET tracker SDK for .NET Standard and Xamarin.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/

The Snowplow .NET Tracker allows you to track Snowplow events from your .NET websites and desktop applications. It's split into two libraries:

#### `Snowplow.Tracker`

This is a fully functional .NET Standard 1.4 tracking library - it will work on any platform that supports .NET Standard 1.4+ (including .NET 461+).

#### `Snowplow.Tracker.PlatformExtensions`

This is a Portable Class Library (PCL) wrapper around the core `Snowplow.Tracker` library that extends functionality in platform specific ways, for example to provide geo-location information when tracking users in a Xamarin mobile application. If you're using Xamarin we encourage you to use this library.

## Snowplow Demo Application

The Xamarin demo application can be deployed on Android and iOS. Simply launch the `Snowplow.Demo.App.sln` solution file with Visual Studio and deploy to either emulators or actual test devices. The .NET Core Console demo application can also be loaded with Visual Studio using `Snowplow.Demo.Console.sln`.

---

# Initialize the .NET tracker
> Initialize and configure the .NET tracker with emitter, storage, and subject settings for event tracking.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/initialization/

Assuming you have completed the .NET Tracker Setup for your project, you are now ready to initialize the .NET Tracker.

### Importing the library

Add the following `using` lines to the top of your `.cs` scripts to access the Tracker:

```csharp
using Snowplow.Tracker.Emitters;
using Snowplow.Tracker.Endpoints;
using Snowplow.Tracker.Logging;
using Snowplow.Tracker.Models;
using Snowplow.Tracker.Models.Events;
using Snowplow.Tracker.Models.Adapters;
using Snowplow.Tracker.Queues;
using Snowplow.Tracker.Storage;
```

You should now be able to setup the Tracker!

### Creating a tracker

#### Using Snowplow\.Tracker (.NET Standard) or Snowplow\.Tracker.PlatformExtensions (PCL)

To use the Tracker in your code simply instantiate the Tracker interface with the following:

```csharp
// Create logger
var logger = new ConsoleLogger();

// Controls the sending of events
var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);

// Controls the storage of events
// NOTE: You must dispose of storage yourself when closing your application!
var storage = new LiteDBStorage("events.db");

// Controls queueing events
var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());

// Controls pulling events of the queue and pushing them to the sender
var emitter = new AsyncEmitter(endpoint, queue, l: logger);

// Contains information about who you are tracking
var subject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");

Tracker.Tracker.Instance.Start(emitter: emitter, subject: subject, trackerNamespace: "some namespace", appId: "some appid", l: logger);
```

This starts a global singleton Tracker which can be accessed anywhere via the `Tracker.Tracker.Instance.{{ method }}` chain.

---

# Platform specific functions for the .NET tracker
> Access platform-specific information for mobile, desktop, and geo-location contexts in Xamarin and .NET applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/platform-specific-functions/

To support multiple platforms we provide several utility functions for fetching platform specific information.

### `GetMobileContext`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

**NOTE**: These mobile contexts do not currently fetch any advertising identifiers.

### `GetGeoLocationContext`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

**NOTE**: To make the `GeoLocation` context work on `iOS` you will need to add the following to your `Info.plist`:

```csharp
<key>UIBackgroundModes</key>
<array>
  <string>location</string>
</array>
<key>NSLocationAlwaysUsageDescription</key>
<string></string>
```

When you send your first event a prompt will be fired asking the user for permission to use their location.

### `GetDesktopContext`

#### PlatformExtensions only

Platforms:

- `.NET Framework 4.6.1+`

### 8.4 `GetLocalFilePath`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`

### 8.5 `IsDeviceOnline`

#### PlatformExtensions only

Platforms:

- `Xamarin.Android API 15+`
- `Xamarin.iOS 8+`
- `.NET Framework 4.6.1+`

---

# Track session data with the .NET tracker
> Track user sessions with persistent session data including session ID, index, and timeout configuration for .NET applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/session/

The Session object is responsible for maintaining persistent data around user sessions over the life-time of an application.

### Constructor

| **Argument Name**   | **Description**                           | **Required?** | **Default** |
| ------------------- | ----------------------------------------- | ------------- | ----------- |
| `savePath`          | The path to save persistent data into     | Yes           | Null        |
| `foregroundTimeout` | The time until a session expires in focus | No            | 600 (s)     |
| `backgroundTimeout` | The time until a session expires in back  | No            | 300 (s)     |
| `checkInterval`     | How often to validate the session timeout | No            | 15 (s)      |
| `logger`            | The logger to use within the application  | No            | Null        |

A full ClientSession construction should look like the following:

```csharp
ClientSession session = new ClientSession ("save_file_path.xml");
```

The timeout's refer to the length of time the session remains active after the last event is sent. As long as events are sent within this limit the session will not timeout.

**NOTE**: When using the Tracker within Xamarin you will need to fetch a correct path for internal storage. Some example code for fetching this path:

```csharp
// Android
public string GetLocalFilePath(string filename)
{
  string path = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
  return Path.Combine(path, filename);
}

// iOS
public string GetLocalFilePath(string filename)
{
  string docFolder = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
  string libFolder = Path.Combine(docFolder, "..", "Library", "Databases");

  if (!Directory.Exists(libFolder))
  {
    Directory.CreateDirectory(libFolder);
  }

  return Path.Combine(libFolder, filename);
}
```

### 6.2 Functions

#### `SetBackground(bool)`

Will set whether or not the application is in the background. It is up to the developer to set this metric if they wish to have a different timeout for foreground and background.

**NOTE**: If a timeout occurs while the application has been backgrounded the SessionChecker will be stopped automatically. Session checking will resume on Foregrounding or on an event being tracked.

---

# Install the .NET tracker
> Install the .NET tracker using NuGet for .NET Standard, Xamarin, and MAUI applications.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/setup/

Version 1.0.0+ marks the conversion of the core library to .NET Standard 1.4. This core (`Snowplow.Tracker`) is directly compatible with [these platforms](https://github.com/dotnet/standard/blob/master/docs/versions.md). We have also published a Portable Class Library (PCL) wrapper providing extra features for Xamarin users. This is called `Snowplow.Tracker.PlatformExtensions`.

The current version is 1.3.0.

## Setup

### .NET Standard users

To add the .NET Tracker as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Tracker
```

### Xamarin users

To add the .NET Tracker as a dependency to your project, install it in the Visual Studio Package Manager Console using [NuGet](https://www.nuget.org/):

```powershell
Install-Package Snowplow.Tracker.PlatformExtensions
```

Remember to add an assembly reference to the .NET Tracker to your project.

### MAUI users

The `Snowplow.Tracker.PlatformExtensions` package is currently not compatible with MAUI apps.

However, as a replacement for the `GetLocalFilePath` function that the package provides, you can use the following implementation:

```cs
/// <summary>
/// Returns a path for storing the Snowplow events and session databases
/// </summary>
public string GetLocalFilePath(string filename)
{
    var path = System.Environment.GetFolderPath(System.Environment.SpecialFolder.LocalApplicationData);
    return Path.Combine(path, filename);
}
```

This function can be used to initialize the tracker as follows:

```cs
var dest = new SnowplowHttpCollectorEndpoint(collectorUri, method: method, port: port, protocol: protocol);
var storage = new LiteDBStorage(GetLocalFilePath("events.db"));
var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());
var emitter = new AsyncEmitter(dest, queue);
var clientSession = new ClientSession(GetLocalFilePath("client_session.dict"));
Instance.Start(emitter: emitter, clientSession: clientSession);
```

---

# Configure a subject with the .NET tracker
> Add user and environment data to tracked events using the Subject class with platform, language, and user properties.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

The Subject class has a number of `Set...()` methods to attach extra data relating to the user of the tracked events.

Here are some examples:

```csharp
Subject s1 = new Subject();
s1.SetUserId("Kevin Gleason");
s1.SetLang("en-gb");
s1.SetScreenResolution(1920, 1080);
```

There are two ways to provide the subject information:

1. Globally for all tracked events. This is useful in client-side applications where all tracked events relate to the same user and share the same properties.
2. For each event individually. This is useful in server-side applications where each event might relate to a different user.

To configure a global subject, pass it to the tracker during initialization:

```csharp
var globalSubject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");

Tracker.Tracker.Instance.Start(emitter: emitter, subject: globalSubject, trackerNamespace: "some namespace", appId: "some appid", l: logger);
```

To pass the subject along with tracked events:

```csharp
Subject eventSubject = new Subject().SetUserId("Kevin Gleason");

Tracker.Instance.Track(
    new Structured()
        .SetCategory("shop")
        .SetAction("add-to-basket")
        .Build(),
    eventSubject
);
```

### `SetUserId`

You can set the user ID to any string:

```csharp
s1.SetUserId( "{{USER ID}}" )
```

Example:

```csharp
s1.SetUserId("alexd")
```

### `SetScreenResolution`

If your C# code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```csharp
s1.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s1.SetScreenResolution(1366, 768)
```

### `SetViewport`

If your C# code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```csharp
s1.SetViewport( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s1.SetViewport(300, 200)
```

### `SetColorDepth`

If your C# code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```csharp
s.SetColorDepth( {{BITS PER PIXEL}} )
```

The number should be a positive integer, in bits per pixel. Example:

```csharp
s.SetColorDepth(32)
```

### `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```csharp
s.SetTimezone( {{TIMEZONE}} )
```

The timezone should be a string:

```csharp
s.SetTimezone("Europe/London")
```

### `SetLang`

This method lets you pass a user's language in to Snowplow:

```csharp
s.SetLang( {{LANGUAGE}} )
```

The language should be a string:

```csharp
s.SetLang('en')
```

### `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```csharp
s.SetIpAddress( {{IP ADDRESS}} )
```

The IP address should be a string:

```csharp
s.SetIpAddress("127.0.0.1");
```

### `SetUseragent`

This method lets you pass a useragent in to Snowplow:

```csharp
s.SetUseragent( {{USERAGENT}} )
```

The useragent should be a string:

```csharp
s.SetUseragent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0");
```

### `SetNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```csharp
s.SetNetworkUserId( {{NUID}} )
```

The network user id should be a string, typically a UUID:

```csharp
s.SetNetworkUserId("a57c0cac-4aab-49c5-8e70-790f415080b0");
```

### `SetDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```csharp
s.SetDomainUserId( {{DUID}} )
```

The domain user id should be a string, typically a UUID:

```csharp
s.SetDomainUserId("be0999db-8e4e-4f0c-9f3e-fe5e5de1669a");
```

### `SetDomainSessionId`

This method lets you pass a Domain Session ID in to Snowplow:

```csharp
s.SetDomainSessionId( {{SID}} )
```

The domain session id should be a string, typically a UUID:

```csharp
s.SetDomainSessionId("8b107139-5b38-45cc-a0eb-be2550e32904");
```

### `SetDomainSessionIndex`

This method lets you pass a Domain Session Index in to Snowplow:

```csharp
s.SetDomainSessionIndex( {{VID}} )
```

The domain user id should be an int:

```csharp
s.SetDomainSessionIndex(1);
```

### `SetPlatform`

This method lets you set the Platform that the Tracker is running on:

```csharp
s.SetPlatform( Platform.{{ option }} )
```

The Platform should be an enum:

```csharp
s.SetPlatform(Platform.Mob);
```

Available platforms:

- Web
- Mob
- Pc
- Srv
- App
- Tv
- Cnsl
- Iot

---

# Configure a tracker with the .NET tracker
> Configure the .NET tracker instance with emitter, subject, session, and logging options for coordinating event tracking.
> Source: https://docs.snowplow.io/docs/sources/net-tracker/tracker/

The Tracker object is responsible for co-ordinating the saving and sending of events as well as managing the optional Session object.

## Constructor

| **Argument Name**            | **Description**                                                          | **Required?** | **Default** |
| ---------------------------- | ------------------------------------------------------------------------ | ------------- | ----------- |
| `emitter`                    | The Emitter object you create                                            | Yes           | Null        |
| `subject`                    | The Subject that defines a user                                          | No            | Null        |
| `clientSession`              | The Session object you create                                            | No            | Null        |
| `trackerNamespace`           | The name of the tracker instance                                         | No            | Null        |
| `appId`                      | The application ID                                                       | No            | Null        |
| `base64Encoded`              | If we [base 64 encode](https://en.wikipedia.org/wiki/Base64) json values | No            | True        |
| `synchronous`                | If loading into storage is done in sync                                  | No            | True        |
| `desktopContextDelegate`     | Function to get the desktop context                                      | No            | Null        |
| `mobileContextDelegate`      | Function to get the mobile context                                       | No            | Null        |
| `geoLocationContextDelegate` | Function to get the geo-location context                                 | No            | Null        |
| `logger`                     | The logger to use within the application                                 | No            | Null        |

A full Tracker construction should look like the following:

```csharp
var logger = new ConsoleLogger();
var endpoint = new SnowplowHttpCollectorEndpoint(emitterUri, method: method, port: port, protocol: protocol, l: logger);
var storage = new LiteDBStorage("events.db");
var queue = new PersistentBlockingQueue(storage, new PayloadToJsonString());
var emitter = new AsyncEmitter(endpoint, queue, l: logger);
var subject = new Subject().SetPlatform(Platform.Mob).SetLang("EN");
var session = new ClientSession("client_session.dict", l: logger);

Tracker.Instance.Start(emitter: emitter, subject: subject, clientSession: session, trackerNamespace: "some namespace", appId: "some appid", encodeBase64: true, l: logger);
```

The `Tracker.Start(...)` and `Tracker.Stop()` methods take full responsibility for starting and stopping the threads required for processing everything asynchronously. **Do not** use any other `Start` and `Stop` functions other than the ones directly for the Tracker to prevent unknown behaviors.

**WARNING**: The `LiteDBStorage` object must be disposed of manually after stopping the Tracker so you will need to maintain a reference to this object.

**NOTE**: The `Subject` variables can all be altered directly from the Tracker via replicated setter methods.

## Functions

The Tracker contains several critical functions that must be used to start Tracking.

### `Start(...)`

This function must be called before any events will start being stored or sent. This is due to the fact that we do not want to start any background processing from the constructors so it is left up to the developer to choose when to start everything up.

If you attempt to access the Tracker singleton before Starting it an exception will be thrown.

This function:

- Starts the background emitter thread
- Starts the background session check timer (Optional)

Once this is run everything should be in place for asynchronous event tracking.

### `Stop()`

If you need to halt the Tracker from tracking events you can run this function. This will bring all event processing, sending and collection to a halt and nothing will be started again until `Start(...)` is fired again.

**WARNING**: If you are using Client Sessionization stopping and then restarting the Tracker will result in the session index incrementing.

### `Track(IEvent, Subject)`

This is the function used for Tracking all events.

```csharp
Tracker.Instance.Track(IEvent newEvent, Subject eventSubject = null);
```

---

# Configure the Node.js tracker
> Configure the Node.js tracker with platform, user ID, session properties, and device information for enriched event tracking.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/configuration/

You may have additional information about your application's environment, current user and so on, which you want to send to your Snowplow pipeline with each event.

#### Setting event properties

The `tracker` instance has a set of `set...()` methods to attach extra data to all tracked events:

- [Set the platform ID with `setPlatform()`](#set-the-platform-id-withsetplatform)
- [Set user ID with `setUserId()`](#set-user-id-withsetuserid)
- [Set domain user ID with `setDomainUserId()`](#set-domain-user-id-with-setdomainuserid)
- [Set session ID with `setSessionId()`](#set-session-id-with-setsessionid)
- [Set session index with `setSessionIndex()`](#set-session-index-with-setsessionindex)
- [Set network user ID with `setNetworkUserId()`](#set-network-user-id-with-setnetworkuserid)
- [Set screen resolution with `setScreenResolution()`](#set-screen-resolution-with-setscreenresolution)
- [Set viewport dimensions with `setViewport()`](#set-viewport-dimensions-withsetviewport)
- [Set color depth with `setColorDepth()`](#set-color-depth-withsetcolordepth)
- [Set the timezone with `setTimezone()`](#set-the-timezone-withsettimezone)
- [Set the language with `setLang()`](#set-the-language-withsetlang)
- [Set the IP Address with `setIpAddress()`](#set-the-ip-address-withsetipaddress)
- [Set the Useragent with `setUseragent()`](#set-the-useragent-withsetuseragent)

### Set the platform ID with `setPlatform()`

You can set the platform:

```javascript
t.setPlatform('{{PLATFORM}}');
```

Example:

```javascript
t.setPlatform('mob');
```

If the platform is not set manually, it defaults to `'srv'` (for server).

For a full list of supported platforms, please see the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields).

### Set user ID with `setUserId()`

You can set the user ID to any string:

```javascript
t.setUserId('{{USER ID}}');
```

Example:

```javascript
t.setUserId('alexd');
```

### Set domain user ID with `setDomainUserId()`

You can set the domain user ID to any string:

```javascript
t.setDomainUserId('{{DOMAIN USER ID}}');
```

Example:

```javascript
t.setDomainUserId('e8091074-f197-4279-a92e-71a34171d477');
```

### Set session ID with `setSessionId()`

You can set the session ID to any string:

```javascript
t.setSessionId('{{SESSION ID}}');
```

Example:

```javascript
t.setSessionId('s8091074-f197-4279-a92e-71a34171d477');
```

### Set session index with `setSessionIndex()`

You can set the session index to any number:

```javascript
t.setSessionIndex('{{SESSION INDEX}}');
```

Example:

```javascript
t.setSessionIndex(10);
/* t.setSessionIndex('10'); will also work as long as input string can be converted to a valid number. */
```

### Set network user ID with `setNetworkUserId()`

You can set the network user ID to any string:

```javascript
t.setNetworkUserId('{{NETWORK USER ID}}');
```

Example:

```javascript
t.setNetworkUserId('b98c76ad-e45c-45db-a101-224718064ac2');
```

### Set screen resolution with `setScreenResolution()`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```javascript
t.setScreenResolution( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```javascript
t.setScreenResolution(1366, 768);
```

### Set viewport dimensions with `setViewport()`

If your code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```javascript
t.setViewport( {{WIDTH}}, {{HEIGHT}} );
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```javascript
t.setViewport(300, 200);
```

### Set color depth with `setColorDepth()`

If your code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```javascript
t.setColorDepth( {{BITS PER PIXEL}} );
```

The number should be a positive integer, in bits per pixel. Example:

```javascript
t.setColorDepth(32);
```

### Set the timezone with `setTimezone()`

This method lets you pass a user's language in to Snowplow:

```javascript
t.setTimezone( {{TIMEZONE}} );
```

Example:

```javascript
t.setTimezone('Europe/London');
```

### Set the language with `setLang()`

This method lets you pass a user's language in to Snowplow:

```javascript
t.setLang( {{LANGUAGE}} );
```

The number should be a positive integer, in bits per pixel. Example:

```javascript
t.setLang('en');
```

### Set the IP Address with `setIpAddress()`

This method lets you set an IP Address on your events:

```javascript
t.setIpAddress( {{IP ADDRESS}} );
```

Example:

```javascript
t.setIpAddress('192.168.0.1');
```

### Set the Useragent with `setUseragent()`

This method lets you override the default Useragent on your events:

```javascript
t.setUseragent( {{USER AGENT}} );
```

Example:

```javascript
t.setUseragent('myapp/0.1.0');
```

---

# Node.js tracker SDK
> Track events in Node.js applications with the Snowplow Node.js tracker SDK written in TypeScript.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/

The Snowplow Node.js Tracker allows you to track Snowplow events from your Node.js applications.

The tracker should be straightforward to use if you are comfortable with JavaScript development; any prior experience with other Snowplow trackers is helpful but not necessary. It is written in TypeScript.

The Node.js tracker, along with the [web trackers](/docs/sources/web-trackers/), are part of [one monorepo](https://github.com/snowplow/snowplow-javascript-tracker).

The current version is 4.6.8.

---

# Install and initialize the Node.js tracker
> Install and initialize the Node.js tracker using npm with emitter configuration for sending events to Snowplow collectors.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/initialization/

The Snowplow Node.js Tracker is tested and compatible with Node.js versions from 18 to 20. Installation requires [npm](https://www.npmjs.org/), [pnpm](https://pnpm.js.org/) or [yarn](https://yarnpkg.com/).

## Installation

Setting up the tracker should be straightforward if you are familiar with npm:

```bash
npm install @snowplow/node-tracker
```

alternative you can use pnpm or yarn:

```bash
pnpm add @snowplow/node-tracker
```

```bash
yarn add @snowplow/node-tracker
```

The Snowplow Node.js Tracker is also **bundled with TypeScript types and interfaces** so will integrate easily with TypeScript applications.

## Initialization

Require the Node.js Tracker module into your code like so:

```javascript
const snowplow = require('@snowplow/node-tracker');
const tracker = snowplow.newTracker( /* ... */ );
```

or, if using ES Modules, you can import the module like so:

```javascript
import { newTracker } from '@snowplow/node-tracker';
```

The `newTracker` call takes 2 arguments – the tracker and emitter configuration. The tracker configuration specifies the tracker namespace and app name. The emitter configuration specifies how events are queued locally and sent to the Snowplow collector.

```javascript
const tracker = newTracker({
  namespace: "my-tracker", // Namespace to identify the tracker instance. It will be attached to all events which the tracker fires.
  appId: "my-app", // Application identifier
  encodeBase64: false, // Whether to use base64 encoding for the self-describing JSON. Defaults to true.
}, {
  endpoint: "https://collector.mydomain.net", // Collector endpoint
  eventMethod: "post", // Method - defaults to POST
  bufferSize: 1, // Only send events once n are buffered. Defaults to 1 for GET requests and 10 for POST requests.
});
```

There are a number of additional parameters that may optionally be configured. Please refer to the following API docs for the full list:

1. [Tracker configuration options](https://snowplow.github.io/snowplow-javascript-tracker/docs/node-tracker/markdown/node-tracker.trackerconfiguration).
2. [Emitter configuration options](https://snowplow.github.io/snowplow-javascript-tracker/docs/node-tracker/markdown/node-tracker.emitterconfigurationbase).

### Using multiple Emitters

In case you want to send the events to multiple Snowplow collectors, you can provide a list of emitter configurations in the `newTracker` call – one for each collector. This may look as follows.

```javascript
const tracker = newTracker({
  namespace: "my-tracker", // Namespace to identify the tracker instance. It will be attached to all events which the tracker fires.
  appId: "my-app", // Application identifier
  encodeBase64: false, // Whether to use base64 encoding for the self-describing JSON. Defaults to true.
}, [
  {
    endpoint: "https://collector1.mydomain.net", // First collector endpoint
  },
  {
    endpoint: "https://collector2.mydomain.net", // Second collector endpoint
  },
]);
```

### Create your own Emitter

Emitter is an object responsible for queuing tracked events and making requests to the Snowplow collector.

By default, the Node tracker makes use of the [fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) to make the HTTP requests. If you want to use a different library or logic for queuing and sending events, you can provide a custom `Emitter` implementation.

Emitters must conform to an [`Emitter` interface](https://github.com/snowplow/snowplow-javascript-tracker/blob/master/trackers/node-tracker/docs/markdown/node-tracker.emitter.md), which looks like:

```typescript
/**
 * Emitter is responsible for sending events to the collector.
 * It manages the event queue and sends events in batches depending on configuration.
 */
interface Emitter {
    /**
     * Forces the emitter to send all events in the event store to the collector.
     * @returns A Promise that resolves when all events have been sent to the collector.
     */
    flush: () => Promise<void>;
    /**
     * Adds a payload to the event store or sends it to the collector.
     * @param payload - A payload to be sent to the collector
     * @returns Promise that resolves when the payload has been added to the event store or sent to the collector
     */
    input: (payload: Payload) => Promise<void>;
    /**
     * Updates the collector URL to which events will be sent.
     * @param url - New collector URL
     */
    setCollectorUrl: (url: string) => void;
    /**
     * Sets the server anonymization flag.
     */
    setAnonymousTracking: (anonymous: boolean) => void;
    /**
     * Updates the buffer size of the emitter.
     */
    setBufferSize: (bufferSize: number) => void;
}
```

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

Once you implement your own `Emitter`, you can pass it to the `newTracker` call as follows.

```ts
export const expressTracker = newTracker(
  {
    namespace: "my-tracker",
    appId: "my-app",
  },
  {
    customEmitter: () => ({
      input: (payload) => {
        return Promise.resolve();
      },
      flush: () => {
        return Promise.resolve();
      },
      setCollectorUrl: (url) => {},
      setAnonymousTracking: (anonymous) => {},
      setBufferSize: (bufferSize) => {},
    }),
  }
);
```

---

# Migration guides for the Node.js tracker
> Upgrade guides to help you migrate between major versions of the Node.js tracker.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the Node.js tracker.

---

# Migration guide for Node.js tracker v3 to v4
> Migration guide for upgrading the Node.js tracker from version 3 to version 4 with breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/migration-guides/v3-to-v4-migration-guide/

Version 4 is a major release with several breaking changes to defaults and behavior.

## Behavior changes

### Removed the use of the got library in favor of fetch

The tracker no longer needs the got library for making HTTP requests to the collector. This has been replaced with the [fetch API](https://nodejs.org/dist/latest-v18.x/docs/api/globals.html). The change has an effect on the public API of the tracker which had to previously accommodate the got library – the following change results from this.

### Removed `tracker` and `emitter` functions in favor or `newTracker`

The initialization of a tracker instance has changed – instead of initializing an emitter (using `gotEmitter`) and tracker (using `tracker`) separately, the SDK handles initialization of both using the `newTracker` call. The `newTracker` call now accepts configuration objects for the tracker and emitter as arguments.

[See the new initialization options here.](/docs/sources/node-js-tracker/initialization/)

### Dropped support for older Node.JS

The support for the following Node.JS versions has been dropped:

- Drop Node.js <18 support

---

# Track events with the Node.js tracker
> Track page views, screen views, structured events, and custom self-describing events with the Node.js tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/node-js-tracker/tracking-events/

Tracking methods supported by the Node.js tracker at a glance:

| **Function**                                                                               | **Description**                                        |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| [`track()`](#track-events-with-the-track-function)                                         | Tracks an event generated from a build functions       |
| [`buildScreenView()`](#track-screen-views-withbuildscreenview)                             | Track the user viewing a screen within the application |
| [`buildPageView()`](#track-pageviews-withbuildpageview)                                    | Track and record views of web pages.                   |
| [`buildSelfDescribingEvent()`](#track-self-describing-events-withbuildselfdescribingevent) | Track a Snowplow custom self describing event          |
| [`buildStructEvent()`](#track-structured-events-with-buildstructevent)                     | Track a Snowplow custom structured event               |

### Track events with the track() function

All events are tracked through the `track()` function on the tracker instance. The type of event is determined by one of the event builder functions, which all begin as `build*()`.

#### Optional arguments

Each build method which is passed to the `track()` function has both default and optional arguments. If you don't want to provide a value for an optional argument, just ignore it in the object or pass `null` and it will be ignored. For example, if you want to track a page view event with a referrer but without a title:

```javascript
t.track(buildPageView({
  pageUrl: 'http://www.example.com',
  referrer: 'http://www.referer.com'
});
```

#### Custom contexts

In short, custom contexts let you add additional information about the circumstances surrounding an event. Each call to `track()` accepts an additional optional context parameter:

```javascript
t.track(buildPageView({
  pageUrl: 'http://www.example.com',
  referrer: 'http://www.referer.com'
}),
[]); // Array of custom contexts
```

The `context` argument should consist of an array of zero or more context objects. Each object should be a self-describing JSON following the same pattern as an self describing event.

**Important:** Even if only one custom context is being attached to an event, it still needs to be wrapped in an array.

If a visitor arrives on a page advertising a movie, the context dictionary might look like this:

```json
{
  "schema": "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
  "data": {
    "movie_name": "Solaris",
    "poster_country": "JP"
  }
}
```

This is how to fire a page view event with the above custom context:

```javascript
t.track(buildPageView({
  pageUrl: 'http://www.example.com',
  referrer: 'http://www.referer.com'
}),
[{
  "schema": "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
  "data": {
    "movie_name": "Solaris",
    "poster_country": "JP"
  }
}]);
```

Note that even though there is only one custom context attached to the event, it still needs to be placed in an array.

#### Timestamps

Each `track()` call supports an optional timestamp as its final argument, after the context argument; this allows you to manually override the timestamp attached to this event.

If you do not pass this timestamp in as an argument, then the Tracker will use the current time to be the timestamp for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument:

```javascript
t.track(buildStructEvent({
  category: "game action",
  action: "save"
}), [], 1368725287000);
```

Timestamp is counted in milliseconds since the Unix epoch - the same format as generated by `new Date().getTime()`.

### Track screen views with `buildScreenView()`

Use `buildScreenView()` to build a user viewing a screen (or equivalent) within your app. Arguments are:

| **Argument** | **Description**                     | **Required?** | **Type**         |
| ------------ | ----------------------------------- | ------------- | ---------------- |
| `name`       | Human-readable name for this screen | No            | Non-empty string |
| `id`         | Unique identifier for this screen   | No            | String           |

`name` and `id` are not individually required, but you must provide at least one of them.

Example:

```javascript
t.track(buildScreenView({
  name: "HUD > Save Game",
  id: "screen23"
}));
```

### Track pageviews with `buildPageView()`

Use `buildPageView()` to track a user viewing a page within your app. Arguments are:

| **Argument** | **Description**                      | **Required?** | **Type**         |
| ------------ | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`    | The URL of the page                  | Yes           | Non-empty string |
| `pageTitle`  | The title of the page                | No            | String           |
| `referrer`   | The address which linked to the page | No            | String           |

Example:

```javascript
t.track(buildPageView({
  pageUrl: "www.example.com",
  url: "example",
  referrer: "www.referrer.com"
}));
```

### Track self describing events with `buildSelfDescribingEvent()`

Use `buildSelfDescribingEvent()` to track a custom event which consists of a name and an unstructured set of properties. This is useful when you want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow).

The arguments are as follows:

| **Argument** | **Description**                      | **Required?** | **Type** |
| ------------ | ------------------------------------ | ------------- | -------- |
| `event`      | The self describing event definition | Yes           | JSON     |

Example:

```javascript
t.track(buildSelfDescribingEvent({
  event: {
    schema: "iglu:com.example_company/save-game/jsonschema/1-0-2",
    data: {
      save_id: "4321",
      level: 23,
      difficultyLevel: "HARD",
      dl_content: true
    }
  }
}));
```

The `event` property must be an object with two fields: `schema` and `data`. `data` is an object containing the properties of the self describing event. `schema` identifies the JSON schema against which `data` should be validated.

### Track structured events with `buildStructEvent()`

Use `buildStructEvent()` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument** | **Description**                                                  | **Required?** | **Type**         |
| ------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | Non-empty string |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | Non-empty string |
| `label`      | A string to provide additional dimensions to the event data      | No            | String           |
| `property`   | A string describing the object or the action performed on it     | No            | String           |
| `value`      | A value to provide numerical data about the event                | No            | Number           |

Example:

```javascript
t.track(buildStructEvent({
  category: "shop",
  action: "add-to-basket",
  label: "product",
  property: "pcs",
  value: 2
}));
```

---

# Configure emitters in the PHP tracker
> Configure sync, socket, curl, and file emitters for sending events to Snowplow collectors with GET or POST requests.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/emitters/

We currently support four different emitters: sync, socket, curl, and an out-of-band file emitter. The most basic emitter only requires you to select the type of emitter to be used and specify the Collector's hostname as parameters.

All emitters support both `GET` and `POST` as methods for sending events to Snowplow Collectors.

For the sake of performance, we recommend using `POST` as the tracker can then batch many events together into a single request. Note that depending on your pipeline architecture, your Collector may have limits on the maximum request size it accepts that could be exceeded by large batch sizes.

It is recommended that after you have finished logging all of your events to call the following method:

```php
$tracker->flushEmitters();
```

This empties the event buffers of all emitters associated with your tracker object and sends any left over events. In future releases, this may be an automatic process but for now, it remains manual.

### Sync

The Sync emitter is a basic synchronous emitter which supports both `GET` and `POST` request types.

By default, this emitter uses the Request type POST, HTTP, and a buffer size of 50.

As of version 0.7.0, the emitter has the capability to retry failed requests. In case connection to the Collector can't be established or the request fails with a 4xx (except for 400, 401, 403, 410, 422) or 5xx status code, the same request is retried. The number of times a request should be retried is configurable but defaults to 1. The default back-off period between subsequent retries, starts with 100 ms (configurable) and increases exponentially.

Example emitter creation:

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 50);
```

Whilst you can force the buffer size to be greater than 1 for a GET Request; it doesn't yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $buffer_size = NULL, $debug = false, $max_retry_attempts = NULL, $retry_backoff_ms = NULL, $server_anonymization = false)
```

Arguments:

| **Argument**            | **Description**                                                                                                                     | **Required?** | **Validation**   |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                                                                                                  | Yes           | Non-empty string |
| `$protocol`             | Collector Protocol (HTTP or HTTPS)                                                                                                  | No            | String           |
| `$type`                 | Request Type (POST or GET)                                                                                                          | No            | String           |
| `$buffer_size`          | Amount of events to store before flush                                                                                              | No            | Int              |
| `$debug`                | Whether or not to log errors                                                                                                        | No            | Boolean          |
| `$max_retry_attempts`   | The maximum number of times to retry a request. Defaults to 1.                                                                      | No            | Int              |
| `$retry_backoff_ms`     | The number of milliseconds to backoff before retrying a request. Defaults to 100 ms, increases exponentially in subsequent retries. | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events; IP and Network User ID information isn't associated with tracked events                | No            | Int              |

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

### Socket

The Socket emitter allows for the much faster transmission of Requests to the Collector by allowing us to write data directly to the HTTP socket. However, this solution is still, in essence, a synchronous process and blocks the execution of the main script.

As of version 0.7.0, the emitter has the capability to retry failed requests. In case connection to the Collector can't be established or the request fails with a 4xx (except for 400, 401, 403, 410, 422) or 5xx status code, the same request is retried. The number of times a request should be retried is configurable but defaults to 1. The default back-off period between subsequent retries, starts with 100 ms (configurable) and increases exponentially.

Example Emitter creation:

```php
$emitter = new SocketEmitter($collector_uri, NULL, "GET", NULL, NULL);
```

Whilst you can force the buffer size to be greater than 1 for a GET Request; it does not yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $ssl = NULL, $type = NULL, $timeout = NULL, $buffer_size = NULL, $debug = NULL, $max_retry_attempts = NULL, $retry_backoff_ms = NULL)
```

Arguments:

| **Argument**            | **Description**                                                                                                                     | **Required?** | **Validation**   |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                                                                                                  | Yes           | Non-empty string |
| `$ssl`                  | Whether to use SSL encryption                                                                                                       | No            | Boolean          |
| `$type`                 | Request Type (POST or GET)                                                                                                          | No            | String           |
| `$timeout`              | Socket Timeout Limit                                                                                                                | No            | Int or Float     |
| `$buffer_size`          | Amount of events to store before flush                                                                                              | No            | Int              |
| `$debug`                | Whether or not to log errors                                                                                                        | No            | Boolean          |
| `$max_retry_attempts`   | The maximum number of times to retry a request. Defaults to 1.                                                                      | No            | Int              |
| `$retry_backoff_ms`     | The number of milliseconds to backoff before retrying a request. Defaults to 100 ms, increases exponentially in subsequent retries. | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events; IP and Network User ID information isn't associated with tracked events                | No            | Int              |

### Curl

The Curl Emitter allows us to have the closest thing to native asynchronous requests in PHP. The curl emitter uses the `curl_multi_init` resource which allows us to send any number of requests asynchronously. This garners a performance gain over the sync and socket emitters as we can now send more than one request at a time.

On top of this, we are also using a modified version of this **[Rolling Curl library](https://github.com/joshfraser/rolling-curl)** for the actual sending of the curl requests. This allows for a more efficient implementation of asynchronous curl requests as we can now have multiple requests sending at the same time, and in addition as soon as one is done a new request is started.

> **Note:** The emitter doesn't retry failed requests to the Collector. Failed requests to the Collector (for example due to it being not reachable) result in lost events.

Example Emitter creation:

```php
$emitter = new CurlEmitter($collector_uri, false, "GET", 2);
```

Whilst you can force the buffer size to be greater than 1 for a GET request, it doesn't yield any performance changes as we can still only send 1 event at a time.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $buffer_size = NULL, $debug = false, $curl_timeout = NULL)
```

Arguments:

| **Argument**            | **Description**                                         | **Required?** | **Validation**   |
| ----------------------- | ------------------------------------------------------- | ------------- | ---------------- |
| `$uri`                  | Collector hostname                                      | Yes           | Non-empty string |
| `$protocol`             | Collector Protocol (HTTP or HTTPS)                      | No            | String           |
| `$type`                 | Request Type (POST or GET)                              | No            | String           |
| `$buffer_size`          | Amount of events to store before flush                  | No            | Int              |
| `$debug`                | Whether or not to log errors                            | No            | Boolean          |
| `$curl_timeout`         | Maximum time the request is allowed to take, in seconds | No            | Int              |
| `$server_anonymization` | Enable Server Anonymization for sent events             | No            | Int              |

#### Curl default settings

The internal emitter default settings are as follows:

- Rolling Window (max number of concurrent requests)

  - POST: 10
  - GET: 30

- Curl Buffer (number of times we need to hit the emitters buffer size before sending)

  - POST: 50
  - GET: 250

Since version 0.8 of the PHP tracker, you can change these settings using the following setter functions:

- `$curl_emitter.setCurlAmount($curl_amount)`: update the curl buffer size (number of times we need to reach the buffer size before we initiate sending)
- `$curl_emitter.setRollingWindow($rolling_window)`: update the rolling window configuration (max number of concurrent requests)

### File

> **Warning:** When running under Windows, PHP can't spawn truly separate processes, and slowly eats more and more resources when more processes are spawned. Thus, Windows might crash under high load when using the File Emitter.

The File Emitter is the only true non-blocking solution. The File Emitter works via spawning workers which grab created files of logged events from a local temporary folder. The workers then load the events using the same asynchronous curl properties from the above emitter.

All of the worker processes are created as background processes so none of them delay the execution of the main script. Currently, they're configured to look for files inside created worker folders until there are none left and they hit their `timeout` limit, at which point the process terminates itself.

If the worker for any reason fails to successfully send a request it renames the entire file to `failed` and leaves it in the `/temp/failed-logs/` folder.

> **Note:** The emitter doesn't retry failed requests to the Collector. Failed requests to the Collector (for example due to it being not reachable) result in lost events.

Example Emitter creation:

```php
$emitter = new FileEmitter($collector_uri, false, "POST", 2, 15, 100, "/tmp/snowplow");
```

The buffer for the file emitter works a bit differently to the other emitters in that here it refers to the number of events needed before an `events-random.log` is produced for a worker. If you are anticipating it taking a long time to reach the buffer be aware that the worker terminates itself after 75 seconds by default (15 x 5). Adjust the timeout amount in the construction of the FileEmitter if the default isn't suitable.

Constructor:

```php
public function __construct($uri, $protocol = NULL, $type = NULL, $workers = NULL, $timeout = NULL, $buffer_size = NULL, $debug = false, $log_dir = NULL)
```

Arguments:

| **Argument**   | **Description**                                                            | **Required?** | **Validation**   |
| -------------- | -------------------------------------------------------------------------- | ------------- | ---------------- |
| `$uri`         | Collector hostname                                                         | Yes           | Non-empty string |
| `$protocol`    | Collector Protocol (HTTP or HTTPS)                                         | No            | String           |
| `$type`        | Request Type (POST or GET)                                                 | No            | String           |
| `$workers`     | Amount of background workers                                               | No            | Int              |
| `$timeout`     | Worker Timeout                                                             | No            | Int or Float     |
| `$buffer_size` | Amount of events to store before flush                                     | No            | Int              |
| `$debug`       | Whether or not to log errors                                               | No            | Boolean          |
| `$log_dir`     | The directory for event log and worker log subdirectories to be created in | No            | String           |

### Emitter debug mode

Debug mode is enabled on emitters by setting the `$debug` argument in the emitter constructor to `true`:

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 50, true);
```

By default, debug mode creates a new directory called `/debug/` in the root of the tracker's directory. It then creates a log file with the following structure; `sync-events-log-[[random number]].log`: which is the type of emitter and a randomized number to prevent it from being accidentally overwritten.

If physically storing the information isn't possible due to not having the correct write permissions or simply not wanted it can be turned off by updating the following value in the Constants class:

```php
const DEBUG_LOG_FILES = false;
```

Now all debugging information is printed to the console.

Every time the events buffer is flushed we can see if the flush was successful. In the case of an error, it records the entire event payload the tracker was trying to send, along with the error code.

#### Event specific information

If debug mode is enabled the emitter begins storing information internally. It stores the HTTP response code and the payload for every request made by the emitter.

```php
array(
    "code" => 200,
    "data" => "{"e":"pv","url":"www.example.com","page":"example","refr":"www.referrer.com"}"
)
```

The `data` is stored as a JSON-encoded string. To locally test whether or not your emitters are successfully sending, we can retrieve this information with the following commands:

```php
$emitters = $tracker->returnEmitters(); # Store all of the emitters as an array.
$emitter = $emitters[0]; # Get the first emitter stored by the tracker
$results = $emitter->returnRequestResults();  # Return the stored results.

# Now that we have results we can work with...
print("Code: ".$results[0]["code"]);
print("Data: ".$results[0]["data"]);
```

This allows you to debug on a request by request basis to ensure that everything is being sent properly.

#### Turn debug off

As debugging stores a lot of information, we can end debug mode by calling the following command:

```php
$tracker->turnOffDebug();
```

This stops all logging activity, both to the external files and to the local arrays. We can go one step further though and pass a `true` boolean to the function. This deletes all of the tracker's associated physical debug log files as well as emptying the local arrays within each linked emitter.

```php
$tracker->turnOffDebug(true);
```

---

# PHP tracker SDK
> Track events in PHP applications and scripts with the Snowplow PHP tracker SDK using trackers, subjects, and emitters.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/

The Snowplow PHP Tracker allows you to track Snowplow events from your PHP apps and scripts.

There are three basic types of object you will create when using the Snowplow PHP Tracker: Trackers, Subjects and Emitters.

A subject represents a user whose events are tracked. A tracker constructs events and sends them to one or more emitters. Each emitter then sends the event to the endpoint you configure, a Snowplow collector.

The current flow of the PHP Tracker is illustrated below:

![](/assets/images/php-tracker-flow-0a55defa12a400416329dcf1e75a4143.png)

---

# Initialize the PHP tracker
> Initialize the PHP tracker with emitter, subject, namespace, and encoding configuration for event tracking.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/initialization/

To instantiate a new Tracker instance we need to make sure the Snowplow Tracker classes are available.

Include these class aliases in your project:

```php
use Snowplow\Tracker\Tracker;
use Snowplow\Tracker\Subject;
use Snowplow\Tracker\Emitters\SyncEmitter;
```

We can now create our Emitter, Subject and Tracker objects.

### Creating a Tracker

The most basic Tracker instance will only require you to provide an [Emitter](/docs/sources/php-tracker/emitters/) and the data [Subject](/docs/sources/php-tracker/subjects/) for which the Tracker will log events.

```php
$emitter = new SyncEmitter($collector_uri, "http", "POST", 10, false);
$subject = new Subject();
$tracker = new Tracker($emitter, $subject);
```

Other Tracker arguments:

| **Argument Name** | **Description**                                                            | **Required?** | **Default** |
| ----------------- | -------------------------------------------------------------------------- | ------------- | ----------- |
| `emitters`        | The emitter to which events are sent                                       | Yes           | `None`      |
| `subject`         | The user being tracked                                                     | Yes           | `Subject()` |
| `namespace`       | The name of the tracker instance                                           | No            | `None`      |
| `app_id`          | The application ID                                                         | No            | `None`      |
| `encode_base64`   | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `True`      |

Another example using all of the allowed arguments:

```php
$tracker = new Tracker($emitter, $subject, "cf", "cf29ea", true);
```

#### `emitters`

This can be either a single emitter or an array of emitters. The tracker will send events to all of these emitters, which will, in turn, send them on to a collector.

```php
$emitter1 = new SyncEmitter($collector_uri);
$emitter2 = new CurlEmitter($collector_uri, false, "GET", 2);

$emitter_array = array($emitter1, $emitter2);

// Tracker Init
$subject = new Subject();
$tracker1 = new Tracker($emitter1, $subject); # Single Emitter
$tracker2 = new Tracker($emitter_array, $subject); # Array of Emitters
```

#### `subject`

The user which the Tracker will track. This will give your events user-specific data such as timezone and language. You change the subject of your tracker at any time by calling `updateSubject($new_subject_object)`. All events sent from this Tracker will now have the new subject information attached.

#### `namespace`

If provided, the `namespace` argument will be attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

#### `app_id`

The `app_id` argument lets you set the application ID to any string.

#### `encode_base64`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the `encode_base64` argument.

---

# Install the PHP tracker
> Install the PHP tracker using Composer for tracking events in PHP applications.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/setup/

Currently the only supported method of installation is through Composer. For a tutorial on setting up a PHP project with Composer please follow this [link](https://getcomposer.org/doc/00-intro.md).

Add the Snowplow PHP Tracker to your project by running:

```bash
composer require snowplow/snowplow-tracker
```

to include it in your `composer.json` file as a dependency.

You can also add it manually:

```json
{
  "require": {
      "snowplow/snowplow-tracker": "0.9.2"
  }
}
```

Assuming you have Composer setup correctly in the root of your project. Type the following command line argument:

```bash
composer update # Will update lockfile and install dependencies
```

This will install the Snowplow Tracker and allow you to initialize a Tracker object:

```php
// Bare minimum Tracker initialization.

use Snowplow\Tracker\Tracker;
use Snowplow\Tracker\Subject;
use Snowplow\Tracker\Emitters\SyncEmitter;

$emitter = new SyncEmitter("collector_uri");
$subject = new Subject();
$tracker = new Tracker($emitter, $subject);
```

---

# Track subject data with the PHP tracker
> Add user and environment data to tracked events using the Subject class with platform, user ID, and device properties.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/subjects/

The Subject object lets you send any additional information about your application's environment, current user, etc to Snowplow.

To create a new subject:

```php
$subject = new Subject();
```

By default the subject has one piece of information in it already, the [platform](/docs/fundamentals/canonical-event/#application-fields) `["p" => "srv"]`.

The Subject class contains a variety of 'set' methods to attach extra data to your event.

- [`setPlatform`](#setplatform)
- [`setUserId`](#setuserid)
- [`setScreenResolution`](#setscreenresolution)
- [`setViewPort`](#setviewport)
- [`setColorDepth`](#setcolordepth)
- [`setTimezone`](#settimezone)
- [`setLanguage`](#setlanguage)
- [`setIpAddress`](#setipaddress)
- [`setUseragent`](#setuseragent)
- [`setNetworkUserId`](#setnetworkuserid)
- [`setDomainUserId`](#setdomainuserid)
- [`setSessionId`](#setsessionid)
- [`setSessionIndex`](#setsessionindex)
- [`setRefr`](#setrefr)
- [`setPageUrl`](#setpageurl)

These set methods can be called either directly onto a subject object:

```php
$subject = new Subject();
$subject->setPlatform("tv");
```

Or they can be called through the tracker object:

```php
$tracker->returnSubject()->setPlatform("tv");
```

You can also change the active Subject a Tracker is operating with:

```php
$subject2 = new Subject();
$tracker->updateSubject($subject2);
```

### `setPlatform`

The default platform is "srv". You can change the platform of the subject by calling:

```php
$subject->setPlatform($platform);
```

For example:

```php
$subject->setPlatform("tv") # Running on a Connected TV
```

For a full list of supported platforms, please see the [Event Parameters reference](/docs/fundamentals/canonical-event/#application-fields).

### `setUserId`

You can set the user ID to any string:

```php
$subject->setUserId($id);
```

Example:

```php
$subject->setUserId("jbeem");
```

### `setScreenResolution`

If your PHP code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```php
$subject->setScreenResolution($width, $height);
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```php
$subject->setScreenResolution(1366, 768);
```

### `setViewPort`

If your PHP code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```php
$subject->setViewPort($width, $height);
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```php
$subject->setViewPort(300, 200);
```

### `setColorDepth`

If your PHP code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```php
$subject->setColorDepth($depth);
```

The number should be a positive integer, in bits per pixel. Example:

```php
$subject->setColorDepth(32);
```

### `setTimezone`

This method lets you pass a user's timezone in to Snowplow:

```php
$subject->setTimezone($timezone);
```

The timezone should be a string:

```php
$subject->setTimezone("Europe/London");
```

### `setLanguage`

This method lets you pass a user's language in to Snowplow:

```php
$subject->setLanguage($language);
```

The language should be a string:

```php
$subject->setLanguage('en');
```

### `setIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```php
$subject->setIpAddress($ip);
```

The IP Address should be a string:

```php
$subject->setIpAddress('127.0.0.1');
```

### `setUseragent`

This method lets you pass a useragent in to Snowplow:

```php
$subject->setUseragent($useragent);
```

The useragent should be a string:

```php
$subject->setUseragent('Agent Smith');
```

### `setNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```php
$subject->setNetworkUserId($networkUserId);
```

The network user id should be a string:

```php
$subject->setNetworkUserId("network-id");
```

### `setDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```php
$subject->setDomainUserId($domainUserId);
```

The domain user id should be a string:

```php
$subject->setDomainUserId("domain-id");
```

### `setSessionId`

This method lets you pass a session ID in to Snowplow:

```php
$subject->setSessionId($sessionId);
```

The session ID should be a string:

```php
$subject->setSessionId("759e1c9a-6b74-403c-8b6f-18eb9f0c2f02");
```

### `setSessionIndex`

This method lets you pass a session index in to Snowplow:

```php
$subject->setSessionIndex($sessionIndex);
```

The session index should be a number:

```php
$subject->setSessionIndex(1);
```

### `setRefr`

This method lets you set the referrer URL to a full URI:

```php
$subject->setRefr("https://example.com/previous-page");
```

### `setPageUrl`

This method lets you set the page URL to a full URI string:

```php
$subject->setPageUrl("https://example.com/current-page");
```

---

# Track an event with the PHP tracker
> Track page views, ecommerce transactions, screen views, and custom events with the PHP tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/php-tracker/tracking-an-event/

Tracking methods supported by the PHP Tracker:

| **Function**                                              | **Description**                                        |
| --------------------------------------------------------- | ------------------------------------------------------ |
| [`trackPageView`](#trackpageview)                         | Track and record views of web pages.                   |
| [`trackEcommerceTransaction`](#trackecommercetransaction) | Track an ecommerce transaction                         |
| [`trackScreenView`](#trackscreenview)                     | Track the user viewing a screen within the application |
| [`trackStructEvent`](#trackstructevent)                   | Track a Snowplow custom structured event               |
| [`trackUnstructEvent`](#trackunstructevent)               | Track a Snowplow custom unstructured event             |

### Optional Tracking Arguments

#### Custom Context

Custom contexts let you add additional information about any circumstances surrounding an event in the form of a PHP Array of name-value pairs. Each tracking method accepts an additional optional contexts parameter after all the parameters specific to that method:

```php
public function trackPageView($page_url, $page_title = NULL, $referrer = NULL, $context = NULL, $tstamp = NULL)
```

An example of a Context Array Structure:

```php
array(
    "schema" => "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
    "data" => array(
        "movie_name" => "Solaris",
        "poster_country" => "JP"
    )
)
```

This is how to fire a page view event with the above custom context:

```php
$tracker->trackPageView(
    "http://www.films.com",
    "Homepage",
    NULL,
    array(
        array(
            "schema" => "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
            "data" => array(
                "movie_name" => "Solaris",
                "poster_country" => "JP"
            )
        )
    )
);
```

#### Timestamp

Each tracking method supports an optional timestamp as its final argument; this allows you to also set the true timestamp (`true_tstamp`) of the event besides the device created timestamp (`dvce_created_tstamp`) that always gets set by the tracker. The optional timestamp argument should be provided in **milliseconds** since the Unix epoch.

Here is an example tracking a structured event and supplying the optional timestamp argument. We can explicitly supply a `NULL` for the intervening arguments which are empty:

```php
$tracker->trackStructEvent("some cat", "save action", NULL, NULL, NULL, 1368725287000);
```

### Event Tracking Methods

#### `trackPageView`

Track a user viewing a page within your app.

Function:

```php
public function trackPageView($page_url, $page_title = NULL, $referrer = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**  | **Description**                      | **Required?** | **Validation**   |
| ------------- | ------------------------------------ | ------------- | ---------------- |
| `$page_url`   | The URL of the page                  | Yes           | Non-empty string |
| `$page_title` | The title of the page                | No            | String           |
| `$referrer`   | The address which linked to the page | No            | String           |
| `$context`    | Custom context for the event         | No            | Array            |
| `$tstamp`     | When the pageview occurred           | No            | Positive integer |

Example Usage:

```php
$tracker->trackPageView("www.example.com", NULL, NULL, NULL, 123123132132);
```

#### `trackEcommerceTransaction`

Track an ecommerce transaction.

Function:

```php
public function trackEcommerceTransaction($order_id, $total_value, $currency = NULL, $affiliation = NULL, $tax_value = NULL, $shipping = NULL, $city = NULL, $state = NULL, $country = NULL, $items = array(), $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**   | **Description**                     | **Required?** | **Validation**   |
| -------------- | ----------------------------------- | ------------- | ---------------- |
| `$order_id`    | ID of the eCommerce transaction     | Yes           | Non-empty string |
| `$total_value` | Total transaction value             | Yes           | Int or Float     |
| `$currency`    | Transaction currency                | No            | String           |
| `$affiliation` | Transaction affiliation             | No            | String           |
| `$tax_value`   | Transaction tax value               | No            | Int or Float     |
| `$shipping`    | Delivery cost charged               | No            | Int or Float     |
| `$city`        | Delivery address city               | No            | String           |
| `$state`       | Delivery address state              | No            | String           |
| `$country`     | Delivery address country            | No            | String           |
| `$items`       | Items in the transaction            | No            | Array            |
| `$context`     | Custom context for the event        | No            | Array            |
| `$tstamp`      | When the transaction event occurred | No            | Positive integer |

Example Usage:

```php

$tracker->trackEcommerceTransaction(
    "test_order_id_1",
    200,
    "GBP",
    "affiliation_1",
    "tax_value_1",
    "shipping_1",
    "city_1",
    "state_1",
    "country_1",
    array(
        array("name" => "name_1","category" => "category_1",
            "price" => 100,"sku" => "sku_1","quantity" => 1),
        array("name" => "name_2","category" => "category_2",
            "price" => 100,"sku" => "sku_2","quantity" => 1)
    )
);
```

The above example contains an order with two order items.

##### `trackEcommerceTransactionItem`

This is a private function that is called from within `trackEcommerceTransaction`. Note that for an item to be added successfully you need to include the following fields in the array, even if the value is `NULL`.

Arguments:

| **Argument** | **Description** | **Required?** | **Validation**   |
| ------------ | --------------- | ------------- | ---------------- |
| `"sku"`      | Item SKU        | Yes           | Non-empty string |
| `"price"`    | Item price      | Yes           | Int or Float     |
| `"quantity"` | Item quantity   | Yes           | Int              |
| `"name"`     | Item name       | No            | String           |
| `"category"` | Item category   | No            | String           |

Example Item:

```php
array(
    array("name" => NULL,
          "category" => NULL,
          "price" => 100,
          "sku" => "sku_1",
          "quantity" => 1)
)
```

If any of these fields are missing the item event will not be created. However the order of these fields is not important.

#### `trackScreenView`

Track a user viewing a screen (or equivalent) within your app.

Function:

```php
public function trackScreenView($name = NULL, $id = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument** | **Description**                     | **Required?** | **Validation**   |
| ------------ | ----------------------------------- | ------------- | ---------------- |
| `$name`      | Human-readable name for this screen | No            | Non-empty string |
| `$id`        | Unique identifier for this screen   | No            | String           |
| `$context`   | Custom context for the event        | No            | Array            |
| `$tstamp`    | When the screen was viewed          | No            | Positive integer |

Although `$name` and `$id` are not individually required, at least one must be provided or the event will fail validation.

Example:

```php
$tracker->trackScreenView("HUD > Save Game", NULL, NULL, 1368725287000);
```

#### `trackStructEvent`

Track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required).

Function:

```php
public function trackStructEvent($category, $action, $label = NULL, $property = NULL, $value = NULL, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument** | **Description**                                                  | **Required?** | **Validation**   |
| ------------ | ---------------------------------------------------------------- | ------------- | ---------------- |
| `$category`  | The grouping of structured events which this `action` belongs to | Yes           | Non-empty string |
| `$action`    | Defines the type of user interaction which this event involves   | Yes           | Non-empty string |
| `$label`     | A string to provide additional dimensions to the event data      | No            | String           |
| `$property`  | A string describing the object or the action performed on it     | No            | String           |
| `$value`     | A value to provide numerical data about the event                | No            | Int or Float     |
| `$context`   | Custom context for the event                                     | No            | Array            |
| `$tstamp`    | When the structured event occurred                               | No            | Positive integer |

Example:

```php
$tracker->trackStructEvent("shop", "add-to-basket", NULL, "pcs", 2);
```

#### `trackUnstructEvent`

Track a custom event which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

Function:

```php
public function trackUnstructEvent($event_json, $context = NULL, $tstamp = NULL)
```

Arguments:

| **Argument**  | **Description**                      | **Required?** | **Validation**   |
| ------------- | ------------------------------------ | ------------- | ---------------- |
| `$event_json` | The properties of the event          | Yes           | Array            |
| `$context`    | Custom context for the event         | No            | Array            |
| `$tstamp`     | When the unstructured event occurred | No            | Positive integer |

Example:

```php
$tracker->trackUnstructEvent(
    array(
        "schema" => "com.example_company/save-game/jsonschema/1-0-2",
        "data" => array(
            "save_id" => "4321",
            "level" => 23,
            "difficultyLevel" => "HARD",
            "dl_content" => true
        )
    ),
    NULL,
    132184654684
);
```

The `$event_json` must be an array with two fields: `schema` and `data`. `data` is a flat array containing the properties of the unstructured event. `schema` identifies the JSON schema against which `data` should be validated.

### Extra Tracker Functions

#### Tracker `flushEmitters`

The `flushEmitters` function can be called after you have successfully created a Tracker with the following function call:

```php
$tracker->flushEmitters();
```

This will tell the tracker to send any remaining events that are left in the buffer to the collector(s).

---

# Pixel tracker tag
> Track views of HTML-only content like emails with the Snowplow Pixel tracker using simple image tags without JavaScript.
> Source: https://docs.snowplow.io/docs/sources/pixel-tracker/

The Pixel tracker is an HTML-only tracking tag (no JavaScript) to track opens / views of HTML content that does not support JavaScript. Examples of use cases include HTML emails.

In a normal JavaScript tag, the name-value pairs of data that are sent through to the Snowplow Collector via the querystring are calculated on the fly. Examples of data points that are calculated like this include `user_id`, or `page_title`.

In an environment where JavaScript isn't permitted, these values need to be set in advance and hardcoded into the tracking tag. For example, if you want to record a different `page_title` for different pages, you'll need to generate a different tracking tag for each of those pages, with the right `page_title` set for each.

## Example Pixel tracking tags

The structure of a Pixel tracking tag is as follows:

```html
<img src="{{Collector-domain}}/i?{{name-value-pairs}}&tv=no-js-0.1.0" />
```

Some things to note about the tag:

- It's an `<img>` tag
- It sends a GET request using the Snowplow Collector `/i` endpoint
- You can send any name-value pairs that are supported by the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/) on the query string
  - The pipeline will drop name-value pairs that aren't supported by the Snowplow Tracker Protocol
- To help with analysis, we recommend you set the tracker version `tv` to `no-js-0.1.0` to identify that this event came from a Pixel tracker

### Page view tag

Here's an example tag to track a [page view](/docs/events/ootb-data/page-and-screen-view-events/) event:

```html
<!--Snowplow start plowing-->
<img src="http://Collector.acme.com/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=web&tv=no-js-0.1.0" />
<!--Snowplow stop plowing-->
```

The query string contains these key-value pairs:

- Event type `e` is `pv`, for page view event
- Page name `page` is `Root README`
- URL `url` is `http://github.com/snowplow/snowplow`
- Application id `aid` is `snowplow`
- Platform `p` is `web`
- Tracker version `tv` is `no-js-0.1.0`

### Custom self-describing event tag

Here's an example tag to track a [custom self-describing](/docs/events/custom-events/#custom-events) event. This tag tracks an event with the schema `iglu:com.acme/email_open/jsonschema/1-0-0` and the data payload:

```json
{
  "emailId": "abc123",
  "campaign": "spring-sale"
}
```

```html
<!--Snowplow start plowing-->
<img src="http://Collector.acme.com/i?e=ue&ue_pr=%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Funstruct_event%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22schema%22%3A%22iglu%3Acom.acme%2Femail_open%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22emailId%22%3A%22abc123%22%2C%22campaign%22%3A%22spring-sale%22%7D%7D%7D&aid=acme&p=web&tv=no-js-0.1.0" />
<!--Snowplow stop plowing-->
```

The query string contains these key-value pairs:

- Event type `e` is `ue`, for self-describing event (they used to be called "unstructured events")
- Event payload `ue_pr` is the [JSON-encoded self-describing event](/docs/fundamentals/schemas/#self-describing-json-data-in-the-event-payload)
- Application id `aid` is `acme`
- Platform `p` is `web`
- Tracker version `tv` is `no-js-0.1.0`

The full [`ue_pr` value](/docs/events/http-requests/) is structured like this:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0",
  "data": {
    "schema": "iglu:com.acme/email_open/jsonschema/1-0-0",
    "data": {
      "emailId": "abc123",
      "campaign": "spring-sale"
    }
  }
}
```

### Entities

You can also include [entities](/docs/fundamentals/entities/) in your Pixel tracking tags. Here's a tag that extends the page view example to add the `email_open` custom data as an entity:

```html
<!--Snowplow start plowing-->
<img src="http://Collector.acme.com/i?e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&co=%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Fcontexts%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%5B%7B%22schema%22%3A%22iglu%3Acom.acme%2Femail_open%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22emailId%22%3A%22abc123%22%2C%22campaign%22%3A%22spring-sale%22%7D%7D%5D%7D&aid=snowplow&p=web&tv=no-js-0.1.0" />
<!--Snowplow stop plowing-->
```

The entity data is tracked in the [`co` parameter](/docs/fundamentals/schemas/#self-describing-json-data-in-the-event-payload) as a [JSON-encoded array](/docs/events/http-requests/):

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.acme/email_open/jsonschema/1-0-0",
      "data": {
        "emailId": "abc123",
        "campaign": "spring-sale"
      }
    }
  ]
}
```

## A warning about using the Pixel tracker on domains you don't own

The Snowplow Collector sets a `network_userid` and drops this on a browser cookie.

Care must therefore be exercised when using the Pixel tracker on domains that you do not own. **It is your responsibility to abide by the terms and conditions of any domain owner for domains where you post content including uploading Pixel tracking tags.** Some domain owners forbid third parties from dropping cookies on their domains. It is your responsibility to ensure you do not violate the terms and conditions of any domain owners that you work with.

## Setting up the pixel tracker

### Identify the event you wish to track

Identify the event you wish to track. This may be opening a particular email that is sent out via your email marketing system, or viewing a product you are selling on a third-party marketplace.

### Construct the pixel tracker URL

Build your pixel tracker URL using the format shown above. Set your Collector domain and configure the query string parameters according to the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/).

### Insert the tracking code into the page or ad you wish to track

If this is an HTML email, you will need to insert it in the email. If it is a webpage hosted on a third-party site, you will need to add it to the source code.

## Click tracking

You can use the Pixel tracker for click tracking aka URI redirects:

- Set your Collector path to `{{Collector-domain}}/r/tp2?{{name-value-pairs}}` - the `/r/tp2` tells Snowplow that you are attempting a URI redirect
- Add a `&u={{uri}}` argument to your Collector URI, where `{{uri}}` is the URL-encoded URI that you want to redirect to
- On clicking this link, the Collector will register the link and then do a 302 redirect to the supplied `{{uri}}`
- As well as the `&u={{uri}}` parameter, you can populate the Collector URI with any other fields from the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/)

Redirect tracking is usually disabled by default, and is disabled by default for all Snowplow customers. To use this feature, you need to enable this in your Collector configuration. Snowplow customers can enable this from within the Pipeline Configuration screen of Snowplow Console.

You should also restrict values which are allowed within the `u` parameter to prevent phishing attacks using this redirect endpoint. One option is to use [AWS WAF](https://aws.amazon.com/waf/) or [Google Cloud Armor](https://cloud.google.com/armor) (depending on your cloud). They let you block traffic that matches rules you define, such as a regex that the value of the `u` parameter must match.

Example:

```html
Check out <a href="http://Collector.acme.com/r/tp2?u=https%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow">Snowplow</a>
```

Snowplow converts the `&u={{uri}}` argument into a `com.snowplowanalytics.snowplow/uri_redirect` self-describing JSON.

How Snowplow attaches the `uri_redirect` to the event depends on what other Tracker Protocol fields you attached to the event:

1. If you attached an `&e={{event type}}` to your event, then the `uri_redirect` will be added to the entities array of your event
2. If you did not attach an `&e={{event type}}` to your event, then this event will be treated as an self-describing event and the `uri_redirect` will be attached as the event itself

---

# Type checking in the Python tracker
> Enable runtime validation and type checking for Python tracker arguments using contracts to ensure proper value ranges and types.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/contracts/

Python is a dynamically typed language, but each of our methods expects its arguments to be of specific types and value ranges. To avoid wrong inputs, the tracker takes two approaches:

1. It specifies [type hints](https://docs.python.org/3/library/typing.html) on each function argument and return value. This allows IDEs to validate function calls in your code and also enables static type checking using tools such as [mypy](http://mypy-lang.org/). However, type violations do not result in runtime errors.
2. It validates expected value ranges during runtime and throws `ValueError` errors in case they are violated. Contracts are defined for arguments to check that they belong to expected value ranges such as:

```python
s = Subject()
tracker.set_platform("coffee") # throws ValueError because "coffee" is not one of supported platforms
tracker.set_screen_resolution(width=-1, height=-1) # throws ValueError because width and height arguments have to be greater than 0
```

You can turn off runtime contract validation like this:

```python
from snowplow_tracker import disable_contracts
disable_contracts()
```

---

# Example applications for the Python tracker
> Explore example Python tracker applications including Redis, Snowplow API, and Tracker API implementations for reference and quick start.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/demos/

Check out the following example applications that can be used as a reference for how to use the tracker, or as a starting point for your own implementation.

- [Redis](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/redis_example)
- [Snowplow API](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/snowplow_api_example)
- [Tracker API](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/tracker_api_example)

---

# Configure emitters in the Python tracker
> Configure emitters to send events to Snowplow collectors with batch size, retry logic, and async support using the Emitter and AsyncEmitter classes.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/emitters/

Tracker instances must be initialized with an emitter. This section will go into more depth about the Emitter class and its subclasses.

## The basic Emitter class

At its most basic, the Emitter class only needs a collector URI:

```python
from snowplow_tracker import Emitter

e = Emitter(endpoint="collector.example.com")
```

This is the signature of the constructor for the base Emitter class:

```python
def __init__(
        self,
        endpoint: str,
        protocol: Literal["http", "https"] = "https",
        port: Optional[int] = None,
        method: Literal["get", "post"] = "post",
        batch_size: Optional[int] = None,
        on_success: Optional[Callable[[PayloadDictList], None]] = None,
        on_failure: Optional[Callable[[int, PayloadDictList], None]] = None,
        byte_limit: Optional[int] = None,
        request_timeout: Optional[Union[float, Tuple[float, float]]] = None,
        max_retry_delay_seconds: int = 60,
        buffer_capacity: Optional[int] = None,
        custom_retry_codes: Dict[int, bool] = {},
        event_store: Optional[EventStore] = None,
        session: Optional[requests.Session] = None,
    )-> None:
```

| **Argument**              | **Description**                                                                        | **Required?** | **Type**                                | **Default** |
| ------------------------- | -------------------------------------------------------------------------------------- | ------------- | --------------------------------------- | ----------- |
| `endpoint`                | The collector URI                                                                      | Yes           | String                                  |             |
| `protocol`                | Request protocol: http or https                                                        | No            | String                                  | `https`     |
| `port`                    | The port to connect to                                                                 | No            | Positive integer                        | `None`      |
| `method`                  | The method to use: “get” or “post”                                                     | No            | String                                  | `post`      |
| `batch_size`              | Number of events to store before flushing                                              | No            | Positive integer                        | `10`        |
| `on_success`              | Callback executed when a flush is successful                                           | No            | Function taking 1 argument              | `None`      |
| `on_failure`              | Callback executed when a flush is unsuccessful                                         | No            | Function taking 2 arguments             | `None`      |
| `byte_limit`              | Number of bytes to store before flushing                                               | No            | Positive integer                        | `None`      |
| `request_timeout`         | Timeout for HTTP requests                                                              | No            | Positive integer or tuple of 2 integers | `None`      |
| `max_retry_delay_seconds` | The maximum time between attempts to send failed events to the collector               | No            | int                                     | 60          |
| `buffer_capacity`         | The maximum capacity of the event buffer                                               | No            | int                                     | `None`      |
| `custom_retry_codes`      | Custom retry rules for HTTP status codes received in emit responses from the Collector | No            | dict                                    | `None`      |
| `event_store`             | Stores the event buffer and buffer capacity                                            | No            | EventStore                              | `None`      |
| `session`                 | Persist parameters across requests by using a session object                           | No            | requests.Session                        | `None`      |

> **Note:** If no `event_store` is provided, an `InMemoryEventStore` will be initialized with a `buffer_capacity` of 10,000

See the [`API docs`](https://snowplow.github.io/snowplow-python-tracker/) for more information.

### `protocol`

`protocol` defaults to "https" but also supports "http".

### `batch_size`

When the emitter receives an event, it adds it to a buffer. When the queue is full, all events in the queue get sent to the collector. The `batch_size` argument allows you to customize the queue size. By default, it is 1 for GET requests and 10 for POST requests. (So in the case of GET requests, each event is fired as soon as the emitter receives it.) If the emitter is configured to send POST requests, then instead of sending one for every event in the buffer, it will send a single request containing all those events in JSON format.

### `byte_limit`

`byte_limit` is similar to `batch_size`, but instead of counting events - it takes into account only the amount of bytes to be sent over the network. _Warning_: this limit is approximate with error < 1%.

### `on_success`

`on_success` is an optional callback that will execute whenever the queue is flushed successfully, that is, whenever every request sent has status code 200. It will be passed one argument: an array of events that were successfully sent.

### `on_failure`

`on_failure` is similar, but executes when the flush is not wholly successful. It will be passed two arguments: the number of events that were successfully sent, and an array of unsent events.

An example:

```python
def success(arr):
    print(str(len(arr)) + " events sent successfully!")

def new_success(arr):
    for event_dict in arr:
        print(event_dict)

def failure(num, arr):
    print(str(num) + " events sent successfully!")
    print("These events were not sent successfully:")
    for event_dict in arr:
        print(event_dict)

e = Emitter(endpoint="collector.example.com", buffer_size=3, on_success=new_success, on_failure=failure)

t = Tracker(namespace="snowplow_tracker", emitter=e)
```

### `request_timeout`

Timeout for HTTP requests. Can be set either as single float value which applies to both "connect" AND "read" timeout, or as tuple with two float values which specify the "connect" and "read" timeouts separately.

### `max_retry_delay_seconds`

The maximum time between attempts to send failed events to the collector.

### `buffer_capacity`

The maximum capacity of the event buffer. When the buffer is full new events are lost.

### `custom_retry_codes`

Custom retry rules for HTTP status codes received in emit responses from the Collector. By default, retry will not occur for status codes 400, 401, 403, 410 or 422. This can be overridden here by parsing a dictionary of status codes and booleans.

### `event_store`

The event store is used to store an event queue with events scheduled to be sent. Events are added to the event store when they are tracked and removed when they are successfully emitted or when emitting fails without any scheduled retries. The default is an InMemoryEventStore object with a buffer\_capacity of 10,000 events.

### `session`

The session object can be parsed into the emitter to use the requests.Session API. This allows users to persist parameters across requests, as well as pool connections to increase efficiency under heavy usage. If no `session` is parsed, the requests API is used.

## What happens if an event fails to send?

After trying to send a batch of events the collector will return an http status code. A 2xx code is always considered successful. If a failure code is returned (anything other than 2xx, with certain exceptions, see below), the events (as PayloadDictList objects) are returned to the buffer. They will be retried in future sending attempts.

To prevent unnecessary requests being made while the collector is unavailable, an exponential backoff is added to all subsequent event sending attempts. This resets after a request is successful. The default maximum backoff time between attempts is 1 minute but this can be configured by setting `max_retry_delay_seconds`.

The status codes 400 Bad Request, 401 Unauthorised, 403 Forbidden, 410 Gone, or 422 Unprocessable Entity are the exceptions: they are not retried by default. Payloads in requests receiving these responses are not returned to the buffer for retry. They are just deleted.

Configure which codes to retry on or not using the `EmitterConfiguration` when creating your tracker. This method takes a dictionary of status codes and booleans (True for retry and False for not retry).

```python
# by default 401 isn't retried, but 500 is
custom_retry_codes = {500: False, 401: True}
emitter_config = EmitterConfiguration(custom_retry_codes=custom_retry_codes)

Snowplow.create_tracker(
    namespace="ns",
    endpoint="collector.example.com",
    emitter_config=emitter_config,
)
```

## Configuring how events are buffered

As events are collected, the are stored in a buffer until there are enough to send. By default, tracked events are stored in the `InMemoryEventStore`. This is an implemenation of the `EventStore` protocol and stores payloads in a `List` object and is cleared once the buffer capacity is reached.

The default buffer capacity is 10,000 events. This is the number of events that can be stored. When the buffer is full, new tracked payloads are dropped, so choosing the right capacity is important. You can set the buffer capacity through the `EmitterConfiguration` object, for example:

```python
emitter_config = EmitterConfiguration(buffer_capacity=25,000)

Snowplow.create_tracker(
    namespace="ns",
    endpoint="collector.example.com",
    emitter_config=emitter_config,
)
```

The emitter will store 25,000 events before starting to lose data.

## The AsyncEmitter class

```python
from snowplow_tracker import AsyncEmitter

e = AsyncEmitter(endpoint="collector.example.com", thread_count=10)
```

The `AsyncEmitter` class works just like the Emitter class, which is its parent class. It has one advantage, though: HTTP(S) requests are sent asynchronously, so the Tracker won't be blocked while the Emitter waits for a response. For this reason, the AsyncEmitter is recommended over the base `Emitter` class.

The AsyncEmitter uses a fixed-size thread pool to perform network I/O. By default, this pool contains only one thread, but you can configure the number of threads in the constructor using the `thread_count` argument, which is the only specific to AsyncEmitter argument.

Here is a complete example with all constructor parameters set:

```python
from snowplow_tracker import AsyncEmitter

e = AsyncEmitter(
    endpoint="collector.example.com",
    protocol = "https",
    port=9090,
    method='post',
    batch_size=10,
    on_success=None,
    on_failure=None,
    thread_count=1,
    byte_limit=None,
    max_retry_delay_seconds = 60,
    buffer_capacity = 5000,
    event_store = None,
    session=None
)
```

### Manual flushing

You can flush the emitter manually using the `flush` method of the `Tracker` instance which is sending events to the emitter. This is a blocking call which synchronously sends all events in the emitter's buffer.

```python
tracker.flush()
```

You can alternatively perform an asynchronous flush, which tells the tracker to send all buffered events but doesn't wait for the sending to complete:

```python
tracker.flush(False)
```

If you are using the AsyncEmitter, you shouldn't perform a synchronous flush inside an on\_success or on\_failure callback function as this can cause a deadlock.

### Multiple emitters

You can configure a tracker instance to send events to multiple emitters by passing the `Tracker` constructor function an array of emitters instead of a single emitter, or by using the `addEmitter` method:

```python
from snowplow_tracker import Subject, Tracker, AsyncEmitter

e1 = AsyncEmitter(endpoint="collector1.cloudfront.net", method="get")
e2 = AsyncEmitter(endpoint="collector2.cloudfront.net", method="post")

tracker = Tracker(namespace="snowplow_tracker", emitters=[e1, e2])

e3 = AsyncEmitter(endpoint="collector3.cloudfront.net", method="post")

tracker.addEmitter(e3)
```

### Custom emitters

You can create your own custom emitter class, either from scratch or by subclassing one of the existing classes. The only requirement for compatibility is that is must have an `input` method which accepts a Python dictionary of name-value pairs.

### Setting flush timer

You can flush your emitter based on some time interval:

```python
e1 = AsyncEmitter(endpoint="collector1.cloudfront.net", method="post")
e1.set_flush_timer(5)  # flush each 5 seconds
```

Automatic flush can also be cancelled:

```python
e1.cancel_flush()
```

---

# Python tracker SDK
> Track events in Python applications and games with the Snowplow Python tracker SDK using subjects, emitters, and trackers.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/

The Snowplow Python Tracker allows you to track Snowplow events from your Python apps and games.

The tracker should be straightforward to use if you are comfortable with Python development; any prior experience with Snowplow's [JavaScript Tracker](/docs/sources/web-trackers/), Google Analytics or Mixpanel (which have similar APIs to Snowplow) is helpful but not necessary.

There are three basic types of objects you will create when using the Snowplow Python Tracker: subjects, emitters, and trackers.

A subject represents a user whose events are tracked. A tracker constructs events and sends them to one or more emitters. Each emitter then sends the event to the endpoint you configure. This will usually be a Snowplow collector, but could also be a custom event store.

---

# Initialize the Python tracker
> Initialize the Python tracker using the Snowplow class or Tracker, Emitter, and Subject classes with configuration for namespace, endpoint, and encoding.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/intialization/

Assuming you have completed the Python Tracker Setup for your Python project, you are now ready to initialize the Python Tracker. There are two ways to this:

1. Using the `Snowplow` class. This is preferable for most use cases.
2. Using the `Tracker`, `Emitter` and `Subject` classes. This option is useful in case you want to replace one of the internal tracker components (for instance, you want to provide a custom `Emitter` class).

## Option 1: Initialization using the Snowplow class

The Snowplow class contains `static` methods to help manage `Tracker` objects.

Import the Snowplow class along with the required configuration objects:

```python
from snowplow_tracker import Snowplow, EmitterConfiguration, Subject, TrackerConfiguration
```

The simplest tracker configuration can be instantiated with the `create_tracker()` method as follows:

```python
Snowplow.create_tracker(namespace='ns', endpoint='collector.example.com')
```

This creates a `Tracker` and `Emitter` with default settings, with events logged to `collector.example.com`.

You can access a tracker in the following way:

```python
Snowplow.get_tracker('ns')
```

The Snowplow class can be used to initialize trackers using the following properties:

| **Argument Name** | **Description**                      | **Required?** | **Default**              |
| ----------------- | ------------------------------------ | ------------- | ------------------------ |
| `namespace`       | The name of the tracker              | Yes           |                          |
| `endpoint`        | The collector URL events are sent to | Yes           |                          |
| `method`          | The method to use: “get” or “post”   | No            | `post`                   |
| `emitter_config`  | The emitter configuration object     | No            | `EmitterConfiguration()` |
| `app_id`          | The application ID                   | No            | `None`                   |
| `subject`         | The user being tracked               | No            | `subject.Subject()`      |
| `tracker_config`  | The tracker configuration object     | No            | `TrackerConfiguration()` |

Optionally, you may choose to use the `TrackerConfiguration` and `EmitterConfiguration` classes to configure the emitter and tracker respectively.

### Tracker configuration using `TrackerConfiguration`

The `TrackerConfiguration` class contains settings to encode the payload and provide a custom json serializer. For example:

```python
tracker_config = TrackerConfiguration(
    encode_base64=False,
    json_encoder=my_custom_encoder
)

Snowplow.create_tracker(
    namespace='ns',
    endpoint='collector.example.com',
    tracker_config = tracker_config
)
```

| **Argument Name** | **Description**                                                     | **Default** |
| ----------------- | ------------------------------------------------------------------- | ----------- |
| `encode_base64`   | Whether JSONs in the payload should be base-64 encoded.             | `True`      |
| `json_encoder`    | Custom JSON serializer that gets called on non-serializable object. | `None`      |

### Emitter configuration using `EmitterConfiguration`

The `EmitterConfiguration` class contains additional settings for the `Emitter` initialization. For example:

```python
emitter_config = EmitterConfiguration(
    batch_size = 50,
    on_success = my_success_function,
    on_failure= my_failure_function,
    byte_limit = 25000,
    request_timeout = (10, 20),
    custom_retry_codes = {500: False, 401: True}, # Don't retry 500, retry 401
    event_store=my_custom_event_store,
    session=requests.Session
)

Snowplow.create_tracker(
    namespace='ns',
    endpoint='collector.example.com',
    emitter_config = emitter_config
)
```

The full list of arguments is below:

| **Argument Name**    | **Description**                                                                                                                                                                                                                                                               | **Default**          |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `batch_size`         | The maximum number of queued events before the buffer is flushed.                                                                                                                                                                                                             | `10`                 |
| `on_success`         | Callback executed after every HTTP request in a flush has status code 200.                                                                                                                                                                                                    | `None`               |
| `on_failure`         | Callback executed if at least one HTTP request in a flush has status code other than 200.                                                                                                                                                                                     | `None`               |
| `byte_limit`         | The size event list after reaching which queued events will be flushed.                                                                                                                                                                                                       | `None`               |
| `request_timeout`    | Timeout for the HTTP requests. Can be set either as single float value which applies to both "connect" AND "read" timeout, or as tuple with two float values which specify the "connect" and "read" timeouts separately.                                                      | `None`               |
| `custom_retry_codes` | Set custom retry rules for HTTP status codes received in emit responses from the Collector. By default, retry will not occur for status codes 400, 401, 403, 410 or 422. This can be overridden here. Note that 2xx codes will never retry as they are considered successful. | `None`               |
| `event_store`        | Stores the event buffer and buffer capacity.                                                                                                                                                                                                                                  | `InMemoryEventStore` |
| `session`            | Persist parameters across requests by using a session object                                                                                                                                                                                                                  | `None`               |

## Option 2: Managing "Tracker", "Emitter", and "Subject" directly

Require the Python Tracker's module into your Python code like so:

```python
from snowplow_tracker import Tracker, Emitter, Subject
```

That's it - you are now ready to initialize a tracker instance.

### Creating a tracker

The simplest tracker initialization only requires you to provide the Emitter's endpoint to which the tracker will log events:

```python
e = Emitter(endpoint="collector.example.com")
t = Tracker(namespace="snowplow_tracker", emitters=e)
```

The tracker parameters are:

| **Argument Name** | **Description**                                                            | **Required?** | **Default**         |
| ----------------- | -------------------------------------------------------------------------- | ------------- | ------------------- |
| `namespace`       | The name of the tracker instance                                           | Yes           |                     |
| `emitters`        | The emitter(s) to which events are sent                                    | Yes           |                     |
| `subject`         | The user being tracked                                                     | No            | `subject.Subject()` |
| `app_id`          | The application ID                                                         | No            | `None`              |
| `encode_base64`   | Whether to enable [base 64 encoding](https://en.wikipedia.org/wiki/Base64) | No            | `True`              |
| `json_encoder`    | Custom JSON serializer                                                     | No            | `None`              |

Here is a more complete example in which every tracker parameter is set:

```python
e = Emitter(endpoint="collector.example.com")
s = Subject().set_platform("srv")

tracker = Tracker(
    namespace="cf",
    emitter=e,
    subject=s,
    app_id="aid",
    encode_base64=False,
    json_encoder=my_custom_encoder
)
```

#### `namespace`

The `namespace` argument is attached to every event fired by the new tracker. This allows you to later identify which tracker fired which event if you have multiple trackers running.

#### `emitters`

This can be a single emitter or an array containing at least one emitter. The tracker will send events to these emitters, which will in turn send them to a collector.

```python
e1 = Emitter(endpoint="mycollector.com")
e2 = Emitter(endpoint="myothercollector.com", port=8080)
tracker = Tracker(namespace="snowplow_tracker", emitters=[e1, e2])
```

#### `subject`

The user which the Tracker will track. This should be an instance of the Subject class. You don't need to set this during Tracker construction; in this case the tracker will set a default subject, which you can also change using the `Tracker.set_subject` method afterwards.

#### `app_id`

The `app_id` argument lets you set the application ID to any string.

#### `encode_base64`

By default, unstructured events and custom contexts are encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the Boolean `encode_base64` argument.

#### `json_encoder`

This parameter allows you to customize the JSON encoder used to serialize objects added to the payload. For example:

```python
from json.encoder import JSONEncoder
def complex_encoder(c):
    if isinstance(c,complex):
        return [c.real, c.imag]
    return JSONEncoder.default(c)

t = Tracker(
    namespace="snowplow_tracker",
    emitter=e,
    json_encoder=complex_encoder
)
```

---

# Emitter logs in the Python tracker
> Configure Python logging levels for the tracker emitters module to control debug, info, and error message output.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/logging/

The emitters.py module has Python logging turned to give you information about requests being sent. The logger prints messages about what emitters are doing. By default, only messages with priority "INFO" or higher will be logged.

To change this:

```python
from snowplow_tracker import logger

# Log all messages, even DEBUG messages
logger.setLevel(10)

# Log only messages with priority WARN or higher
logger.setLevel(30)

# Turn off all logging
logger.setLevel(60)
```

---

# Install the Python tracker
> Install the Python tracker from PyPI using pip for event tracking in Python applications with minimal dependencies.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/setup/

As a programming language that lets you work more quickly and integrate your systems more effectively, Python is available in a huge number of different computing environments and platforms, from Civilization IV through [Django framework](https://www.djangoproject.com/) to Ubuntu One.

To make the Snowplow Python Tracker work out-of-the-box with as many different Python programs as possible, we have tried to:

1. Minimize external dependencies and third party libraries
2. Provide setup instructions

## Dependencies

To make the Snowplow Python Tracker work with as many different Python programs as possible, we have tried to keep external dependencies to a minimum. The main external dependencies currently are:

- [Requests](https://pypi.python.org/pypi/requests) – HTTP library
- [Python typing extensions](https://pypi.org/project/typing-extensions/) – Backported type hints for Python

These dependencies can be installed from the package manager of the host system or through PyPi.

## PyPI

The Snowplow Python Tracker is published to [PyPI](https://pypi.python.org/), the the official third-party software repository for the Python programming language.

This makes it easy to either install the tracker locally, or to add it as a dependency into your own Python app.

## pip

To install the Snowplow Python Tracker locally, assuming you already have `pip` installed:

```bash
$ pip install snowplow-tracker --upgrade
```

To add the Snowplow Tracker as a dependency to your own Python app, edit your `requirements.txt` and add:

```txt
snowplow-tracker==1.0.3
```

## Python version support

Please refer to the table below to identify the recommended tracker version for your Python version.

| Python Version | snowplow-tracker Version |
| -------------- | ------------------------ |
| >=3.5          | 1.0.3                    |
| 2.7            | 0.9.1                    |

---

# Track subject data with the Python tracker
> Configure subject data for tracked events including user ID, platform, screen resolution, timezone, and IP address for tracker-level or event-level subjects.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

The Subject class has a number of `set_xxx()` methods to attach extra data relating to the user of the tracked events.

Here are some examples:

```python
from snowplow_tracker import Subject
subject = Subject()

subject.set_user_id("user_id")
subject.set_lang("en-gb")
subject.set_screen_resolution(1920, 1080)
```

There are two ways to provide the subject information:

1. For all tracked events. This is useful in client-side applications where all tracked events relate to the same user and share the same properties.
2. For each event individually. This is useful in server-side applications where each event might relate to a different user.

> **Note:** When the same parameter is set for a tracker subject and an event subject, the event subject will take priority.

The full set of subject methods are listed below:

| **Subject Method**                                      | **Description**                                                |
| ------------------------------------------------------- | -------------------------------------------------------------- |
| [`set_platform`](#set_platform)                         | Override the platform for the event (e.g. `srv`, `app`, `web`) |
| [`set_user_id`](#set_user_id)                           | Identify the Subject with a business ID                        |
| [`set_screen_resolution`](#set_screen_resolution)       | Set the dimensions of the Subject's device                     |
| [`set_viewport`](#set_viewport)                         | Set the dimensions of the Subject's viewport                   |
| [`set_color_depth`](#set_color_depth)                   | Set the color depth of the Subject device's display            |
| [`set_timezone`](#set_timezone)                         | Set the local timezone for the Subject                         |
| [`set_lang`](#set_lang)                                 | Set the preferred language or locale of the Subject            |
| [`set_ip_address`](#set_ip_address)                     | Set the IP address of the Subject                              |
| [`set_user_agent`](#set_useragent)                      | Set the User Agent string of the Subject                       |
| [`set_domain_user_id`](#set_domain_user_id)             | Set the Domain User ID of the Subject                          |
| [`set_network_user_id`](#set_network_user_id)           | Set the Network User ID of the Subject                         |
| [`set_domain_session_id`](#set_domain_session_id)       | Set the Domain Session ID of the Subject                       |
| [`set_domain_session_index`](#set_domain_session_index) | Set the Domain Session Index of the Subject                    |

### Setting a Tracker Subject

To configure a tracker subject, pass it to the tracker during initialization:

```python
subject = Subject().set_platform("mob").set_user_id("user-12345").set_lang("en")

t = Tracker(
        namespace="snowplow_tracker",
        emitters=emitter,
        subject=subject
)
```

If you initialize a `Tracker` instance without a subject, a default `Subject` instance will be attached to the tracker. You can access that subject like this:

```python
t = Tracker(
        namespace="snowplow_tracker",
        emitters=my_emitter
)
t.subject.set_platform("mob").set_user_id("user-12345").set_lang("en")
```

### Setting an Event Subject

To configure an event level subject, pass it to the event during initialization.

```python
event_subject = Subject().set_screen_resolution(1920, 1080)

id = tracker.get_uuid()
screen_view = ScreenView(
  id_=id,
  name="name",
  event_subject=event_subject
)

tracker.track(screen_view)
```

### `set_platform`

The default platform is `pc`. You can change the platform the subject is using by calling for example:

```python
s.set_platform('mob')
```

For a full list of supported platforms, please see the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields).

### `set_user_id`

You can set the user ID to any string:

```python
s.set_user_id("user_id")
```

### `set_screen_resolution`

Set the screen resolution as below. Both numbers should be positive integers in the order width followed by height.

```python
s.set_screen_resolution(1366, 768)
```

### `set_viewport`

Set the viewport dimensions as below. Both numbers should be positive integers in the order width followed by height.

```python
s.set_viewport(300, 200)
```

### `set_color_depth`

Set the bit depth of the device's color palette for displaying images as below. The number should be a positive integer, in bits per pixel. For example:

```python
s.set_color_depth(32)
```

### `set_timezone`

This method lets you pass a user's timezone into Snowplow. The timezone should be a [IANA TZ identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) string.

```python
s.set_timezone("Europe/London")
```

### `set_lang`

This method lets you pass a user's language into Snowplow. The language should be a string.

```python
s.set_lang('en')
```

### `set_ip_address`

If you have access to the user's IP address, you can set it like this:

```python
s.set_ip_address('34.633.11.139')
```

The set value will be used in the [IP Lookup](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/) and [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) enrichments, if enabled.

> **Tip:** On web applications, the remote address of an incoming request may reflect load balancers or reverse proxies instead of the actual Subject's IP. You may need to consider the [Forwarded](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) or [X-Forwarded-For](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) HTTP headers instead, though these headers can be forged by the requester.

### `set_useragent`

If you have access to the user's useragent (sometimes called "browser string"), you can set it like this:

```python
s.set_useragent('Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0')
```

The set value will be used in the [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/), [UA Parser](/docs/pipeline/enrichments/available-enrichments/ua-parser-enrichment/), [IAB](/docs/pipeline/enrichments/available-enrichments/iab-enrichment/) and similar enrichments.

### `set_domain_user_id`

The `domain_userid` field of the Snowplow event model corresponds to the ID stored in the first party cookie set by the Snowplow JavaScript Tracker. If you want to match up server-side events with client-side events, you can set the domain user ID for server-side events like this:

```python
s.set_domain_user_id('c7aadf5c60a5dff9')
```

### `set_network_user_id`

The `network_user_id` field of the Snowplow event model corresponds to the ID stored in the third party cookie set by the Snowplow Collector. You can set the network user ID for server-side events like this:

```python
s.set_network_user_id('ecdff4d0-9175-40ac-a8bb-325c49733607')
```

### `set_domain_session_id`

This method lets you pass a Domain Session ID in to Snowplow:

```python
s.set_domain_session_id('ecdff4d0-9175-40ac-a8bb-325c49733607')
```

> **Tip:** The Domain User ID, Domain Session ID, and Domain Session Index values can all be extracted from the same cookie.

### `set_domain_session_index`

This method lets you pass a Domain Session index in to Snowplow:

```python
s.set_domain_session_index(4)
```

## Tracking multiple subjects

You may want to track more than one subject concurrently. To avoid data about one subject being added to events pertaining to another subject, create two subject instances and switch between them using `Tracker.set_subject`:

```python
from snowplow_tracker import Subject, Emitter, Tracker

# Create a simple Emitter which will log events to https://d3rkrsqld9gmqf.cloudfront.net/com.snowplowanalytics.snowplow/tp2
e = Emitter(endpoint="d3rkrsqld9gmqf.cloudfront.net")

# Create a Tracker instance
t = Tracker(emitters=e, namespace="cf", app_id="CF63A")

# Create a Subject corresponding to a pc user
s1 = Subject()

# Set some data for that user
s1.set_platform("pc")
s1.set_user_id("0a78f2867de")

# Set s1 as the tracker subject
# All events fired will have the information we set about s1 attached
t.set_subject(s1)

# Track user s1 viewing a page
page_view = PageView(
        page_url="https://www.snowplow.io",
        page_title="Homepage",
)
t.track(page_view)

# Create another Subject instance corresponding to a mobile user
s2 = Subject()

# All methods of the Subject class return the Subject instance so methods can be chained:
s2.set_platform("mob").set_user_id("0b08f8be3f1")

# Change the tracker subject from s1 to s2
# All events fired will have instead have information we set about s2 attached
t.set_subject(s2)

# Track user s2 viewing a page
page_view = PageView(
        page_url="https://www.snowplow.io",
        page_title="Homepage",
)
t.track(page_view)

# Switch back to s1 and track a structured event.
t.set_subject(s1)
struct_event = StructuredEvent(
        category="shop",
        action="add-to-basket",
        label="web-shop",
        property_="pcs",
        value=2,
    )
t.track(struct_event)
```

---

# Track specific events with the Python tracker
> Track page views, screen views, structured events, and custom self-describing events with the Python tracker using Event classes and custom context.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/tracking-specific-events/

The Python tracker makes it simple to track a selection of out-of-the-box events as well as the ability to define your own custom events.

To track an event, simply pass the `Event` object to the `tracker.track()` method. For example, tracking a page view:

```python
page_view = PageView(
  page_url="https://www.snowplow.io",
  page_title="Homepage",
)
tracker.track(page_view)
```

Every tracked event payload has a unique event\_id UUID string (`eid`). Other properties include the name\_tracker (`namespace`) and `app_id` set when the Tracker was initialized. From version 1 onwards, `tracker.track()` returns the payload's `eid`.

Snowplow events have a defined structure and protocol that is identical regardless of the tracker used. Further information on this structure can be found [here](/docs/events/).

The Python tracker [Github repository](https://github.com/snowplow/snowplow-python-tracker) includes 3 example apps demonstrating different ways to send events to your collector.

## Event Tracking

The Python tracker provides classes for tracking different types of events. They are listed below:

| **Event**         | **Description**                               |
| ----------------- | --------------------------------------------- |
| `SelfDescribing`  | Track custom events with custom schemas       |
| `PageView`        | Track views of web pages                      |
| `PagePing`        | Track engagement on web pages over time       |
| `ScreenView`      | Track views of a screen (non-web e.g. in-app) |
| `StructuredEvent` | Track custom events without schemas           |

## Creating a custom event (`SelfDescribing`)

To track data using a `SelfDescribing` event, the data must be structured as a `SelfDescribingJson` object. These require two fields, a URI for a self-describing JSON schema and the data in the form of a `PayloadDict`. The data must be valid against the schema.

A simple initialization for a link click event looks like this:

```python
link_click = SelfDescribing(
  SelfDescribingJson(
    "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1",
    {"targetUrl": "https://www.snowplow.io"},
  ),
)
tracker.track(link_click)
```

## Creating a `PageView` Event

Track views of a web page with the `PageView` event.

| **Property**     | **Description**                      | **Type**                   | **Required?** |
| ---------------- | ------------------------------------ | -------------------------- | ------------- |
| `page_url`       | URL of the viewed page               | `string`                   | Yes           |
| `page_title`     | Title of the viewed page             | `string`                   | No            |
| `referrer`       | The address which linked to the page | `string`                   | No            |
| `event_subject`  | The subject for the event            | `Subject`                  | No            |
| `context`        | Custom context for the event         | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the page view occurred          | `int or float`             | No            |

Example:

```python
page_view = PageView(
  page_url="https://www.snowplow.io",
  page_title="Homepage",
  referrer="https://docs.snowplow.io/docs",
)
tracker.track(page_view)
```

## Creating a `PagePing` Event

Track engagement with a web page over time, via a `PagePing` event. Each ping represents a single heartbeat.

| **Property**     | **Description**                                    | **Type**                   | **Required?** |
| ---------------- | -------------------------------------------------- | -------------------------- | ------------- |
| `page_url`       | URL of the viewed page                             | `string`                   | Yes           |
| `page_title`     | Title of the viewed page                           | `string`                   | No            |
| `referrer`       | The address which linked to the page               | `string`                   | No            |
| `min_x`          | Minimum page x offset seen in the last ping period | `int`                      | No            |
| `max_x`          | Maximum page x offset seen in the last ping period | `int`                      | No            |
| `min_y`          | Minimum page y offset seen in the last ping period | `int`                      | No            |
| `max_y`          | Maximum page y offset seen in the last ping period | `int`                      | No            |
| `event_subject`  | The subject for the event                          | `Subject`                  | No            |
| `context`        | Custom context for the event                       | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the page ping occurred                        | `int or float`             | No            |

Example:

```python
page_ping = PagePing(
  page_url="https://www.snowplow.io",
  page_title="Homepage",
  referrer="https://docs.snowplow.io/docs",
)
tracker.track(page_ping)
```

## Creating a `ScreenView` Event

Use the `ScreenView` to track a user viewing a screen (or equivalent) within your app.

| **Property**      | **Description**                                             | **Type**                   | **Required?** |
| ----------------- | ----------------------------------------------------------- | -------------------------- | ------------- |
| `id_`             | Unique identifier for this screen (UUID)                    | `string`                   | No            |
| `name`            | Human-readable name for this screen                         | `Non-empty string`         | No            |
| `type`            | The type of screen that was viewed e.g feed / carousel.     | `string`                   | No            |
| `previous_name`   | The name of the previous screenview.                        | `string`                   | No            |
| `previous_id`     | The id of the previous screenview.                          | `string`                   | No            |
| `previous_type`   | The type of the previous screenview.                        | `string`                   | No            |
| `transition_type` | The type of transition that led to the screen being viewed. | `string`                   | No            |
| `event_subject`   | The subject for the event                                   | `Subject`                  | No            |
| `context`         | Custom context for the event                                | `List(SelfDescribingJson)` | No            |
| `true_timestamp`  | When the screen was viewed                                  | `int or float`             | No            |

Example:

```python
id = tracker.get_uuid()
screen_view = ScreenView(
  id_=id,
  name="name",
  type="feed",
  previous_name="Home Page",
  previous_id="1368725287000",
  previous_type="feed"
)
tracker.track(screen_view)
```

## Creating a `StructuredEvent`

Use `StructuredEvent` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Property**     | **Description**                                                  | **Type**                   | **Required?** |
| ---------------- | ---------------------------------------------------------------- | -------------------------- | ------------- |
| `category`       | The grouping of structured events which this `action` belongs to | `Non-empty string`         | Yes           |
| `action`         | Defines the type of user interaction which this event involves   | `Non-empty string`         | Yes           |
| `label`          | A string to provide additional dimensions to the event data      | `string`                   | No            |
| `property`       | A string describing the object or the action performed on it     | `string`                   | No            |
| `value`          | A value to provide numerical data about the event                | `int or float`             | No            |
| `event_subject`  | The subject for the event                                        | `Subject`                  | No            |
| `context`        | Custom context for the event                                     | `List(SelfDescribingJson)` | No            |
| `true_timestamp` | When the structured event occurred                               | `int or float`             | No            |

Example:

```python
struct_event = StructuredEvent(
  category="shop",
  action="add-to-basket",
  label="web-shop",
  property_="pcs",
  value=2,
)
tracker.track(struct_event)
```

## Common tracking parameters

All events are tracked with specific event classes on the tracker instance, of the form `track(Event)`. The parameters that are common for all track methods are:

- `event_subject`
- `context`
- `true_timestamp`

### Event subject

It is possible to set the Subject per-event, in order to augment the event with extra information without having to change the Subject at the Tracker level. This provides a thread safe way to track multiple subjects.

Event level subjects are combined with any tracker subjects that have been initialized, with the event subject taking priority over tracker subject parameters.

This is supported as an optional keyword argument by all event classes. For example:

```python

event_subject = Subject().set_user_id("1234")
page_view = PageView(
  page_url="https://www.snowplow.io",
  page_title="Homepage",
  event_subject=event_subject
)

tracker.track(page_view)
```

More detail on the `Subject` class can be found [here](/docs/sources/python-tracker/subject/).

### Custom context

Custom context can be used to augment any standard Snowplow event type, including self describing events, with additional data. You can read more about custom contexts and the possible use cases [here](/docs/fundamentals/entities/#custom-entities).

Custom context can be added as an extra argument to any of Snowplow’s Event classes. The `context` argument should consist of a list of one or more instances of the `SelfDescribingJson` class.

For example, if a server-side Python application can determine visitor's geoposition, this can be attached to the event using the `geolocation_context` that is predefined on [Iglu Central](https://github.com/snowplow/iglu-central):

```python
from snowplow_tracker import SelfDescribingJson

geo_context = SelfDescribingJson(
  "iglu:com.snowplowanalytics.snowplow/geolocation_context/jsonschema/1-0-0",
  {
    "latitude": -23.2,
    "longitude": 43.0
  }
)
```

As another example, if a visitor arrives on a page advertising a movie, the context object might look like this (`movie_poster` is custom context, not predefined):

```python
poster_context = SelfDescribingJson(
  "iglu:com.acme_company/movie_poster/jsonschema/2-1-1",
  {
    "movie_name": "Solaris",
    "poster_country": "JP",
    "poster_year": "1978-01-01"
  }
)
```

This is how to fire a page view event with both above contexts:

```python
page_view = PageView(
  page_url="https://www.snowplow.io",
  page_title="Homepage",
  context=[poster_context, geo_context]
)

tracker.track(page_view)
```

**Important:** Even if only one custom context is being attached to an event, it still needs to be parsed as a list.

### Timestamp argument

Each event class supports an optional timestamp as an argument. The timestamp should be in milliseconds since the Unix epoch, the same format as generated by `time.time() * 1000`.

Generally, according to the Snowplow Tracker Protocol, every event tracked will be recorded with two timestamps:

- the `dvce_created_tstamp`, which is the timestamp when the event was created

- the `dvce_sent_tstamp`, which is the timestamp when the event was sent

These are going to be used downstream, to calculate the `derived_tstamp` for the event, which takes also into account the collector timestamp, in order to best approximate the exact time the event occurred.

The optional timestamp argument is for the cases where you might want to set the event timestamp yourself. If this argument is not provided or set to None, then the Python Tracker will use the current time to be the `dvce_created_tstamp` for the event.

Here is an example tracking a structured event and supplying the optional timestamp argument. We can explicitly supply `None` for the intervening arguments which are empty:

```python
struct_event = StructuredEvent(
  category="shop",
  action="add-to-basket",
  label="web-shop",
  property_="pcs",
  value=2,
  true_timestamp=1368725287000
)
tracker.track(struct_event)
```

---

# Upgrade the Python tracker
> Upgrade to newer versions of the Python tracker with migration guides for v1.0.0 and v0.13.0+ breaking changes and new features.
> Source: https://docs.snowplow.io/docs/sources/python-tracker/upgrading/

This page gives instructions to upgrading to newer versions of the Python Tracker

## Upgrading to v1.0.0

There are a number of breaking changes and new features in the V1 release, in order to make use of the latest tracker make sure to:

- Tracker `namespace` is now a required property in the tracker and should be parsed on initialization of a new tracker.
- The `track_xxx()` functions have been deprecated in favor of the `Event` class, further information on tracking events using the new API can be found on the [Event Tracking](/docs/sources/python-tracker/tracking-specific-events/) page.
- The `track()` function in the tracker now returns an event id, instead of a tracker instance. Chaining methods using the `track()` function will no longer work, and methods should be applied to the tracker itself.
- The `track()` function no longer accepts a `Payload` object as an argument and instead accepts an `Event` object.
- The parameters in the `Tracker` constructor have changed order, to avoid errors we suggest explicitly defining the variables.
- Event subjects and tracker subjects are now combined when an event is tracked on a given tracker. The old API will continue to work, however the contents of the payload in the event will contain additional tracker subject data.
- The `Redis` and `Celery` emitters have now been removed from the Python tracker. An example of a redis implementation can be found in the [examples](https://github.com/snowplow/snowplow-python-tracker/tree/master/examples/redis_example) section of the tracker repo.
-

## Upgrading to v0.13.0+

There are a number of breaking changes in this release, in order to make use of the new features please make sure to:

- Use the `batch_size` argument instead of `buffer_size` in the Emitter initialization.
- Users should no longer use the `on_failure` method to determine retry behaviors. Retry failed events is now a built in feature which can be configured in the `Emitter` directly or via `EmitterConfiguration` object.

---

# Remove the React Native tracker at runtime
> Remove individual trackers by namespace or remove all trackers at runtime using the React Native tracker API.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/advanced-usage/

The React Native tracker provides two functions that allow you to remove a tracker (or all of them) at runtime.

### removeTracker

As each tracker is identified by its namespace, in order to remove a tracker at runtime, you need to pass its namespace to the `removeTracker` function.

For example, assuming an existing tracker with namespace `sp1` :

```javascript
import { newTracker, removeTracker } from '@snowplow/react-native-tracker';

// ...

removeTracker('sp1');
```

### removeAllTrackers

The function removeAllTrackers, which accepts no arguments, will remove all trackers created in your app.

```javascript
import { removeAllTrackers } from '@snowplow/react-native-tracker';

removeAllTrackers();
```

---

# Anonymous tracking for the React Native tracker
> Enable anonymous tracking to disable user and session identifiers with client-side and server-side anonymization options.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/anonymous-tracking/

Anonymous tracking is a feature that enables anonymization of various user and session identifiers, to support user privacy when consent for tracking the identifiers isn't given.

On React Native, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, `domain_userid`, `network_userid`, and `user_ipaddress`, set by you in `Subject`
  - `userId`, a device ID, set automatically in the [session](/docs/sources/web-trackers/tracking-events/session/) entity
  - For mobile only, IDFA identifiers (`appleIdfa` and `appleIdfv` for iOS, `androidIdfa` for Android) in the mobile entity [if configured](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/#platform-context)

- Client-side session identifiers:
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

> **Note:** The Collector captures the IP address from the request HTTP headers, and updates the `user_ipaddress` event property. However, if you set the `user_ipaddress` property in the `Subject`, that value has priority.
>
> Similarly, if you set the `network_userid` property in the `Subject`, that value is used instead of the Collector cookie value.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

There are several levels to the anonymization depending on which of the categories are affected.

## Available anonymization options

Because of a change in architecture, the anonymous tracking options for the React Native tracker changed between version 2.x and version 4.x.

| Feature                                         | Version 4.x | Version 1.3 - 2.x |
| ----------------------------------------------- | ----------- | ----------------- |
| Full client-side anonymization                  | ❌           | ✅                 |
| Client-side anonymization with session tracking | ❌           | ✅                 |
| Toggle anonymous tracking at runtime            | ❌           | ✅                 |
| Server-side anonymization                       | ✅           | ✅                 |

## 1. Full client-side anonymization

In this case, you anonymize both client-side user identifiers and client-side session identifiers. This means disabling the session entity altogether.

**Version 4.x:**

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    sessionContext: false, // Session entity won't be added to events
    platformContext: false, // Mobile entity won't be added to events

});
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ✅ if set            |
| `domain_userid`     | Atomic                   | ✅ if set            |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌ no mobile entity  |
| `appleIdfv`         | Mobile (platform) entity | ❌ no mobile entity  |
| `androidIdfa`       | Mobile (platform) entity | ❌ no mobile entity  |
| `network_userid`    | Atomic                   | ✅                   |
| `user_ipaddress`    | Atomic                   | ✅                   |

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(
    'appTracker',
    { endpoint: COLLECTOR_URL },
    {
        trackerConfig: {
            sessionContext: false,   // Session entity won't be added to events
            userAnonymisation: true, // Remove user identifiers from Subject and platform entity
        },
    }
);
```

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ no session entity                           |
| `sessionId`         | Session entity           | ❌ no session entity                           |
| `previousSessionId` | Session entity           | ❌ no session entity                           |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

***

## 2. Client-side anonymization with session tracking

This setting disables client-side user identifiers but tracks session information.

**Version 4.x:**

Newer versions of the React Native tracker don't support client-side anonymization with session tracking. Unlike in version 2.x, you can't track the session entity with an anonymized `userId`.

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(
    'appTracker',
    { endpoint: COLLECTOR_URL },
    {
        trackerConfig: {
            sessionContext: true,    // Session entity is tracked
            userAnonymisation: true, // User identifiers anonymized
        },
    }
);
```

With `userAnonymisation: true` and `sessionContext: true`, events include the session entity but the `userId` property is set to a null UUID (`00000000-0000-0000-0000-000000000000`). The IDFA identifiers are also removed from the platform entity.

| Identifier          | Location in event        | Included in event?                            |
| ------------------- | ------------------------ | --------------------------------------------- |
| `user_id`           | Atomic                   | ❌                                             |
| `domain_userid`     | Atomic                   | ❌                                             |
| `userId`            | Session entity           | ❌ null UUID                                   |
| `sessionId`         | Session entity           | ✅                                             |
| `previousSessionId` | Session entity           | ❌                                             |
| `appleIdfa`         | Mobile (platform) entity | ❌                                             |
| `appleIdfv`         | Mobile (platform) entity | ❌                                             |
| `androidIdfa`       | Mobile (platform) entity | ❌                                             |
| `network_userid`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ✅/❌ removed if you provided this in `Subject` |

> **Note:** When anonymous tracking is enabled or disabled using `tracker.setUserAnonymisation(true | false)`, the tracker starts a new session. This results in a new `sessionId`.

***

## 3. Server-side anonymization

Server-side anonymization affects user identifiers set by the Collector: the `network_userid` property (set in the server-side cookie) and the user IP address. Use the `serverAnonymization` flag to prevent the Collector from setting these values.

> **Note:** The new API uses the US English spelling "anonymization", rather than the British English "anonymisation" seen in the older API.

**Version 4.x:**

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    serverAnonymization: true,
});
```

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(
    'appTracker',
    { endpoint: COLLECTOR_URL },
    {
        emitterConfig: {
            serverAnonymisation: true,
        },
    }
);
```

***

Setting this flag adds an `SP-Anonymous` HTTP header to requests sent to the Collector. The Snowplow pipeline then anonymizes the `network_userid` and `user_ipaddress` identifiers.

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅ if set                                            |
| `domain_userid`     | Atomic                   | ✅ if set                                            |
| `userId`            | Session entity           | ✅                                                   |
| `sessionId`         | Session entity           | ✅                                                   |
| `previousSessionId` | Session entity           | ✅                                                   |
| `appleIdfa`         | Mobile (platform) entity | ✅ if set                                            |
| `appleIdfv`         | Mobile (platform) entity | ✅ if set                                            |
| `androidIdfa`       | Mobile (platform) entity | ✅ if set                                            |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

## 4. Full anonymization

Full anonymization combines client-side and server-side anonymization to remove all user and session identifiers from events.

**Version 4.x:**

Avoid configuring unwanted identifiers for full anonymization in version 4.x of the React Native tracker.

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    sessionContext: false,     // No session entity
    serverAnonymization: true, // Collector won't set network_userid or capture IP
    },
});
```

| Identifier          | Location in event        | Included in event?                                  |
| ------------------- | ------------------------ | --------------------------------------------------- |
| `user_id`           | Atomic                   | ✅ if set                                            |
| `domain_userid`     | Atomic                   | ✅ if set                                            |
| `userId`            | Session entity           | ❌ no session entity                                 |
| `sessionId`         | Session entity           | ❌ no session entity                                 |
| `previousSessionId` | Session entity           | ❌ no session entity                                 |
| `appleIdfa`         | Mobile (platform) entity | ✅ if set                                            |
| `appleIdfv`         | Mobile (platform) entity | ✅ if set                                            |
| `androidIdfa`       | Mobile (platform) entity | ✅ if set                                            |
| `network_userid`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |
| `user_ipaddress`    | Atomic                   | ❌/✅ still present if you provided this in `Subject` |

**Version 1.3 - 2.x:**

```typescript
const tracker = createTracker(
    'appTracker',
    { endpoint: COLLECTOR_URL },
    {
        trackerConfig: {
            sessionContext: false,     // Session entity won't be added to events
            userAnonymisation: true,   // Remove user identifiers from Subject and platform entity
        },
        emitterConfig: {
            serverAnonymisation: true, // Collector won't set network_userid or capture IP
        },
    }
);
```

| Identifier          | Location in event        | Included in event?  |
| ------------------- | ------------------------ | ------------------- |
| `user_id`           | Atomic                   | ❌                   |
| `domain_userid`     | Atomic                   | ❌                   |
| `userId`            | Session entity           | ❌ no session entity |
| `sessionId`         | Session entity           | ❌ no session entity |
| `previousSessionId` | Session entity           | ❌ no session entity |
| `appleIdfa`         | Mobile (platform) entity | ❌                   |
| `appleIdfv`         | Mobile (platform) entity | ❌                   |
| `androidIdfa`       | Mobile (platform) entity | ❌                   |
| `network_userid`    | Atomic                   | ❌                   |
| `user_ipaddress`    | Atomic                   | ❌                   |

***

---

# Track client side properties with the React Native tracker
> Configure and update subject properties including user ID, IP address, timezone, language, and screen resolution at tracker initialization or runtime.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/client-side-properties/

The tracker allows you to add a persistent set of information through the `SubjectConfiguration` which represents the basic information about the user and the app.

- `userId` = null: a custom user identifier.
- `networkUserId` = null: override the network user ID (UUIDv4) assigned by the Collector and stored in cookies.
- `domainUserId` = null: a domain user ID (DUID). When using the [JavaScript tracker](/docs/sources/web-trackers/), this is a generated identifier stored in a first-party cookie. The React Native tracker does not assign it automatically.
- `useragent` = null: a custom user-agent. It overrides the user-agent used by default.
- `ipAddress` = null: an IP address (not automatically set).
- `timezone` (set by the tracker): the current timezone label.
- `language` (set by the tracker): the language set in the device.
- `screenResolution` (set by the tracker): the screen resolution.
- `screenViewPort` = null: the screen viewport size
- `colorDepth` = null: the color depth.

The fields tracked using `SubjectConfiguration` are relevant in client-side tracking. Some are set automatically in all events during enrichment, even when no Subject is added. These properties are marked with `*` below, and discussed below. Timezone, marked with `**`, is only set when a `Subject` is tracked with the event.

Add these fields to an event using `Subject`:

| Property           | Field in raw event | Column(s) in enriched event             |
| ------------------ | ------------------ | --------------------------------------- |
| `userId`           | `uid`              | `user_id`                               |
| `ipAddress`\*      | `ip`               | `user_ipaddress`                        |
| `timezone`\*\*     | `tz`               | `os_timezone`                           |
| `language`         | `lang`             | `br_lang`                               |
| `useragent`\*      | `ua`               | `useragent`                             |
| `viewport`         | `vp`               | `br_viewheight`, `br_viewwidth`         |
| `screenResolution` | `res`              | `dvce_screenheight`, `dvce_screenwidth` |
| `colorDepth`       | `cd`               | `br_colordepth`                         |
| `networkUserId`\*  | `tnuid`            | `network_userid`                        |
| `domainUserId`     | `duid`             | `domain_userid`                         |

As always, be aware of privacy when tracking [personal identifiable information](https://snowplow.io/blog/2020/09/06/user-identification-and-privacy/) such as email addresses or IP addresses. The tracker provides anonymous tracking functionality to mask certain user identifiers. Refer to the [section on anonymous tracking to learn more](/docs/sources/react-native-tracker/anonymous-tracking/).

## Overriding autogenerated event properties

All enriched Snowplow events contain values for `user_ipaddress`, `useragent`, and `network_userid`.

The `user_ipaddress` is automatically added to all enriched events (unless [anonymous tracking with server anonymization](/docs/sources/react-native-tracker/anonymous-tracking/) is enabled). To manually override this, use a `Subject` and set an `ipAddress` string; use an empty string to prevent IP address tracking. Alternatively, use the [IP anonymization enrichment](/docs/pipeline/enrichments/available-enrichments/ip-anonymization-enrichment/).

The `useragent` is also automatically added but it can be overriden on configuration. Snowplow pipelines provide multiple useragent-parsing [enrichments](/docs/pipeline/enrichments/available-enrichments/). To manually override the detected useragent, use a `Subject` and set a `useragent` string.

The `network_userid` is the cookie value for the event Collector's third-party cookie. The cookie is named `sp` (or `micro` for Snowplow Micro pipelines). To override the Collector cookie’s value with your own generated ID, use a `Subject` object and set `networkUserId`.

The `network_userid` is stored in the tracker and it's kept the same until the app is deleted or the Collector endpoint is changed or the cookie is expired. It is not assigned to events if anonymous tracking with server anonymization is enabled.

## Setting the subject configuration

The client-side properties can be set during tracker initialization using the `newTracker` configuration:

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    // pass the subject properties:
    userId: 'my-user-id',
});
```

See the full [list of options in the configuration section](/docs/sources/react-native-tracker/#configuring-the-tracker).

### Setting the subject data in a tracker instance

It is also possible to set or change the subject properties at runtime, using the `set..` methods of the React Native Tracker. The available methods are:

1. `setUserId`

With this method you can set the userId to a new string. To unset the userId, pass a null value as an argument.

```typescript
tracker.setUserId('newUser');
```

2. `setNetworkUserId`

With this method you can set the `network_userid` to a new string(UUIDv4). To unset, pass a null value as an argument.

```typescript
tracker.setNetworkUserId('44df44bc-8844-4067-9a89-f83c4fe1e62f');
```

3. `setDomainUserId`

With this method you can set the `domain_userid` to a new string(UUIDv4). To unset, pass a null value as an argument.

```typescript
tracker.setDomainUserId('0526be47-32cb-44b2-a9e6-fefeaa5ec6fa');
```

4. `setIpAddress`

With this method you can set the `user_ipaddress` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setIpAddress('123.45.67.89');
```

5. `setUseragent`

With this method you can set the `useragent` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setUseragent('some-useragent-string');
```

6. `setTimezone`

With this method you can set the `os_timezone` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setTimezone('Africa/Cairo');
```

7. `setLanguage`

With this method you can set the `br_lang` to a new string. To unset, pass a null value as an argument.

```typescript
tracker.setLanguage('fr');
```

8. `setScreenResolution`

With this method you can set the `dvce_screenwidth` and `dvce_screenheight` fields to new integer values. The argument to this method is an array that represents the ScreenSize as `[width, height]`. For example:

```typescript
tracker.setScreenResolution([123, 456]);
```

9. `setScreenViewport`

With this method you can set the `br_viewwidth` and `br_viewheight` fields to new integer values. The argument to this method is an array that represents the ScreenSize as `[width, height]`. For example:

```typescript
tracker.setScreenViewport([123, 456]);
```

10. `setColorDepth`

With this method you can set the `br_colordepth` to a new value. For example:

```typescript
tracker.setColorDepth(20);
```

Finally, there is an extra "wrapper" method to set may subject properties at once:

- `setSubjectData`

This method accepts as an argument a SubjectConfiguration, with the new values as needed. For example:

```typescript
tracker.setSubjectData({
    userId: 'tester',
    domainUserId: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    language: 'es',
    colorDepth: 50,
    screenResolution: [300, 300],
});
```

---

# Global context for React Native trackers
> Add global context entities to all events using context generators, filter functions, and rulesets with the React Native tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/

**Global context** (also known as global entities) lets you define your own contexts once (e.g. on tracker initialization) and then have this context sent with every single event subsequently recorded in the app. This saves having to manually build and send the context array with every single event fired.

Here is an example that adds a global context entity to all subsequently tracked events:

```typescript
// Create a context entity and add it to global context
let contextEntity = {
  schema: 'iglu:com.acme/user_context/jsonschema/1-0-0',
  data: { userid: 1234, name: 'John Doe' }
};
tracker.addGlobalContexts([contextEntity]);

// The global context will be added to this screen view event as well
tracker.trackScreenViewEvent({ name: 'my-screen-name' });
```

You may not want to add the same context entity to all tracked events. There are several ways to make global context dynamic:

1. Using a **[context generator](#context-generators)** that generates context entities on the fly when events are sent.
2. Using **[filter functions](#filter-functions)** that filter which events to add global context entities to.
3. Using **[rulesets](#rulesets)** that define rules which types of events to accept or reject when generating global context entities.

### Context generators

Generating context on-the-fly is accomplished with **context generators**. A context generator is a callback that will be evaluated with an optional argument that contains useful information.

A sample context generator that conditionally generates a context entity could look like this:

```javascript
const contextGenerator = (args) => {
    if (args.eventType == 'pv') {
        return {
            schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
            data: { test: 1 },
        };
    }
};
tracker.addGlobalContexts([contextGenerator]);
```

The optional input is an associative array that contains three elements:

- `event` : self-describing JSON
- `eventType` : string
- `eventSchema` : string (schema URI)

Keep in mind that the arguments `eventType` and `eventSchema` are data found in `event`. `eventType` and `eventSchema` are provided for convenience, so that simple tasks don't require users to search through the event payload.

#### `eventType`

This argument is a string taken from the event payload field, `e`.

`eventType` takes the following values:

| Type                         | `e` |
| ---------------------------- | --- |
| Screen view tracking         | ue  |
| Pageview tracking            | pv  |
| Custom structured event      | se  |
| Custom self describing event | ue  |

Further information about the event payload can be found in the [tracker protocol documentation](/docs/events/).

#### `eventSchema`

Users should be aware of the behavior of the argument `eventSchema`. Since 'first-class events' (e.g. structured events, transactions, pageviews, etc.) lack a proper schema (their event type is determined by the `e` field), callbacks will be provided the upper-level schema that defines the payload of all events:

`iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4`

For self describing events, `eventSchema` will be the schema that describes the self describing event, not the event payload. Again, this behavior isn't necessarily uniform, but provides more utility to differentiate events.

### Conditional context providers

We can augment context primitives by allowing them to be sent conditionally. While it's possible to define this functionality within context generators (with conditional logic), conditional context providers simplify common ways of sending contexts that follow certain rules.

The general form is an array of two objects:

`[conditional part, context primitive or [array of primitives]]`

The conditional part is standardized into two options:

- a filter function
- a schema ruleset

### Filter functions

Filter functions take the standard callback arguments defined for context generators, but instead of returning a Self Describing JSON, return a boolean value. As should be expected: `true` will attach the context part, `false` will not attach the context part.

### Example

```javascript
// A filter that will only attach contexts to structured events
function structuredEventFilter(args) {
  return args['eventType'] === 'se';
}
var globalContextDefinition = [structuredEventFilter, contextEntityToBeAdded];
tracker.addGlobalContexts([globalContextDefinition]);
```

### Rulesets

Rulesets define when to attach context primitives based on the event schema. This follows the standard behavior for all callbacks (the schema used to evaluate is the same provided in `eventSchema`, namely the payload schema for "first-class events" and otherwise the schema found within the self describing event).

Here's the specific structure of a ruleset, it's an object with certain optional rules that take the form of fields, each holding an array of strings:

```json
{
    accept: [],
    reject: []
}
```

Some examples, take note that wild-card matching URI path components is defined with an asterisk, `*`, in place of the component:

```javascript
// Only attaches contexts to this one schema
var ruleSetAcceptOne = {
  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0']
};

// Only attaches contexts to these schemas
var ruleSetAcceptTwo = {
  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0',
  'iglu:com.mailchimp/subscribe/jsonschema/1-0-0']
};

// Only attaches contexts to schemas with mailchimp vendor
var ruleSetAcceptVendor = {
  accept: ['iglu:com.mailchimp/*/jsonschema/*-*-*']
};

// Only attaches contexts to schemas that aren't mailchimp vendor
var ruleSetRejectVendor = {
  reject: ['iglu:com.mailchimp/*/jsonschema/*-*-*']
};

// Only attach to Snowplow first class events
var ruleSet = {
  accept: ['iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4']
};
```

You can add a global context entity with a ruleset as follows:

```javascript
var globalContextDefinition = [ruleSet, contextEntityToAdd];
tracker.addGlobalContexts([globalContextDefinition]);
```

### Rule requirements

All rules and schemas follow a standard form:

`protocol:vendor/event_name/format/version`

And rules must meet some requirements to be considered valid:

- Two parts are invariant: protocol and format. They are always `iglu` and `jsonschema` respectively.
  - Wildcards can therefore be used only in `vendor`, `event_name` and `version`.

- Version matching must be specified like so: `*-*-*`, where any part of the versioning can be defined, e.g. `1-*-*`, but only sequential parts are to be wildcarded, e.g. `1-*-1` is invalid but `1-*-*` is valid.

- Vendors require the first two "larger parts":
  - `com.acme.*`

- Vendors cannot be defined with non-wildcarded parts between wildcarded parts:

  - `com.acme.*.marketing.*` is invalid
  - `com.acme.*.*` is valid

### Named global entities

From version 4 onwards, you can alternatively supply the definitions as a mapping object. This associates each global entity with a "name" or "tag" that can be used to reference it in future calls.

For example, in the below we associate a global entity definition with the name `example`, and can then update or remove it by name in later calls.

```javascript
// add the entity defined by `globalContextDefinition` with the name "example"
var globalContextDefinition = [ruleSet, contextEntityToAdd];
tracker.addGlobalContexts({ example: globalContextDefinition });

// update the global entity named "example"; this replaces the entity from `globalContextDefinition`
var globalContextDefinition2 = [ruleSet, differentContextEntity];
tracker.addGlobalContexts({ example: globalContextDefinition2 });

// remove the global entity named "example"; it will no longer appear on subsequent events
tracker.removeGlobalContexts(["example"]);
```

Names can be whatever makes sense to you, but each name can only be associated with a single entity definition at a time (additions with the same name will replace the previous value). The value for the named entity can be the same as unnamed global entities defined with the array method, including generator functions, filter functions, and rulesets and are processed equivalently.

### Global contexts methods

These are the standard methods to add and remove global context entities:

#### `addGlobalContexts`

To add global contexts:

```javascript
// unnamed global entities
tracker.addGlobalContexts([array of global contexts]);

// named global entities
tracker.addGlobalContexts({
  name: globalContext1,
  name2: globalContext2,
});
```

#### `removeGlobalContexts`

To remove a global context:

```javascript
// unnamed global entities
tracker.removeGlobalContexts([array of global contexts])

// named global entities
tracker.removeGlobalContexts(["name", "name2"]) // array of entity names
```

Unnamed global context entities are removed by doing a JSON match of the context to be removed with the added context. So the objects have to be the same in order for them to be matched.

For example:

```javascript
var entity = {
  schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
  data: { test: 1 },
};
tracker.addGlobalContexts([entity]); // add a global context
tracker.removeGlobalContexts([entity]); // remove the global context
```

#### `clearGlobalContexts`

To remove all named and unnamed global context entities:

```javascript
tracker.clearGlobalContexts();
```

---

# Track custom events with the React Native tracker
> Track self-describing events and attach custom context entities using schemas with the React Native tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/custom-tracking-using-schemas/

Self-describing (self-referential) JSON schemas are at the core of Snowplow tracking. Read more about them [here](/docs/fundamentals/schemas/). They allow you to track completely customised data, and are also used internally throughout Snowplow pipelines.

In all our trackers, self-describing JSON are used in two places. One is in the `SelfDescribing` event type that wraps custom self-describing JSONs for sending. The second use is to attach entities to any tracked event. The entities can describe the context in which the event happen or provide extra information to better describe the event.

## Tracking self-describing events

You may wish to track events which are not directly supported by Snowplow. The solution is [self-describing events](/docs/fundamentals/events/#self-describing-events). Self-describing events are [based on JSON Schemas](/docs/fundamentals/schemas/) and can have arbitrarily many fields.

To define your own custom event, you will need to [create a corresponding schema](/docs/event-studio/data-structures/). Snowplow uses the schema to validate that the JSON containing the event properties is well-formed.

A Self Describing event is a [self-describing JSON](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

**Required properties**

- `schema`: (string) – A valid Iglu schema path. This must point to the location of the custom event’s schema, of the format: `iglu:{vendor}/{name}/{format}/{version}`.
- `data`: (object) – The custom data for your event. This data must conform to the schema specified in the `schema` argument, or the event will fail validation and become a [failed event](/docs/fundamentals/failed-events/).

To track a custom self-describing event, use the `trackSelfDescribingEvent` method of the tracker.

For example, to track a link-click event, which is one whose schema is already published in [Iglu Central](https://github.com/snowplow/iglu-central):

```typescript
tracker.trackSelfDescribingEvent({
    schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
    data: {targetUrl: 'http://a-target-url.com'}
});
```

### Tracking a custom entity

Custom context can be used to augment any standard Snowplow event type, including self-describing events, with additional data. We refer to this custom context as [entities](/docs/fundamentals/entities/).

The context is an array of entities. More than one entity (of either different or the same type) can be attached to an event. The `context` argument (if it is provided at all) should be a non-empty array.

As with self-describing events, if you want to create your own custom context, you will need to [create a corresponding schema](/docs/event-studio/data-structures/). Snowplow uses the schema to validate that the JSON containing the context properties is well-formed.

Custom contexts can be optionally added as an extra argument to any of the Tracker’s `track..()` methods.

**Note:** Even if only one custom context is being attached to an event, it still needs to be wrapped in an array. Also an empty array is acceptable, which will attach no entities to the event.

For example, a custom context to describe a screen could be:

```typescript
const myScreenContext: EventContext = {
    schema: 'iglu:com.example/screen/jsonschema/1-2-1',
    data: {
        screenType: 'test',
        lastUpdated: '2021-06-11'
    }
};
```

Another example custom context to describe a user on a screen could be:

```typescript
const myUserEntity: EventContext = {
    schema: 'iglu:com.example/user/jsonschema/2-0-0',
    data: {
        userType: 'tester'
    }
};
```

Then, to track, for example, a screenViewEvent with both of these contexts attached:

```typescript
tracker.trackScreenViewEvent(
   { name: 'myScreenName' },
   [ myScreenContext, myUserEntity ]
);
```

It is also possible to add custom contexts globally, so that they are applied to all events within an application. For more information, see [the Global Contexts section](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

---

# Track events from hybrid apps with the React Native tracker
> Track events from WebViews in React Native hybrid apps using event forwarding from the Web tracker or WebView tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/hybrid-apps/

> **Info:** This feature is available since v1.3. To use the Web plugin, you will need v4.2+ of the React Native tracker.

Hybrid apps are mobile apps that in addition to a React Native interface, provide part of the UI through an embedded Web view. Snowplow events are tracked from both the React Native code as well as the Web view. Our goal is to have both events tracked from both places to share the same session and appear as tracked with the same tracker.

## Event forwarding

We recommend using the Web tracker (v4.3+) to forward all Web events to the React Native tracker.

1. Implement the React Native tracker.

2. Implement the [Snowplow Web/JavaScript tracker](/docs/sources/web-trackers/) in the WebView in your app. Make sure to include the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/).

3. Subscribe to WebView event messages.

   ```typescript
   import { getWebViewCallback } from '@snowplow/react-native-tracker';

   const YourWebViewComponent = () => {
       return <WebView
           onMessage={getWebViewCallback()}
           source={{uri: WEB_VIEW_URI}}
           ... />;
   ```

4. Track events as usual.

The Web tracker will automatically intercept all web events and forward them to the React Native tracker. The forwarded events will have the tracker version from Web, e.g. "js-4.1.0", but will otherwise be tracked like the mobile events. They may contain additional information not present in the React Native mobile events, such as a browser useragent string or URL, or Web context entities e.g. the [WebPage entity](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity).

The forwarded events are filtered out of the Web tracker event queue so that they are not tracked twice.

The WebView plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## WebView Tracker

If you don't want to implement a Web tracker in your WebView, you can use the [Snowplow WebView tracker](/docs/sources/webview-tracker/) directly. This could be suitable if you only want to track a small number of events from the WebView. We recommend event forwarding for most use cases.

1. Implement the React Native tracker.

2. Implement the [Snowplow WebView tracker](/docs/sources/webview-tracker/) in the WebView in your app.

3. Subscribe to WebView event messages.

   ```typescript
   import { getWebViewCallback } from '@snowplow/react-native-tracker';

   const YourWebViewComponent = () => {
       return <WebView
           onMessage={getWebViewCallback()}
           source={{uri: WEB_VIEW_URI}}
           ... />;
   ```

4. Manually track [WebView events](/docs/sources/webview-tracker/).

All event types can be tracked with WebView tracker v0.3.0+, but it requires more work than using the event forwarding.

---

# React Native tracker SDK
> Track events in React Native apps for iOS, Android, and Web with the Snowplow tracker SDK supporting Expo and React Navigation.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/

The Snowplow React Native Tracker is purely implemented in JavaScript/TypeScript without the use of native iOS/Android modules. This enables it to support all these platforms and frameworks:

- React Native for mobile and Web
- Expo bare and managed workflow

The current version is 4.6.8.

## Initializing a tracker

The React Native Tracker can be configured with a set of configuration objects. Once you import the module into your app, the `newTracker` function can be used to setup a tracker, which is identified internally by its namespace.

The `newTracker` function can be used to create multiple instances of the tracker in the same app. If you call it using a namespace already used, it will reconfigure the tracker with the same namespace. If you call it with a different namespace, the tracker will create a new independent tracker.

```typescript
import { newTracker } from '@snowplow/react-native-tracker';

const tracker = newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
});
```

## Configuring the tracker

The `newTracker` provides a wide range of optional configuration options that let you configure the autotracking functionality of the tracker, specify how requests to the collector should be constructed and much more. The following snippet shows an overview of all the options and their meaning:

```typescript
const tracker = newTracker({
    // Network configuration:
    // Represents the network communication configuration allowing the tracker to be able to send events to the Snowplow collector.

    // URL of the collector that is going to receive the events tracked by the tracker. The URL can include the schema/protocol (e.g.: `http://collector-url.com`). In case the URL doesn’t include the schema/protocol, the HTTPS protocol is automatically selected.
    endpoint: COLLECTOR_URL,
    // HTTP method to use for the requests ('get' or 'post')
    eventMethod: 'post',
    // A custom path which will be added to the endpoint URL to specify the complete URL of the collector when paired with the POST method.
    postPath: 'com.snowplowanalytics.snowplow/tp2',
    // Custom headers for HTTP requests to the Collector
    customHeaders: {},

    // Tracker configuration:
    // Represents the configuration of the tracker and the core tracker properties indicating what should be tracked in terms of automatic tracking and contexts/entities to attach to the events.

    // Identifies the tracker instance
    namespace: "sp1",
    // Identifies the application
    appId: 'my-app-id',
    // Version of the application (e.g., semver or git commit hash)
    appVersion: '1.0.0',
    // Build number of the application
    appBuild: '1',
    // Platform of the device, allowed options are:
    // - `'mob'`: Mobile/Tablet
    // - `'web'`: Web(including mobile web)
    // - `'pc'`: Desktop/Laptop/Netbook
    // - `'srv'`: Server-side app
    // - `'app'`: General app
    // - `'tv'`: Connected TV
    // - `'cnsl'`: Games Console
    // - `'iot'`: Internet of things
    devicePlatform: 'mob',
    // Whether to base64 encode self-describing event and entity data
    encodeBase64: false,
    // Tracks a deep link entity with the first screen view following a deep link received event
    deepLinkContext: true,
    // Tracks a screen entity referring to the last screen view event
    screenContext: true,
    // Tracks foreground, background events and a lifecycle entity with the app state
    lifecycleAutotracking: true,
    // Tracks an install event on the first start
    installAutotracking: false,
    // Tracks screen_end event and screen_summary entity with information about the screen activity
    screenEngagementAutotracking: true,

    // Platform context configuration:
    // Tracks an entity with information about the device
    platformContext: true,
    // Chooses which optional properties of the platform context to track
    platformContextProperties: [
        PlatformContextProperty.Carrier,
        PlatformContextProperty.Language
    ],
    // Provides custom functions to retrieve the platform context information
    platformContextRetriever: {
        getAppAvailableMemory: () => Promise.resolve(1024),
    },

    // Tracker plugins – An array of plugins to be used by the tracker
    plugins: [SnowplowEcommercePlugin()],

    // Session configuration:
    // Represents the configuration of the Session context which gets attached to each event tracked and changes based on the timeout set for the inactivity of app when in foreground or background.

    // Whether to track session information
    sessionContext: true,
    // Timeout for the session when app is in foreground
    foregroundSessionTimeout: 1800,
    // Timeout for the session when app is in background
    backgroundSessionTimeout: 1800,

    // Emitter configuration:
    // Represents the tracker configuration from the emission perspective. It can be used to setup details about how the tracker should treat the events once they have been processed but not yet sent.

    // Number of events to buffer before making a request to collector
    bufferSize: 1,
    // Max size of POST requests
    maxPostBytes: 40000,
    // Max size of GET requests
    maxGetBytes: 40000,
    // Whether to anonymise server-side user identifiers including the `network_userid` and `user_ipaddress`
    serverAnonymization: false,
    // The maximum amount of events that will be buffered in the event store
    maxEventStoreSize: 1000,
    // Whether to use AsyncStorage for event store persistence or an in-memory event store
    useAsyncStorageForEventStore: true,

    // Subject configuration:
    // Represents the configuration of the subject. The SubjectConfiguration can be used to provide the basic information about the user.

    // The custom user identifier. Commonly used for user self-identification – for example after sign in
    userId: 'my-user-id',
    // Override the `network_userid` field assigned by collector. Should contain a valid UUID4 string.
    networkUserId: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    // Populates the `domain_userid` field, which is a user identifier generated and tracked by the JavaScript tracker on Web. Should contain a valid UUID4 string.
    domainUserId: '7cdd5ea8-b0f5-47ea-a8bb-5ec8e98cdbd6',
    // The custom useragent. It populates the `useragent` field.
    useragent: 'some-useragent-string',
    // The IP address of the user. It populates the `user_ipaddress` field.
    ipAddress: '123.45.67.89',
    // The current timezone label. Populates the `os_timezone` field.
    timezone: 'Europe/London',
    // The language set in the device.
    language: 'en',
    // The screen resolution. The screen resolution size can be set as an array of [width, height]. Populates the event fields `dvce_screenwidth` and `dvce_screenheight`.
    screenResolution: [123, 456],
    // The screen viewport. The screen viewport size can be set as an array of [width, height]. Populates the event fields `br_viewwidth` and `br_viewheight`.
    screenViewport: [123, 456],
    // The color depth. Populates the `br_colordepth` field.
    colorDepth: 20,
});
```

To learn more about the specific parts of the configuration and autotracked information, see:

- [Platform and application context entities.](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/)
- [Session tracking.](/docs/sources/react-native-tracker/tracking-events/session-tracking/)
- [App lifecycle tracking.](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/)
- [Screen tracking.](/docs/sources/react-native-tracker/tracking-events/screen-tracking/)
- [Installation tracking.](/docs/sources/react-native-tracker/tracking-events/installation-tracking/)

## Tracker methods

### Tracking events

The React Native Tracker can track a range of out-of-the-box events. As an example, the following snippet tracks a screen view event:

```ts
tracker.trackScreenViewEvent({
    name: 'my-screen-name',
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    type: 'carousel',
    transitionType: 'basic'
});
```

The available events are:

- [Self-describing events.](/docs/sources/react-native-tracker/custom-tracking-using-schemas/)
- [Structured events.](/docs/sources/react-native-tracker/tracking-events/#tracking-structured-events)
- [Screen view events.](/docs/sources/react-native-tracker/tracking-events/#tracking-screen-view-events)
- [Page view events.](/docs/sources/react-native-tracker/tracking-events/#tracking-page-view-events)
- [Timing events.](/docs/sources/react-native-tracker/tracking-events/#tracking-timing-events)

You can find out about all available track methods with details and examples in the [Tracking events](/docs/sources/react-native-tracker/tracking-events/) section that follows.

### Adding or removing global contexts

Global context lets you define your own context entities once and then have them sent with every single event subsequently recorded in the app. This saves having to manually build and send the context array with every single event fired and enables you to add entities to autotracked events.

The tracker methods to do so are:

- **addGlobalContexts**
- **removeGlobalContexts**
- **clearGlobalContexts**

You can find out more information and examples about [how to add or remove global contexts here](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

### Setting the Subject

It is possible to change subject data at runtime, as the user journey evolves. Once initialized, the React Native tracker provides the methods to do so for each subject property:

- **setUserId**
- **setNetworkUserId**
- **setDomainUserId**
- **setIpAddress**
- **setUseragent**
- **setTimezone**
- **setLanguage**
- **setScreenResolution**
- **setScreenViewport**
- **setColorDepth**

Alternatively, you can use the **setSubjectData** method to provide a new SubjectConfiguration.

You can find out more on how to use the tracker's methods to [set the subject information at runtime here](/docs/sources/react-native-tracker/client-side-properties/).

### Getting session context data from the tracker

You can also get back session data from the tracker at runtime, that may be useful for your tracking and data modeling design. The available methods are:

- **getSessionUserId**: To get the session user identifier.
- **getSessionId**: To get the session identifier.
- **getSessionIndex**: To get the session index.
- **getIsInBackground**: To get whether the app is currently in background state or not.
- **getBackgroundIndex**: To get the number of background transitions in the current session.
- **getForegroundIndex**: To get the number of foreground transitions in the current session.

You can find out more on how to use the tracker's methods to get session data at runtime in the corresponding [session tracking documentation](/docs/sources/react-native-tracker/tracking-events/session-tracking/#getting-session-data-from-the-tracker).

## Removing a tracker

The React Native Tracker API also provides functions to remove a tracker or remove all trackers at runtime. You can find out how in the [Removing trackers at runtime](/docs/sources/react-native-tracker/advanced-usage/) documentation.

---

# Migration guides for the React Native tracker
> Upgrade guides for React Native tracker versions with breaking changes and migration instructions.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/

This section contains information to help you upgrade to newer versions of the React Native tracker.

---

# Migrate from v0 to v1
> Migration guide for upgrading React Native tracker from v0.2.0 to v1.0.0 with API changes and configuration updates.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v0-x-to-v1/

This sections describes the differences between v0.2.0 and [v1.0.0](/docs/sources/react-native-tracker/) of the Snowplow React Native Tracker and the steps needed to upgrade, which is also recommended.

The v1 introduces a new API for initializing and configuring a tracker, which is a breaking change from v0. There are also few more changes to consider, for which you will find a dedicated section below.

In the following sections we assume a starting version of 0.2.0. If you have instrumented a tracker version prior to v0.2.0, you can begin from the [React Native Tracker v0 reference](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v0-reference/) page first.

## Tracker initialization

In the v1 of the React Native Tracker, a tracker is configured with a set of configuration objects, about which you can find detailed information [here](/docs/sources/react-native-tracker/).

The `createTracker` function now accepts 3 arguments:

1. **Required** - The tracker namespace
2. **Required** - The NetworkConfiguration
3. _Optional_ - The TrackerControllerConfiguration

In the following examples, the changes needed are described:

### Required arguments

**v0.2.0**

```typescript
import { createTracker } from '@snowplow/react-native-tracker';

const tracker = createTracker(
    'my-tracker-namespace',
    {
        endpoint: 'my-endpoint.com',
        appId: 'my-app-id'
    }
);
```

**v1.0.0**

```typescript
import { createTracker } from '@snowplow/react-native-tracker';

const tracker = createTracker(
    'my-tracker-namespace',
    {
      endpoint: 'my-endpoint.com',
    },
    {
        trackerConfig: {
            appId: 'my-app-id'
        }
    }
);
```

**Note**: The `appId` is **not** a required argument for v1. It is only included in the examples, since it was a required argument for v0.2.0.

### Endpoint

In v0.2.0, the collector URL cannot include the schema/protocol, which can be configured using the `protocol` property (which defaults to HTTPS).

In v1, the `protocol` property has been removed. The collector URL can include the schema/protocol (e.g.: `http://collector-url.com`). In case the URL doesn't include the schema/protocol, the HTTPS protocol is automatically selected.

**v0.2.0 - example http**

```typescript
const tracker = createTracker(
    'my-tracker-namespace',
    {
        endpoint: 'my-endpoint.com',
        protocol: 'http',
        appId: 'my-app-id'
    }
);
```

**v1.0.0 - example http**

```typescript
const tracker = createTracker(
    'my-tracker-namespace',
    {
      endpoint: 'http://my-endpoint.com',
    },
    {
        trackerConfig: {
            appId: 'my-app-id'
        }
    }
);
```

### Initialization options

The tracker initialization options in v0.2.0 are fairly limited compared to the ones [available for v1](/docs/sources/react-native-tracker/quick-start-guide/#instrumentation).

The following example depicts how to map the configuration options of v0.2.0 to v1 configuration.

**v0.2.0**

```typescript
const tracker = createTracker('my-namespace', {
    endpoint: 'my-endpoint.com',
    appId: 'my-app-id',
    method: 'post',
    base64Encoded: true,     // (a)
    platformContext: true,
    applicationContext: true,
    sessionContext: true,
    screenContext: true,
    foregroundTimeout: 600,  // (b)
    backgroundTimeout: 300,  // (c)
    checkInterval: 15,       // (d)
    lifecycleEvents: true,   // (e)
    installTracking: true    // (f)
  }
);
```

**v1.0.0**

```typescript
const tracker = createTracker(
    'my-namespace',
    {
        endpoint: 'my-endpoint.com'
        method: 'post'
    },
    {
        trackerConfig: {
            appId: 'my-app-id',
            base64Encoding: true,    // (a)
            platformContext: true,
            applicationContext: true,
            sessionContext: true,
            screenContext: true,
            lifecycleAutotracking: false,  // (e)
            installAutotracking: true,     // (f)
        },
        sessionConfig: {
            foregroundTimeout: 1800,  // (b)
            backgroundTimeout: 1800   // (c)
        }
    }
);
```

- **(a)** - Renamed `base64Encoded` to `base64Encoding`, which becomes a `TrackerConfig` property
- **(b)** - Default value change for `foregroundTimeout`, which becomes a `SessionConfig` property
- **(c)** - Default value change for `backgroundTimeout`, which becomes a `SessionConfig` property
- **(d)** - `checkInterval` is removed
- **(e)** - Renamed `lifecycleEvents` to `lifecycleAutotracking`
- **(f)** - Renamed `installTracking` to `installAutotracking`

## Tracking events

Generally, the tracking methods of v1 continue to have the same argument logic:

```typescript
// pseudocode
tracker.track..(eventData, eventContexts);
```

The differences are about the eventData properties, where they now match the corresponding schema properties. This affects:

### `trackScreenViewEvent`

**v0.2.0**

```typescript
tracker.trackScreenViewEvent({
  screenName: 'my-screen-name',   // (a)
  screenType: 'carousel',         // (b)
  transitionType: 'basic'
});
```

**v1.0.0**

```typescript
tracker.trackScreenViewEvent({
    name: 'my-screen-name',                      // (a)
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    type: 'carousel',                            // (b)
    previousName: 'previous-screen',
    previousId: '00d71340-342e-4f3d-b9fd-4de728ffba7a',
    previousType: 'feed',
    transitionType: 'basic'
});
```

Apart from the fact that more ScreenViewEvent properties are available, the changes to v0.2.0 ScreenViewEvent properties are:

- **(a)** - `screenName` becomes `name`
- **(b)** - `screenType` becomes `type`

### `trackPageViewEvent`

**v0.2.0**

```typescript
tracker.trackPageViewEvent({
  pageUrl: 'https://my-url.com',
  pageTitle: 'My page title',
  pageReferrer: 'http://some-other-url.com'  // (a)
});
```

**v1.0.0**

```typescript
tracker.trackPageViewEvent({
  pageUrl: 'https://my-url.com',
  pageTitle: 'My page title',
  referrer: 'http://some-other-url.com'      // (a)
});
```

The changes to v0.2.0 PageViewEvent properties are:

- **(a)** - `pageReferrer` becomes `referrer`

## Setting the subject

In v1.0.0, setting the subject can be done both when configuring the tracker (through the [SubjectConfiguration](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v2-reference/introduction/#configuring-the-tracker) object) and at runtime ([using the `set..` tracker methods](/docs/sources/react-native-tracker/previous-version/react-native-tracker-v2-reference/client-side-properties/#setting-the-subject-data-in-a-tracker-instance)), so the exact migration steps depend on the specifics of your app.

A notable difference between the SubjectConfiguration properties of v1.0.0 from the `SubjectData` properties of v0.2.0 is the way to set screen dimensions either for the screen resolution or for the screen viewport.

**v0.2.0**

```typescript
tracker.setSubjectData({
    screenWidth: 111,
    screenHeight: 222,
    viewportWidth: 333,
    viewportHeight: 444
});
```

**v1.0.0**

```typescript
tracker.setSubjectData({
    screenResolution: [111, 222],
    screenViewport: [333, 444]
});

// or, through more specific methods
tracker.setScreenResolution([111, 222]);
tracker.setScreenViewport([333, 444]);
```

In v1.0.0, the screen dimensions are not specified separately. Instead, the ScreenSize is represented through an array: `[width, height]`.

## Installation considerations for iOS

Because of a version mismatch [issue](https://github.com/snowplow-incubator/snowplow-react-native-tracker/issues/110) that existed in v0.2.0 between the package version and the podspec version of the React Native Tracker, the following steps are needed in order to make sure the RNSnowplowTracker Pod references the new v1.0.0.

```bash
# remove olds Pods and Podfile.lock
rm -rf Pods
rm Podfile.lock

# clean the caches
rm -rf ~/Library/Caches/CocoaPods
rm -rf ~/Library/Developer/Xcode/DerivedData/*

# deintegrate
pod deintegrate

# setup
pod setup

# install pods
pod install
```

---

# Migrate from v1 to v2
> Migration guide for upgrading React Native tracker from v1 to v2 with installation changes, iOS platform updates, and automatic tracking configuration.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v1-x-to-v2/

Version 2 of the React Native tracker underwent a large rewrite of the internal code, both on the side of the React Native project as well as in the underlying iOS and Android trackers (which have been upgraded to version 5). However, the public APIs and tracker behavior have remained largely unchanged except for a few things discussed on this page.

## Installation on iOS

In addition to adding the tracker as a dependency in your `package.json`, you will also need to add the FMDB dependency to you `ios/Podfile` (unless using Expo Go) with `modular_headers` enabled. Add this line to the end of the file:

```rb
pod 'FMDB', :modular_headers => true
```

## Supported iOS platforms

The supported platforms have changed on iOS:

- Minimum iOS deployment target changed from 9.0 to 11.0.
- On macOS, from 10.10 to 10.13.
- On tvOS, from 9.0 to 12.0.
- On watchOS, from 2.0 to 6.0.

## IDFA tracking on iOS

The approach to track the IDFA identifier has changed in the underlying iOS tracker version 5. This means that you will need to take a different approach to set it up. The new approach [is described in the documentation here](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

## Automatic screen view tracking default

The automatic screen view tracking configuration (set by `TrackerConfiguration.screenViewAutotracking`) has been changed to disabled by default. In case you rely on the default value and want to keep using the feature, please set it manually when creating a new tracker.

The feature was disabled because it does not track screen views for React Native screens but for UIKit and Android Activity screens which is not what users expected in most cases.

---

# Migrate from v2 to v4
> Migration guide for upgrading React Native tracker from v2 to v4 with JavaScript-only architecture, new platform support, and configuration changes.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/migration-guides/migrating-from-v2-x-to-v4/

Version 4 of the React Native tracker brings a significant rearchitecture of the tracker, which is now JavaScript-only instead of building on top of the iOS and Android native trackers. This brings support for new platforms (Web, Expo Go) and new features (JS tracker plugins, improved global context), but also brings some limitations.

### Initialize tracker instance using `newTracker` instead of `createTracker`

The method for creating a new tracker instance has been changed to `newTracker`. This is not just a rename, but a change in the structure of its parameters, which are now more in line with the rest of our JavaScript trackers.

You can review the [documentation page](/docs/sources/react-native-tracker/) for the current set of configuration options.

The parameters are now provided in a flattened object instead of separated under categories. Some have also been renamed:

- `method` to `eventMethod`
- `requestHeaders` to `customHeaders`
- `base64Encoding` to `encodeBase64`
- `serverAnonymisation` to `serverAnonymization`
- `customPostPath` to `postPath`

The `logLevel` parameter has been removed. Also global context configuration has been moved outside of the configuration, see [the global context docs page](/docs/sources/react-native-tracker/custom-tracking-using-schemas/global-context/).

### User anonymization not available yet

The `userAnonymisation` configuration option has been removed. We will reintroduce it in the upcoming versions of the React Native tracker.

The session context entity can still be removed using the `sessionContext` option and one can disable the user identifiers in the platform context and subject properties individually. Read more about the options [in the documentation here](/docs/sources/react-native-tracker/anonymous-tracking/).

### GDPR configuration not available

We have deprecated and removed the GPDR configuration. As a more up-to-date replacement, one can make use of the [consent plugins available for the JavaScript tracker](/docs/sources/web-trackers/tracking-events/consent-gdpr/).

### Application context is not tracked automatically

The application context entity with the current app version and build number is not tracked automatically anymore, but the version information can be provided in the `newTracker` call in which case the tracker will track this entity with events. Read more [in the documentation here](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

### Removed geolocation context entity

The geolocation context entity is no longer available to be tracked by the tracker. You may track it manually using global context and [the geolocation schema](/docs/events/ootb-data/geolocation/#geolocation-entity).

### Removed exception and diagnostic autotracking

The exception and diagnostic autotracking features have been deprecated and removed.

### Removed screen view autotracking for UIKit and Android Activities

Screen view autotracking for iOS UIKit and Android Activity views has been deprecated and removed as it is not relevant in most React Native apps. See the [documentation for screen view tracking options](/docs/sources/react-native-tracker/tracking-events/screen-tracking/).

### Less autotracked information in the platform context entity

There are fewer automatically tracked properties in the platform context entity, but it's possible to provide values for them manually. Refer to [the documentation to learn more](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/).

### Not possible to access from native code

Since the tracker is now purely implemented in JavaScript, it is no longer possible to access it from mobile native iOS or Android code.

### Base64 encoding disabled by default

In v2, the tracker defaulted to base64-encoding payloads for Self-Describing events and Entities. In v4, the default is now to not do this. In most cases, this behavior is better for performance and has no other impacts.

In some pipeline architectures, the Collector is protected by a Web Application Firewall; if your custom data contains content that appears risky to the WAF (e.g. HTML content) the removal of base64 encoding may mean those events now get blocked by the firewall.

You can return to the old behavior by specifying `encodeBase64: true` when calling `newTracker` as described above.

---

# React Native tracker quick start guide
> Install and initialize the React Native tracker using npm, configure the endpoint, and start tracking events in your mobile app.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/quick-start-guide/

In order to start sending events using the Snowplow React Native Tracker, the following steps are involved:

## Installation

To install the tracker, add it as a dependency to your React Native app:

```bash
npm install --save @snowplow/react-native-tracker
```

Note that in addition to `react` and `react-native`, the tracker has two peer dependencies that need to be added as dependencies to your app:

1. `@react-native-async-storage/async-storage` for event and session local storage implementation.
2. `react-native-get-random-values` as a polyfill for the Crypto Web APIs.

## Instrumentation

Next, in your app create a new tracker using the `newTracker` method. As a minimal example:

```typescript
import { newTracker } from '@snowplow/react-native-tracker';

const tracker = newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
});
```

The `newTracker` function takes the tracker configuration as the only argument. There are two required properties within the configuration:

1. The tracker namespace, which identifies each tracker instance.
2. The collector URL endpoint.

## Track events

Once the tracker is initialized, you can use the tracker methods to track events, about which you can find out more in the following Tracking events section. As an example, you can track a screen view event like this:

```ts
tracker.trackScreenViewEvent({
    name: 'my-screen-name',
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    type: 'carousel',
    transitionType: 'basic'
});
```

---

# Track out-of-the-box events with the React Native tracker
> Track screen views, structured events, page views, timing events, and ecommerce transactions with the React Native tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/

The React Native tracker captures two types of out-of-the-box events, automatically captured and manual events.

## Auto-tracked events and entities

Many of the automatic tracking options available on iOS and Android are also available in React Native – these can be enabled or disabled in the TrackerConfiguration passed to `newTracker` as part of the TrackerController configuration object:

```typescript
const tracker = newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,

    // auto-tracked events:
    lifecycleAutotracking: false,
    installAutotracking: true,
    screenEngagementAutotracking: true,

    // auto-tracked context entities:
    applicationContext: true,
    platformContext: true,
    sessionContext: true,
    deepLinkContext: true,
    screenContext: true,
);
```

The automatically captured data are:

- [**Platform and Application Context Tracking**](/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/): Captures contextual information about the device and the app.
- [**Session Tracking**](/docs/sources/react-native-tracker/tracking-events/session-tracking/): Captures the session which helps to keep track of the user activity in the app.
- [**App Lifecycle Tracking**](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/): Captures application lifecycle state changes (foreground/background transitions).
- [**Screen View and Engagement Tracking**](/docs/sources/react-native-tracker/tracking-events/screen-tracking/): Captures each time a new “screen” is loaded.
- [**Installation Tracking**](/docs/sources/react-native-tracker/tracking-events/installation-tracking/): Captures an install event which occurs the first time an application is opened.

## Manually-tracked events

All tracker's `track` methods take two arguments: An object of key-value pairs for the event’s properties, and an optional array of [custom event contexts](/docs/sources/react-native-tracker/custom-tracking-using-schemas/).

### Tracking screen view events

Track the user viewing a screen within the application.

To track a ScreenViewEvent, use the `trackScreenViewEvent` tracker method. For example:

```typescript
tracker.trackScreenViewEvent({
    name: 'my-screen-name',
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    type: 'carousel',
    transitionType: 'basic'
});
```

The tracker will automatically assign references to the previously tracked screen view event in the `previousName`, `previousId`, and `previousType` properties.

**Required properties**

- `name`: (string) - The name of the screen viewed

**Optional properties**

- `id`: (string) - The id(UUID) of screen that was viewed
- `type`: (string) - The type of screen that was viewed
- `previousName`: (string) - The name of the previous screen that was viewed (assigned automatically)
- `previousId`: (string) - The id(UUID) of the previous screen that was viewed (assigned automatically)
- `previousType`: (string) - The type of screen that was viewed (assigned automatically)
- `transitionType`: (string) - The type of transition that led to the screen being viewed

### Tracking page view events

Track a page view event with the `trackPageViewEvent()` method. Typically this is uncommon in apps, but is sometimes used where fitting data into an existing page views model is required. To track page views from an in-app browser, it is advisable to use the javascript tracker in-browser.

An example:

```typescript
tracker.trackPageViewEvent({
  pageUrl: 'https://my-url.com',
  pageTitle: 'My page title',
  referrer: 'http://some-other-url.com'
});
```

Required properties

- `pageUrl`: (string) – Page Url for the page view event. Must be a valid url.

Optional properties

- `pageTitle`: (string) – Page Title for the page view event.
- `referrer`: (string) – Url for the referring page to the page view event. Must be a vaild url.

### Tracking timing events

Use the `trackTimingEvent` tracker method to track user timing events such as how long resources take to load.

For example:

```typescript
tracker.trackTimingEvent({
    category: 'timing-category',
    variable: 'timing-variable',
    timing: 5,
    label: 'optional-label'
});
```

**Required properties**

- `category`: (string) - Defines the timing category
- `variable`: (string) - Define the timing variable measured
- `timing`: (number) - Represent the time

**Optional properties**

- `label`: An optional string to further identify the timing event

### Tracking consent granted events

Use the `trackConsentGrantedEvent` method to track a user opting into data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied.

For example:

```typescript
tracker.trackConsentGrantedEvent({
    expiry: '2022-01-01T00:00:00Z',
    documentId: 'doc-id',
    version: '1.2',
    name: 'doc-name',
    documentDescription: 'consent doc description'
});
```

**Required properties**

- `expiry`: (string) - The expiry (date-time string, e.g.: '2022-01-01T00:00:00Z')
- `documentId`: (string) - The consent document id
- `version`: (string) - The consent document version

**Optional properties**

- `name`: (string) - The consent document name
- `documentDescription`: (string) - The consent document description

### Tracking consent withdrawn events

Use the `trackConsentWithdrawnEvent` method to track a user withdrawing consent for data collection. A consent document context will be attached to the event using the `id` and `version` arguments supplied. To specify that a user opts out of all data collection, `all` should be set to `true`.

For example:

```typescript
tracker.trackConsentWithdrawnEvent({
    all: true,
    documentId: 'doc-id',
    version: '1.2',
    name: 'doc-name',
    documentDescription: 'consent doc description'
});
```

**Required properties**

- `all`: (boolean) - Whether user opts out of all data collection
- `documentId`: (string) - The consent document id
- `version`: (string) - The consent document version

**Optional properties**

- `name`: (string) - The consent document name
- `documentDescription`: (string) - The consent document description

### Tracking ecommerce transaction events

Modelled on Google Analytics ecommerce tracking capability, an ecommerce-transaction event can be tracked as follows:

1. **Create a EcommerceTransaction event object**. This will be the object that is loaded with all the properties relevant to the specific transaction that is being tracked including all the items (see `EcommerceItem` right below) in the order, the prices of the items, the price of shipping and the `order_id`.
2. **Track the transaction** using the `trackEcommerceTransaction` method.

The procedured outlined above will result in the following events being tracked:

1. An EcommerceTransaction event
2. As many as EcommerceItem events as the items passed to the EcommerceTransaction object, that will also inherit the `orderId` from the parent transaction event

#### EcommerceItem

More specifically, to start with, the properties of an `EcommerceItem` are:

**Required properties**

- `sku`: (string) - The item sku
- `price`: (number) - The item price
- `quantity`: (number) - The quantity purchased

**Optional properties**

- `name`: (string) - The item name
- `category`: (string) - The item category
- `currency`: (string) - The item currency

For example:

```typescript
const ecomItem: EcommerceItem = {
    sku: 'DD44',
    name: 'T-Shirt',
    category: 'Green Medium',
    price: 15,
    quantity: 1,
    currency: 'USD'
};
```

#### EcommerceTransaction

An ecommerce transaction object has the following properties:

**Required properties**

- `orderId`: (string) - The order ID of the transaction
- `totalValue`: (number) - The total value of the transaction
- `items`: (array of `EcommerceItem`) - The ecommerce items purchased in the transaction.

**Optional properties**

- `affiliation`: (string)
- `taxValue`: (number)
- `shipping`: (number)
- `city`: (string)
- `state`: (string)
- `country`: (string)
- `currency`: (string)

```typescript
const ecomTransaction: EcommerceTransactionProps = {
    orderId: '1234',
    totalValue: 15,
    items: [ ecomItem ],
    affiliation: 'Womens Apparel',
    taxValue: 1.5,
    shipping: 2.99,
    city: 'San Jose',
    state: 'California',
    country: 'USA',
    currency: 'USD'
};
```

#### `trackEcommerceTransaction`

Then, to track the ecommerce transaction as described in the above examples:

```typescript
tracker.trackEcommerceTransactionEvent(ecomTransaction);
```

### Tracking deep link received events

The Deep Link is received by the mobile operating system and passed to the related app. Our React Native tracker can't automatically track the Deep Link, but we provide an out-of-the-box event that can be used by the developer to manually track it as soon as the Deep Link is received in the app.

It will be the duty of the tracker to automatically attach the information of the Deep Link to the first Screen View tracked.

In practice, when the app receives a Deep Link the developer can track it through the `trackDeepLinkReceivedEvent` endpoint:

```typescript
tracker.trackDeepLinkReceivedEvent({
    url: 'https://deeplink.com',
    referrer: 'http://refr.com',
});
```

Please refer to the [Linking API](https://reactnative.dev/docs/linking) in React Native for information on how to retrieve the linked URLs.

The tracker keeps memory of the tracked Deep Link event and will attach a Deep Link entity to the first ScreenView tracked in the tracker. This is helpful during the analysis of the data because it will be clear the relation between the content visualized by the user (ScreenView event) and source (DeepLink entity) that originated that visualisation.

This behavior is enabled by default but it can be disabled using the `deepLinkContext` boolean property when passing `trackerConfig` to `createTracker`:

```typescript
const tracker = createTracker(
    namespace, networkConfig,
    {
        trackerConfig: {
            deepLinkContext: false
        }
    }
);
```

The Deep Link Received event can be used in pair with a campaign-attribution-enrichment appropriately enabled in the Snowplow pipeline. It works exactly like for Page View events in the web/JS tracker. When the user taps on an advertising banner or a marketing email or message, it can trigger the launch of the app through the Deep Linking feature. The referral from the advertising campaigns, websites, or other source can be composed by UTM parameters used to attribute the user activity back to the campaign. The Campaign Attribution Enrichment can parse the DeepLinkReceived event extracting the UTM parameters in the deep link url.

Required properties

- `url`: (string) – URL in the received deep-link

Optional properties

- `referrer`: (string) – Referrer URL, source of this deep-link

### Tracking push and local notification events

Push Notifications are a cornerstone of the user experience on mobile. A Push Notification is a message (like an SMS or a mobile alert) that can quickly show information to the user even if an app is closed or the phone is in stand-by. Push Notifications can be used in many different ways: they can notify of a new message in a chat, rather than informing of a news or just notify of an achievement in a fitness app.

To track an event when a push (or local) notification is used, it is possible to use the `trackMessageNotificationEvent` endpoint:

```typescript
tracker.trackMessageNotificationEvent({
    title: 'title',
    body: 'body',
    trigger: 'push',
    action: 'action',
    attachments: [
        {
            identifier: 'att_id',
            type: 'att_type',
            url: 'http://att.url',
        },
    ],
    bodyLocArgs: ['bodyArg1', 'bodyArg2'],
    bodyLocKey: 'bodyKey',
    category: 'category',
    contentAvailable: true,
    group: 'group',
    icon: 'icon',
    notificationCount: 3,
    notificationTimestamp: '2022-02-02T15:17:42.767Z',
    sound: 'chime.mp3',
    subtitle: 'subtitle1',
    tag: 'tag',
    threadIdentifier: 'threadIdentifier',
    titleLocArgs: ['titleArg1', 'titleArg2'],
    titleLocKey: 'titleKey',
});
```

Required properties

- `title`: (string) – The notification's title.
- `body`: (string) – The notification's body.
- `trigger`: (string) – The trigger that raised the notification message. Must be one of: push, location, calendar, timeInterval, other.

Optional properties

- `action`: (string) – The action associated with the notification.
- `attachments`: (array of `MessageNotificationAttachmentProps`) – Attachments added to the notification (they can be part of the data object).
- `bodyLocArgs` (array of string) - Variable string values to be used in place of the format specifiers in bodyLocArgs to use to localize the body text to the user's current localization.
- `bodyLocKey`: (string) – The key to the body string in the app's string resources to use to localize the body text to the user's current localization.
- `category`: (string) – The category associated to the notification.
- `contentAvailable`: (boolean) – The application is notified of the delivery of the notification if it's in the foreground or background, the app will be woken up (iOS only).
- `group`: (string) – The group which this notification is part of.
- `icon`: (string) – The icon associated to the notification (Android only).
- `notificationCount`: (integer) – The number of items this notification represent.
- `notificationTimestamp`: (string) – The time when the event of the notification occurred.
- `sound`: (string) – The sound played when the device receives the notification.
- `subtitle`: (string) – The notification's subtitle. (iOS only)
- `tag`: (string) – An identifier similar to 'group' but usable for different purposes (Android only).
- `threadIdentifier`: (string) – An identifier similar to 'group' but usable for different purposes (iOS only).
- `titleLocArgs`: (array of string) – Variable string values to be used in place of the format specifiers in titleLocArgs to use to localize the title text to the user's current localization.
- `titleLocKey`: (string) – The key to the title string in the app's string resources to use to localize the title text to the user's current localization.

#### MessageNotificationAttachmentProps

Attachment object that identifies an attachment in the Message Notification event.

**Required properties**

- `identifier`: (string) – Attachment identifier.
- `type`: (string) – Attachment type.
- `url`: (string) – Attachment URL.

For example:

```typescript
const attachment: MessageNotificationAttachmentProps = {
    identifier: 'id',
    type: 'type',
    url: 'http://att.url',
};
```

### Tracking structured events

Our philosophy in creating Snowplow is that users should capture important consumer interactions and design suitable data structures for this data capture. You can read more about that philosophy [here](/docs/event-studio/). Using `trackSelfDescribingEvent` captures these interactions with custom schemas, as desribed above.

However, as part of a Snowplow implementation there may be interactons where custom Self Describing events are perhaps too complex or unwarranted. They are then candidates to track using `trackStructuredEvent`, if none of the other event-specific methods outlined below are appropriate.

For example:

```typescript
tracker.trackStructuredEvent({
  category: 'my-category',
  action: 'my-action',
  label: 'my-label',
  property: 'my-property',
  value: 50.00
});
```

**Required properties**

- `category`: The name you supply for the group of objects you want to track e.g. ‘media’, ‘ecomm’
- `action`: A string which defines the type of user interaction for the web object e.g. ‘play-video’, ‘add-to-basket’

**Optional properties**

- `label`: (string) - identifies the specific object being actioned e.g. ID of the video being played, or the SKU or the product added-to-basket
- `property`: (string) - describes the object or the action performed on it. This might be the quantity of an item added to basket
- `value`: (number) - quantifies or further describes the user action. This might be the price of an item added-to-basket, or the starting time of the video where play was just pressed

---

# Track installs with the React Native tracker
> Track app installation events automatically on first launch with the React Native tracker installation autotracking feature.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/installation-tracking/

Installation tracking tracks an install event which occurs the first time an application is opened. The tracker will record when it's first been installed, so deleting and reinstalling an app will trigger another install event.

If installation autotracking is not enabled, the tracker will still keep track of when the app was first installed, so that when enabled, the tracker will send the recorded install event with a timestamp reflecting when it was first installed.

The installation autotracking is disabled by default. It can be set in `TrackerConfiguration` like in the example below:

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    installAutotracking: true, // disabled by default
});
```

---

# Track application lifecycle changes with the React Native tracker
> Automatically track foreground and background transitions with lifecycle events and attach lifecycle context entities to all events.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/

The tracker can capture application lifecycle state changes. In particular, when the app changes state from foreground to background and vice versa.

The lifecycle tracking is enabled by default. It can be disabled using the `lifecycleAutotracking` option like in the example below:

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    lifecycleAutotracking: true, // enabled by default
});
```

Once enabled, the tracker will automatically track a [`Background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event) when the app is moved to background and a [`Foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event) when the app moves back to foreground (becomes visible in the screen).

The tracker attaches a [`LifecycleEntity`](/docs/events/ootb-data/mobile-lifecycle-events/#lifecycle-entity) to all the events tracked by the tracker reporting if the app was visible (foreground state) when the event was tracked.

The `LifecycleEntity` value is conditioned by the internal state of the tracker only. To make an example, if the app is in foreground state but the developer tracks a `Background` event intentionally, it would force the generation of a `LifecycleEntity` that mark the app as non visible, even if it's actually visible in the device.

---

# Track platform and application data with the React Native tracker
> Track platform context with device information and application context with app version and build using the React Native tracker.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/platform-and-application-context/

Platform and application data tracking features capture information about the device and the app.

They can be configured through `TrackerConfiguration` like in the example below:

```typescript
const tracker = await newTracker({
  namespace: 'appTracker',
  endpoint: COLLECTOR_URL,

  // application information
  appId: 'my-app-id',
  appVersion: '1.0.0',
  appBuild: '1',

  platformContext: true, // enabled by default
});
```

## Application information

### App ID

Set the application ID using the `appId` field of the tracker configuration object. This will be attached to every event the tracker fires. The information will be available in the `app_id` column of the events table.

### App version and build number

To track the application version, you can pass it using the `appVersion` configuration option. The version of can be a semver-like structure (e.g 1.1.0) or a Git commit SHA hash.

Additionally, you can track the build name of the application (e.g., `s9f2k2d` or `1.1.0 beta`) using the `build` option.

Depending on whether `build` property is configured, the information will be tracked in a context entity using either of these schemas:

1. If only `appVersion` is configured, [the Web `application` entity is used](/docs/events/ootb-data/app-information/#entity-definitions).
2. If both `appVersion` and `build` are configured, [the mobile `application` entity is used](/docs/events/ootb-data/app-information/#entity-definitions).

## Platform context

The [platform context entity](/docs/events/ootb-data/device-and-browser/#mobile-entity) contains information about the user's device.

By default only the following properties are tracked automatically:

| Property             | Type   | Description                                                                                                                                             | Required in schema |
| -------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `osType`             | String | Type of the operating system (e.g., "ios", "tvos", "watchos", "osx", "android")                                                                         | Yes                |
| `osVersion`          | String | Version of the mobile operating system.                                                                                                                 | Yes                |
| `deviceManufacturer` | String | Device vendor.                                                                                                                                          | Yes                |
| `deviceModel`        | String | Model of the device.                                                                                                                                    | Yes                |
| `language`           | String | System language currently used on the device (ISO 639)                                                                                                  | No                 |
| `resolution`         | String | Screen resolution in pixels. Arrives in the form of WIDTHxHEIGHT (e.g., 1200x900). Doesn't change when device orientation changes                       | No                 |
| `scale`              | Number | Scale factor used to convert logical coordinates to device coordinates of the screen (uses UIScreen.scale on iOS and DisplayMetrics.density on Android) | No                 |

The platform context entity can be used to track more information. However, you will need to pass it manually using callback functions.

```ts
const tracker = await newTracker({
  namespace: "sp1",
  endpoint: "http://example.com",
  // provide custom getters for the platform context properties
  platformContextRetriever: {
    getAppleIdfv: async () => "apple-idfv",
    getAppSetId: async () => "app-set-id",
    getAvailableStorage: async () => 1000,
  }
});
```

The following table lists all the extra properties that can be set:

| Property                | Type                                 | Description                                                                                                                                                             | Required in schema |
| ----------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `carrier`               | String                               | Carrier of the SIM inserted in the device.                                                                                                                              | No                 |
| `networkType`           | String                               | One of: "mobile", "wifi", "offline"                                                                                                                                     | No                 |
| `networkTechnology`     | String                               | Radio access technology that the device is using.                                                                                                                       | No                 |
| `openIdfa`              | String                               | Deprecated property.                                                                                                                                                    | No                 |
| `appleIdfa`             | String                               | Advertising identifier on iOS.                                                                                                                                          | No                 |
| `appleIdfv`             | String                               | UUID [identifier for vendors](https://developer.apple.com/documentation/uikit/uidevice/1620059-identifierforvendor) on iOS.                                             | No                 |
| `androidIdfa`           | String                               | Advertising identifier on Android.                                                                                                                                      | No                 |
| `physicalMemory`        | Integer                              | Total physical system memory in bytes                                                                                                                                   | No                 |
| `systemAvailableMemory` | Integer                              | Available memory on the system in bytes (Android only)                                                                                                                  | No                 |
| `appAvailableMemory`    | Integer                              | Amount of memory in bytes available to the current app (iOS only)                                                                                                       | No                 |
| `batteryLevel`          | Integer                              | Remaining battery level as an integer percentage of total battery capacity                                                                                              | No                 |
| `batteryState`          | String                               | Battery state for the device. One of: "unplugged", "charging", "full".                                                                                                  | No                 |
| `lowPowerMode`          | Boolean                              | A Boolean indicating whether Low Power Mode is enabled (iOS only)                                                                                                       | No                 |
| `availableStorage`      | Integer                              | Bytes of storage remaining                                                                                                                                              | No                 |
| `totalStorage`          | Integer                              | Total size of storage in bytes                                                                                                                                          | No                 |
| `isPortrait`            | Boolean                              | A Boolean indicating whether the device orientation is portrait (either upright or upside down)                                                                         | No                 |
| `appSetId`              | String                               | Android vendor ID scoped to the set of apps published under the same Google Play developer account (see <https://developer.android.com/training/articles/app-set-id>)   | No                 |
| `appSetIdScope`         | String (either "app" or "developer") | Scope of the `appSetId`. Can be scoped to the app or to a developer account on an app store (all apps from the same developer on the same device will have the same ID) | No                 |

---

# Track screen views with the React Native tracker
> Track screen views automatically with React Navigation or React Native Navigation, enable screen context entities, and measure screen engagement.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/screen-tracking/

Screen view tracking captures screen changes within the app.

To track a ScreenViewEvent, use the `trackScreenViewEvent` tracker method [as explained here](/docs/sources/react-native-tracker/tracking-events/#tracking-screen-view-events). For example:

```typescript
tracker.trackScreenViewEvent({
    name: 'my-screen-name',
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e',
    type: 'carousel',
    transitionType: 'basic'
});
```

## Tracking screen views in React Navigation

If you are using the [React Navigation](https://reactnavigation.org/) library for navigation in your app, you can implement automatic screen view tracking by adding a callback to your `NavigationContainer`. The steps are explained [in the documentation for React Navigation](https://reactnavigation.org/docs/screen-tracking/).

## Tracking screen views in React Native Navigation

When using the [React Native Navigation](https://wix.github.io/react-native-navigation/docs/before-you-start/) library for navigation in your app, you can automatically track screen views by registering a listener when your component appears on screen. Use the `Navigation.events().registerComponentDidAppearListener` callback to subscribe the listener and track screen views as [documented here](https://wix.github.io/react-native-navigation/api/events/#componentdidappear).

## Screen context entity

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    screenContext: true, // enabled by default
});
```

If the `screenContext` property is enabled, the tracker attaches a [`Screen` entity](http://iglucentral.com/schemas/com.snowplowanalytics.mobile/screen/jsonschema/1-0-0) to all the events tracked by the tracker reporting the last (and probably current) screen visible on device when the event was tracked.

The `Screen` entity is conditioned by the internal state of the tracker only. To make an example, if the developer manually tracks a `ScreenView` event, all the following events will have a `Screen` entity attached reporting the same information as the last tracked ScreenView event.

## Screen engagement tracking

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    screenEngagementAutotracking: true, // enabled by default
});
```

Screen engagement tracking is a feature that enables tracking the user activity on the screen. This consists of the time spent and the amount of content viewed on the screen.

Concretely, it consists of the following metrics:

1. Time spent on screen while the app was in foreground (tracked automatically).
2. Time spent on screen while the app was in background (tracked automatically).
3. Number of list items scrolled out of all list items (requires some manual tracking).
4. Scroll depth in pixels (requires some manual tracking).

This information is attached using a [`screen_summary` context entity](/docs/events/ootb-data/page-activity-tracking/#screen-summary-entity) to the following events:

1. [`screen_end` event](/docs/events/ootb-data/page-activity-tracking/#screen-end-event) that is automatically tracked before a new screen view event.
2. [`application_background` event](/docs/events/ootb-data/mobile-lifecycle-events/#background-event).
3. [`application_foreground` event](/docs/events/ootb-data/mobile-lifecycle-events/#foreground-event).

Screen engagement tracking is enabled by default, but can be configured using the `TrackerConfiguration.screenEngagementAutotracking` option.

For a demo of how mobile screen engagement tracking works in action, **[please visit this demo](https://snowplow-incubator.github.io/mobile-screen-engagement-demo/)**.

#### Updating list item view and scroll depth information

To update the list item viewed and scroll depth information tracked in the screen summary entity, you can track the `ListItemView` and `ScrollChanged` events with this information. When tracked, the tracker won't send these events individually to the collector, but will process the information into the next `screen_summary` entity and discard the events. You may want to track the events every time a new list item is viewed on the screen, or whenever the scroll position changes.

To update the list items viewed information:

```js
tracker.trackListItemViewEvent({
    index: 1,
    itemsCount: 10,
});
```

To update the scroll depth information:

```js
tracker.trackScrollChangedEvent({
    yOffset: 10,
    xOffset: 20,
    viewHeight: 100,
    viewWidth: 200,
    contentHeight: 300,
    contentWidth: 400,
});
```

---

# Track sessions with the React Native tracker
> Track user sessions with client_session entities, configure foreground and background timeouts, and retrieve session data at runtime.
> Source: https://docs.snowplow.io/docs/sources/react-native-tracker/tracking-events/session-tracking/

Session tracking captures the session which helps to keep track of the user activity in the app.

Client session tracking is enabled by default. It can be set using the `sessionContext` option as explained below.

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    sessionContext: true,
});
```

When enabled, the tracker appends a [`client_session` entity](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) to each event it sends and it maintains this session information as long as the application is installed on the device.

Sessions correspond to tracked user activity. A session expires when no tracking events have occurred for the amount of time defined in a timeout (by default 30 minutes). The session timeout check is executed for each event tracked. If the gap between two consecutive events is longer than the timeout the session is renewed. There are two timeouts since a session can timeout in the foreground (while the app is visible) or in the background (when the app has been suspended, but not closed).

The timeouts for the session can be configured like in the example below:

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    foregroundSessionTimeout: 1800, // seconds
    backgroundSessionTimeout: 1800 // seconds
});
```

The lifecycle events (`Foreground` and `Background` events) have a role in the session expiration. The lifecycle events can be enabled as explained in [App Lifecycle Tracking](/docs/sources/react-native-tracker/tracking-events/lifecycle-tracking/). Once enabled they will be fired automatically when the app moves from foreground state to background state and vice versa.

When the app moves from foreground to background, the `Background` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the foreground timeout. When the app moves from background to foreground, the `Foreground` event is fired. If session tracking is enabled, the session entity will be attached to the event checking the session expiration using the background timeout.

For instance, with this configuration:

```typescript
const tracker = await newTracker({
    namespace: 'appTracker',
    endpoint: COLLECTOR_URL,
    foregroundSessionTimeout: 360, // seconds
    backgroundSessionTimeout: 15 // seconds
});
```

the session would expire if the app is in background for more than 15 seconds, like in this example:

```text
time: 0s - ScreenView event - foreground timeout session check - session 1
time: 3s - Background event - foreground timeout session check (3 < 360) - session 1
time: 30s - Foreground event - background timeout session check (30 > 15) - session 2
```

In the above example, the `Foreground` event triggers a new session because the time spent in background (without tracked events) is bigger than the background timeout for the session.

## Getting session data from the tracker

The React Native tracker has implemented callbacks that enable you to get session data back from the tracker at runtime. The way to do so is by using a tracker's `get..` methods. As with all tracker methods, these callbacks are also asynchronous, and they return a Promise that resolves to the respective value (or `undefined`) when fulfilled.

The available methods, are:

### getSessionUserId

This method returns a promise that resolves to the identifier (string UUIDv4) for the user of the session.

```javascript
const sessionUserId = await tracker.getSessionUserId();
```

### getSessionId

This method returns a promise that resolves to the identifier (string UUIDv4) for the session.

```javascript
const sessionId = await tracker.getSessionId();
```

### getSessionIndex

This method returns a promise to resolve to the index (number) of the current session for this user.

```javascript
const sessionIdx = await tracker.getSessionIndex();
```

### getIsInBackground

This method returns a promise to resolve to whether (boolean) the app is currently in background.

```javascript
const isInBackground = await tracker.getIsInBackground();
```

### getBackgroundIndex

This method returns a promise to resolve to the number of background transitions in the current session.

```javascript
const bgIndex = await tracker.getBackgroundIndex();
```

### getForegroundIndex

This method returns a promise to resolve to the number of foreground transitions in the current session.

```javascript
const fgIndex = await tracker.getForegroundIndex();
```

### getSessionState

This method returns a promise to resolve to the object containing the full state used to build the `client_session` entity in a single call, rather than the individual fields returned by the other methods.

```javascript
const sessionState = await tracker.getSessionState();
```

---

# Add entities and other data to events with the Roku tracker
> Add event context entities, subject data with user and platform information, and custom timestamps to Roku tracker events.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset. Each of them involves a different class from the Roku tracker.

1. Event context using self-describing data: Attach event context, describing anything you like, in the form of self-describing JSONs.
2. Subject: Include information about the user, or the platform on which the event occurred.
3. Timestamp: Override the default event timestamp with your own timestamp.

You can attach any of these as additional arguments to the track event methods.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```brightscript
m.global.snowplow.screenView = {
    id: "screen23",
    context: [
        {
            schema: "iglu:com.my_company/movie_poster/jsonschema/1-0-0",
            data: {
                movie_name: "Solaris",
                poster_country: "JP",
                poster_date: "1978-01-01"
            }
        },
        {
            schema: "iglu:com.my_company/customer/jsonschema/1-0-0",
            data: {
                p_buy: 0.23,
                segment: "young_adult"
            }
        }
    ]
}
```

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event, such as their user ID, what type of device they used, or what size screen that device had.

This information can be entered during tracker initialization by passing a `subject` associative array to the `init` method. All of the information is optional.

Some subject information is filled automatically by the tracker. This includes the platform of the user, timezone, language, resolution, and viewport.

The following table lists all the properties that can be set in tracker initialization. These are all part of the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/).

If not provided, from v0.3.0 onwards the `domainUserId` and `domainSessionId` will be generated by the tracker and stored in the app [Registry](https://developer.roku.com/en-au/docs/developer-program/getting-started/architecture/file-system.md). This can be disabled by setting the `sessionContext` option to `false`. By default, this information is attached via the `client_session` entity, similar to mobile platforms. You can also include these values in the `domain_userid`, `domain_sessionid`, and `domain_sessionidx` fields by enabling the `sessionAsCookies` setting.

Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

| Property             | Description                                  |
| -------------------- | -------------------------------------------- |
| `userId`             | Unique user identifier in your system        |
| `domainUserId`       | Cookie-based unique identifier for user      |
| `domainSessionId`    | Cookie-based unique identifier for session   |
| `networkUserId`      | Cookie-based unique identifier for user      |
| `appId`              | Unique identifier for application            |
| `applicationContext` | Enable an entity with the channel version    |
| `sessionContext`     | Whether to enable or disable session state   |
| `sessionAsCookies`   | Enable session identifiers in atomic fields  |
| `userAnonymous`      | Use a nil ID for the `domainUserId` value    |
| `sessionAnonymous`   | Use a nil ID for the `domainSessionId` value |

Example:

```brightscript
m.global.snowplow.init = {
    subject: {
        userId: "...",
        domainUserId: "...",
        domainSessionId: "...",
        networkUserId: "...",
        appId: "...",
        applicationContext: true,
        sessionContext: false,
        sessionAsCookies: true,
        userAnonymous: true,
        sessionAnonymous: true,
    }
}
```

## Event timestamps

Processed Snowplow events have five different timestamps.

| Timestamp name     | Description                                                                           |
| ------------------ | ------------------------------------------------------------------------------------- |
| `dtm`              | Device timestamp. Added automatically during event creation.                          |
| `ttm`              | True timestamp. This can be manually set as an alternative to `dtm`.                  |
| `stm`              | Sent timestamp. Added automatically on event sending.                                 |
| `collector_tstamp` | Added by the event collector.                                                         |
| `etl_tstamp`       | Added after event enrichment during the processing pipeline.                          |
| `derived_tstamp`   | Either a calculated value (`collector_tstamp` - (`stm` - `dtm`)) or the same as `ttm` |

Overriding the default event timestamp (`dtm`) with `ttm` can be useful in some situations. For example, if the Snowplow event refers to an action that happened previously but is only now being tracked. The type of the timestamp is a string with number of milliseconds since the Unix epoch.

Example:

```brightscript
m.global.snowplow.structured = {
    se_ca: "category",
    se_ac: "action",
    se_va: 10,
    ttm: CreateObject("roDateTime").AsSeconds().ToStr() + "000"
}
```

---

# Configure how events are sent for the Roku tracker
> Configure Roku tracker network settings with collector endpoint, HTTP method, server anonymization, and tracker namespaces for multiple collectors.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/configuration/

When you initialize your tracker (using the `init` method), you will need to provide its network configuration as an associative array under the `network` property. The configuration consists of the collector endpoint and HTTP method type (see below table).

Example:

```brightscript
m.global.snowplow.init = {
    network: {
        collector: "http://...",
        method: "POST"
    }
}
```

Properties of the network configuration:

| Network configuration parameter | Description                                                 |
| ------------------------------- | ----------------------------------------------------------- |
| `collector`                     | URI for the Snowplow collector endpoint.                    |
| `method`                        | HTTP method to use. `GET` and `POST` methods are supported. |
| `serverAnonymous`               | Enable the `SP-Anonymous` header for server anonymization.  |

## Tracker namespaces

You may initialize multiple trackers, each with a different namespace. In this way, you can send events to multiple Snowplow collectors from your application. An example scenario for this would be sending events to both a staging and production pipeline. Each tracker processes events in its own background thread.

To initialize a tracker with a custom namespace, set the `namespace` property in the associative array passed to the `init` method. When no namespace is given, a default namespace is assigned to the tracker. Reinitializing trackers with the same namespace results in updating the configuration for the already initialized trackers. The following example initializes a tracker with the namespace "ns1":

```brightscript
m.global.snowplow.init = {
    namespace: "ns1",
    network: { ... }
}
```

Trackers can be individually addressed using their namespaces when tracking events. To send events to a specific tracker, call its namespace as follows: `m.global.snowplow.trackerNamespace.structured = {...}`. Here is an example of tracking a screen view event using a tracker with the namespace "ns1":

```brightscript
m.global.snowplow.ns1.screenView = {
    id: "screen23",
    name: "HUD > Save Game"
}
```

To track events with all initialized trackers, simply call methods to track events on the Snowplow instance without specifying a tracker namespace. The following example tracks a screen view event using all initialized trackers:

```brightscript
m.global.snowplow.screenView = {
    id: "screen23",
    name: "HUD > Save Game"
}
```

## Logging

The package makes use of [roku-log](https://github.com/georgejecook/roku-log), a logging framework for Roku Scenegraph apps. roku-log enables configuring the logging output and setting log levels for the severity of the output. Under the `debug` log level, the tracker outputs information about each request. On the other hand, setting the log level to `error` will only output crashes and tracking errors. Please refer to the roku-log [instructions](https://github.com/georgejecook/roku-log) to learn about setting it up.

---

# Roku device info context
> Automatically attach Roku device information including model, OS version, RIDA, network status, and display properties to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/device-context/

Device Info Context is an entity attached to events that provides information about the Roku device. It gives information about the device model and OS, adds unique identifiers for the channel and device, includes playback settings, current device usage, locale, display properties, network status, and device features.

The context is automatically added to all tracked events by default. If you prefer not to add the context to events, you may disable it using the `subject.deviceContext` option during tracker initialization:

```brightscript
m.global.snowplow.init = {
   subject: {
      deviceContext: false ' disabling device info context
   },
   ...
}
```

The context entity reflects the [ifDeviceInfo interface](https://developer.roku.com/en-gb/docs/references/brightscript/interfaces/ifdeviceinfo.md) in Roku SDK and picks some useful properties from it. It adds the following information:

| Attribute               | Type      | Description                                                                                 | Required? |
| ----------------------- | --------- | ------------------------------------------------------------------------------------------- | --------- |
| `model`                 | String    | Model name of the Roku device (e.g., 3940EU)                                                | yes       |
| `modelDisplayName`      | String    | Model display name (e.g., Roku Express 4K)                                                  | yes       |
| `modelType`             | String    | Type of device (STB or TV)                                                                  | yes       |
| `osVersion`             | String    | Roku OS version (e.g., 10.5.1.4059) (Roku OS ≥ 9.2)                                         | no        |
| `channelClientId`       | String    | A unique device identifier that is different across channels (Roku OS ≥ 8.1)                | no        |
| `isRIDADisabled`        | Boolean   | Indicates whether tracking via Roku's ID for Advertisers (RIDA) is disabled (Roku OS ≥ 8.1) | no        |
| `RIDA`                  | String    | A persistent unique identifier (UUID) for the device (Roku OS ≥ 8.1)                        | no        |
| `captionsMode`          | String    | Global captions are turned on or off, or are in instant replay mode                         | yes       |
| `audioOutputChannel`    | String    | Type of audio output                                                                        | yes       |
| `memoryLevel`           | String    | General memory level of the channel (normal, low, critical) (Roku OS ≥ 8.1)                 | no        |
| `timeSinceLastKeypress` | Integer   | Number of seconds since the last remote keypress was received                               | yes       |
| `userCountryCode`       | String    | ISO 3166-1 country code of the user's Roku account (Roku OS ≥ 8.1)                          | no        |
| `countryCode`           | String    | Roku Channel Store associated with a user's Roku account                                    | yes       |
| `videoMode`             | String    | Video playback resolution                                                                   | yes       |
| `displayWidth`          | Integer   | Physical width of the attached display in centimeters                                       | yes       |
| `displayHeight`         | Integer   | Physical height of the attached display in centimeters                                      | yes       |
| `displayProperties`     | String\[] | List of keys for display properties of the screen                                           | yes       |
| `connectionType`        | String    | Type of internet connection the device is using (e.g., WiFiConnection)                      | yes       |
| `internetStatus`        | Boolean   | Internet connection status of the device (Roku OS ≥ 10.0) .                                 | no        |
| `features`              | String\[] | List of features that the current device/firmware supports                                  | yes       |

The `displayProperties` list may include keys for the following display properties: `Internal` (display part of Roku player), `Hdr10` (display supports HDR10), `Hdr10Plus` (display supports HDR10), `DolbyVision` (display supports Dolby Vision). The `features` list of activated features of the device may include `5.1_surround_sound`, `can_output_5.1_surround_sound`, `sd_only_hardware`, `usb_hardware`, `sdcard_hardware`, `ethernet_hardware`, `gaming_hardware`, `energy_star_compliant`, `soundbar_hardware`, and `handsfree_voice`.

## Roku's ID for Advertisers (RIDA)

`RIDA` is a Universally Unique Identifier (UUID) that is unique and persistent for the Roku device. It is designed to generally follow the guidelines established for the IDFA (Identifier for Advertising) that are available on other platforms such as iOS and Android. The ID is intended for Roku publishers to enable frequency capping and targeted advertising on the Roku platform.

Users can disable RIDA tracking by selecting "Limit ad tracking" in the device settings. In this case, the `IsRIDADisabled` property will be true and `RIDA` will be a temporary ID that expires after 30 days.

RIDA is tracked by the tracker by default. However, there may be cases where you may not want to track the identifier. To disable tracking RIDA, deactivate the `subject.RIDATracking` option during tracker initialization:

```brightscript
m.global.snowplow.init = {
   subject: {
      RIDATracking: false ' disabling tracking RIDA
   },
   ...
}
```

---

# Example Roku app with Snowplow tracking
> Demo Roku video player channel showing Snowplow tracking implementation in BrightScript with setup and deployment instructions.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/example-app/

The tracker comes with a demo app that shows it in use. It is a simple video player with a fixed collection of played videos and an interface to switch between them. The demo app may serve you to understand how to embed Snowplow tracking within a BrightScript channel.

The project in included in the [tracker repository](https://github.com/snowplow-incubator/snowplow-roku-tracker), in the `src-demo-app` subfolder. The following steps to build and deploy the channel assume that you have enabled developer mode on your Roku device and it is connected to your network.

1. Download the [Roku tracker project](https://github.com/snowplow-incubator/snowplow-roku-tracker).
2. Run `npm install` within the project.
3. Install ropm `npm i ropm -g`.
4. Install ropm packages `ropm install`.
5. Create `.env` file with environment variables in the root of this repository with the following content:\
   `ROKU_IP=192.168.100.129`\
   `ROKU_PASSWORD=XXXX`
6. Add configuration for Snowplow collector to `src-demo-app/manifest`:\
   `snowplow_collector=http://192.168.100.127:9090`\
   `snowplow_method=POST`
7. Start the demo app using `npm run demo-app`.

Alternatively, you may run the demo app from Visual Studio Code as the debug configuration is already prepared. Install the BrightScript extension to Visual Studio Code and choose "Run demo app" in the debug options.

Events will be sent to the Snowplow collector as you navigate through the app.

---

# Roku tracker SDK
> Track events in BrightScript and BrighterScript Roku applications with SceneGraph channel support using the Snowplow Roku tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/

The Snowplow Roku Tracker allows you to track Snowplow events in your BrightScript and BrighterScript applications. The tracker is compatible with SceneGraph channels. The current version is 0.3.1.

The tracker is published on NPM as [@snowplow/roku-tracker](https://www.npmjs.com/package/@snowplow/roku-tracker) and can be installed with [ropm](https://github.com/rokucommunity/ropm). It may also be included in BrightScript apps directly by copying source files from releases [published on Github](https://github.com/snowplow-incubator/snowplow-roku-tracker).

We've created a simple [demo Roku channel](/docs/sources/roku-tracker/example-app/) to show you the tracker in use.

---

# Media tracking with the Roku tracker
> Enable automatic audio and video playback tracking for Roku Audio and Video nodes with support for media tracking v1 and v2 schemas.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/

Media tracking enables collecting events about audio/video playback from the [Audio](https://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/audio.md) and [Video](https://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/video.md) nodes provided in the Roku SceneGraph SDK for media playback. Once enabled for a node, the tracker subscribes to selected events and tracks them automatically. This makes adding media tracking into your Roku channels really simple.

There are [multiple versions](/docs/events/ootb-data/media-events/) of media tracking available, and the Roku tracker supports [v1](/docs/sources/roku-tracker/media-tracking/v1/) and (from v0.3.0) [v2](/docs/sources/roku-tracker/media-tracking/v2/).

---

# Roku media tracking (v1)
> Track Roku audio and video playback events with media tracking v1 schemas including player state changes and position tracking.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/v1/

Media tracking has [multiple versions](/docs/sources/web-trackers/tracking-events/media/) of schema available. This document is for version 1 of the schemas, it is recommended to [migrate to v2 tracking](/docs/sources/roku-tracker/media-tracking/v2/#migrating-from-media-v1-apis).

## Usage

To start media tracking for an Audio/Video node, assign a `roAssociativeArray` with the node to the `enableVideoTracking` property:

```brightscript
m.global.snowplow.enableVideoTracking = {
    video: m.Video
}
```

From tracker version 0.3.0:

- You can also use `enableAudioTracking` and pass the node via `audio` instead of `video`
- The v2 `enableMediaTracking` call and `media` option will also work, the tracker will use v1 tracking if the optional `version` option is defined as `1`

In addition to the node and version selector, the `enableVideoTracking` property accepts several optional attributes listed in the tables below.

| Attribute                   | Type                  | Description                                                                                                       | Required?                                        |
| --------------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `media` / `audio` / `video` | Audio / Video         | Audio/Video node to be tracked                                                                                    | yes                                              |
| `version`                   | Integer               | Tracking schema version to use; defaults to 1 if not provided for `enableVideoTracking` and `enableAudioTracking` | no                                               |
| `label` / `options.label`   | String                | An identifiable custom label sent with the event                                                                  | no                                               |
| `options.captureEvents`     | String\[]             | Types of events to capture                                                                                        | no, defaults to all events except for `position` |
| `options.boundaries`        | Integer\[]            | Percentage boundaries in playback for which events will be sent                                                   | no, defaults to `[10, 25, 50, 75]`               |
| `options.positionInterval`  | Integer               | Interval in seconds in which `position` events should be reported                                                 | no, defaults to 5                                |
| `context` / `entities`      | SelfDescribingJSON\[] | Array of custom entities to include with all generated media events                                               | no                                               |

To stop tracking events from the node, set the `disableVideoTracking`/`disableAudioTracking` property. The tracker will then stop observing and tracking events from the node. The property should be set with an `roAssociativeArray` with exactly one attribute, `video` (or `audio`/`media`), like so:

```brightscript
m.global.snowplow.disableVideoTracking = {
    video: m.video
}
```

## Media Player Events

### Media Events

All playback events are tracked automatically and conform to the `media_player_event` schema. The schema has two properties:

| Property | Description               | Required? |
| -------- | ------------------------- | --------- |
| `type`   | Type of event             | Yes       |
| `label`  | Identifiable custom label | No        |

The events are enriched with two context entities. The first is the `media_player` context entity that provides the following properties:

| Property          | Description                                                                             | Required? |
| ----------------- | --------------------------------------------------------------------------------------- | --------- |
| `currentTime`     | The current playback time                                                               | Yes       |
| `duration`        | A double-precision floating-point value indicating the duration of the media in seconds | No        |
| `ended`           | If playback of the media has ended                                                      | Yes       |
| `isLive`          | If the media is live                                                                    | Yes       |
| `loop`            | If the video should restart after ending                                                | Yes       |
| `muted`           | If the media element is muted                                                           | Yes       |
| `paused`          | If the media element is paused                                                          | Yes       |
| `percentProgress` | The percent of the way through the media                                                | No        |
| `playbackRate`    | Playback rate (1 is normal)                                                             | No        |
| `volume` .        | Volume percent                                                                          | Yes       |

The second is the Roku `video` context with these properties:

| Property               | Description                                                                                                             | Required? |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------- |
| `videoId`              | ID generated when video tracking of the video node was initialized.                                                     | Yes       |
| `contentId`            | ID of video provided in content metadata.                                                                               | No        |
| `contentTitle`         | Title of video provided in content metadata.                                                                            | No        |
| `contentUrl`           | URL of video provided in content metadata.                                                                              | No        |
| `contentType`          | Category of video (e.g., movie, season, series) provided in content metadata.                                           | No        |
| `streamFormat`         | Container format of video (e.g., mp4, wma, mkv) provided in content metadata.                                           | No        |
| `streamUrl`            | URL of the current stream.                                                                                              | No        |
| `measuredBitrate`      | Measured bitrate (bps) of the network when the stream was selected.                                                     | No        |
| `streamBitrate`        | Current bitrate of the stream.                                                                                          | No        |
| `isUnderrun`           | Indicates whether the stream was downloaded due to an underrun.                                                         | No        |
| `isResumed`            | Indicates whether the playback was resumed after trickplay.                                                             | No        |
| `videoFormat`          | Video codec of the currently playing video stream (e.g., hevc, mpeg2, mpeg4\_15).                                       | No        |
| `timeToStartStreaming` | Time in seconds from playback being started until the video actually began playing.                                     | No        |
| `width`                | Width of the video play window in pixels. 0 if the play window is set to the width of the entire display screen.        | Yes       |
| `height`               | Height of the video play window in pixels. 0 if the play window is set to the height of the entire display screen.      | Yes       |
| `errorStr`             | A diagnostic message indicating a video play error. Refer to the Roku Video documentation for the format of the string. | No        |

## Captured Types of Events

The tracker automatically captures several types of events. You may configure which types of events to track by passing a list of the types into the `options.captureEvents` property during tracker initialization:

```brightscript
m.global.snowplow.enableVideoTracking = {
    video: m.Video,
    options: {
        captureEvents: ["playing", "paused"]
    }
}
```

Overall, the types of events can be grouped into two categories: playback position events, and state change events.

### Playback Position Events

These "ping" events report changes in the playback position of the video while playing. There are two types of the events:

1. `percentprogress` events report when the playback reaches certain boundaries defined as percentages of the total playback duration.
2. `position` events are regular events sent when the playback changes by a certain number of seconds.

#### Event Type: `percentprogress`

Percent progress events are sent when predefined percentage boundaries in playback are reached. The boundaries may be configured using a `boundaries` list when initializing video tracking:

```brightscript
m.global.snowplow.enableVideoTracking = {
    video: m.Video,
    options: {
        captureEvents: ["percentprogress"],
        boundaries: [25, 50, 75, 100]
    }
}
```

The tracker sends percent progress events by default for the following percentage boundaries: 10%, 25%, 50%, and 75% of total playback.

These events can not be reported for live streams since they assume that the total duration of the video is known.

#### Event Type: `position`

Position events report the current playback position in regular intervals. These events are not reported by default. They can be activated using the `options.captureEvents` property.

The events are sent whenever the playback head changes by more than the predefined position interval. The interval defaults to 5 seconds. It can be configured using the `options.positionInterval` property:

```brightscript
m.global.snowplow.enableVideoTracking = {
    video: m.Video,
    options: {
        captureEvents: ["position"],
        positionInterval: 15 ' seconds
    }
}
```

These events do not require the total duration of the video to be known and can be reported for live streams as well.

### State Change Events

State change events are sent when the state of the video node changes due to interaction from the user, changes in the playback, or through any other influence. Transitions between the following states are captured:

- `playing`
- `paused`
- `buffering`
- `stopped`
- `finished`
- `error`
  - Error occured in the playback. The `errorStr` property in the `video` context will provide more details on the error.

The event type of the events (`type` property) reflects the current state.

All of the state changes are reported by default. In order to capture only selected states, use the `options.captureEvents` property:

```brightscript
m.global.snowplow.enableVideoTracking = {
    video: m.Video,
    options: {
        captureEvents: ["playing", "paused"]
    }
}
```

---

# Roku media tracking (v2)
> Track Roku media playback with v2 schemas including session entities, ping events, and percentage progress tracking for audio and video nodes.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/media-tracking/v2/

> **Note:** Media tracking has [multiple versions](/docs/sources/web-trackers/tracking-events/media/) of schemas available. This document is for version 2 of the schemas; tracker versions earlier than v0.3.0 [only support v1 tracking](/docs/sources/roku-tracker/media-tracking/v1/).

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

```brightscript
' 1. Start tracking when player loads (events are auto-tracked)
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  label: "My Video Title"
}

' 2-3. Player state and events are automatically tracked from the Video node

' 4. End tracking when done
m.global.snowplow.disableMediaTracking = {
  media: m.Video
}
```

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

The Roku tracker automatically reads most player properties from the Audio/Video node. Configure additional options in `enableMediaTracking`:

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,           ' Audio/Video node to track (required)
  version: 2,               ' Schema version (defaults to 2)
  id: "session-123",        ' Custom session ID (auto-generated if not set)
  label: "My Video",        ' Human-readable title for the media_player entity
  rokuVideoContext: true    ' Whether to include the roku_video_context entity
}
```

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  pings: true,        ' Enabled by default
  pingInterval: 30    ' Seconds between pings (default: 30)
}

' Or turn off pings entirely
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  pings: false
}
```

### Percentage progress

Track events when playback reaches specified percentage boundaries.

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  boundaries: [25, 50, 75]  ' By default, the tracker will fire events at these percentages
}
```

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  sessions: true
}
```

### Custom entities

You can attach custom entities to all events for this media tracking instance.

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  entities: [
      {
          schema: "iglu:com.example/video_metadata/jsonschema/1-0-0",
          data: {"contentId": "abc123", "category": "tutorial"}
      }
  ]
}
```

### Filter events

Only track specific event types by providing an allowlist:

```brightscript
m.global.snowplow.enableMediaTracking = {
  media: m.Video,
  version: 2,
  captureEvents: ["play", "pause", "end"]
}
```

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

The Roku tracker automatically tracks the following events from Audio/Video nodes:

- Ready, play, pause, end events
- Seek start, seek end events
- Ping events
- Percent progress events (if configured)
- Buffer start, buffer end events
- Quality change events
- Error events

To track events manually or track events not auto-tracked, use `trackMediaEvent`:

```brightscript
m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_click_event/jsonschema/1-0-0",
  data: { "percentProgress": 50 }
}
```

### Player state events

```brightscript
m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.snowplowanalytics.snowplow.media/playback_rate_change_event/jsonschema/1-0-0",
  data: { "newRate": 1.5 }
}

m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.snowplowanalytics.snowplow.media/volume_change_event/jsonschema/1-0-0",
  data: { "newVolume": 80 }
}
```

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

```brightscript
' Ad break start
m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_break_start_event/jsonschema/1-0-0",
  data: {},
  adBreak: {
      "breakId": "break-1",
      "name": "pre-roll",
      "breakType": "linear",
      "podSize": 3
  }
}

' Ad start
m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.snowplowanalytics.snowplow.media/ad_start_event/jsonschema/1-0-0",
  data: {},
  ad: {
      "adId": "ad-1",
      "name": "Product Ad",
      "duration": 30
  }
}
```

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

```brightscript
m.global.snowplow.trackMediaEvent = {
  media: m.Video,
  schema: "iglu:com.example/video_interaction/jsonschema/1-0-0",
  data: { "action": "share", "platform": "twitter" }
}
```

## Update player state

Manual player state updates aren't needed for the Roku tracker. The tracker automatically tracks player state from the registered media node.

## Roku video entity

By default, the Roku tracker attaches a `video` entity to all media events.

### `video`

**Type:** Entity

Schema for a Roku video event (reflects the Video node: https\://developer.roku.com/en-gb/docs/references/scenegraph/media-playback-nodes/video.md)

**Schema:** `iglu:com.roku/video/jsonschema/1-0-0`

**Example:**

```json
{
  "videoId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "contentId": "movie-12345",
  "contentTitle": "Sample Movie",
  "contentType": "movie",
  "streamFormat": "mp4",
  "measuredBitrate": 5000000,
  "streamBitrate": 4500000,
  "videoFormat": "hevc",
  "timeToStartStreaming": 1.5,
  "width": 1920,
  "height": 1080
}
```

**Properties:**

| Property                        | Description                                                                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `videoId` _string_              | _Required._ ID generated when video tracking of the video node was initialized.                                                     |
| `contentId` _string_            | _Optional._ ID of video provided in content metadata.                                                                               |
| `contentTitle` _string_         | _Optional._ Title of video provided in content metadata.                                                                            |
| `contentUrl` _string_           | _Optional._ URL of video provided in content metadata.                                                                              |
| `contentType` _string_          | _Optional._ Category of video (e.g., movie, season, series) provided in content metadata.                                           |
| `streamFormat` _string_         | _Optional._ Container format of video (e.g., mp4, wma, mkv) provided in content metadata.                                           |
| `streamUrl` _string_            | _Optional._ URL of the current stream.                                                                                              |
| `measuredBitrate` _integer_     | _Optional._ Measured bitrate (bps) of the network when the stream was selected.                                                     |
| `streamBitrate` _integer_       | _Optional._ Current bitrate of the stream.                                                                                          |
| `isUnderrun` _boolean_          | _Optional._ Indicates whether the stream was downloaded due to an underrun.                                                         |
| `isResumed` _boolean_           | _Optional._ Indicates whether the playback was resumed after trickplay.                                                             |
| `videoFormat` _string_          | _Optional._ Video codec of the currently playing video stream (e.g., hevc, mpeg2, mpeg4\_15).                                       |
| `timeToStartStreaming` _number_ | _Optional._ Time in seconds from playback being started until the video actually began playing.                                     |
| `width` _integer_               | _Required._ Width of the video play window in pixels. 0 if the play window is set to the width of the entire display screen.        |
| `height` _integer_              | _Required._ Height of the video play window in pixels. 0 if the play window is set to the height of the entire display screen.      |
| `errorStr` _string_             | _Optional._ A diagnostic message indicating a video play error. Refer to the Roku Video documentation for the format of the string. |

## Migrating from media v1 APIs

If you are migrating from the [v1 media tracking](/docs/sources/roku-tracker/media-tracking/v1/), v2 has compatibility options to make the change easier.

For backwards compatibility with the earlier tracking API, you can continue using `enableVideoTracking` or `enableAudioTracking` instead of `enableMediaTracking`, and pass the node via the `audio`/`video` option instead of `media`. For most other options, v2 will accept either v1 or v2 `roAssociativeArray` formats. The tracking will work with any combination of these method and option names regardless of node type, as long as `version: 2` is specified. You can also specifically opt into version 1 by passing `version: 1` for any of these methods, though v2 options may not be recognized.

If using specific `version` values, the same version should also be specified for the corresponding `disable*Tracking` method. Similarly, mixing `enableVideoTracking` and `disableMediaTracking` will not work unless versions are specified consistently.

The additional options differ by version; but version 2 will accept the options from version 1 if it can't find the newer equivalent settings.

| v2 Attribute                | v1 Fallback                 | Type                  | Description                                                                                   |
| --------------------------- | --------------------------- | --------------------- | --------------------------------------------------------------------------------------------- |
| `media` / `audio` / `video` | `media` / `audio` / `video` | Audio / Video         | Audio/Video node to be tracked                                                                |
| `version`                   | `version`                   | Integer               | Tracking schema version to use; should be set to 2 if not using `enableMediaTracking`         |
| `id`                        | N/A                         | UUID                  | A unique ID to use for the media `session` entity to group all tracked events                 |
| `label`                     | `options.label`             | String                | An identifiable custom label sent with the event in the `media_player` entity                 |
| `sessions`                  | Default enabled             | boolean               | Whether to attach the media `session` entity on tracked events                                |
| `pings`                     | Default enabled             | boolean               | Whether to periodically generate media `ping_event`s while content is playing                 |
| `pingInterval`              | `options.positionInterval`  | Integer               | Interval in seconds in which `ping_event`s should be reported                                 |
| `boundaries`                | `options.boundaries`        | Integer\[]            | Percentage boundaries in playback for which `percentage_progress_event`s will be sent         |
| `captureEvents`             | `options.captureEvents`     | String\[]             | Types of events to capture. If specified, the event names should not have the `_event` suffix |
| `context` / `entities`      | `context`                   | SelfDescribingJSON\[] | Array of custom entities to include with all generated events                                 |

---

# Getting started with the Roku tracker
> Install the Roku tracker using ropm or manual installation, initialize tracking, and track structured events in your BrightScript app.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/quick-start-guide/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

The following steps will guide you through setting up the Roku tracker in your project and tracking a simple event.

## Installation

There are two options to install the Roku tracker package to your project:

1. using Roku package manager (ropm),
2. by manually copying package files.

### Using Roku Package Manager (ropm)

[ropm](https://github.com/rokucommunity/ropm) is a package manager for the Roku platform. If you are using ropm in your project, you may install the Roku tracker using the following command:

```bash
ropm install snowplow@npm:@snowplow/roku-tracker
```

### Manual Installation

The Roku tracker may be installed by simply copying source files to your Roku project. You may download and unpack the `dist.zip` or `dist.tar.gz` package from the latest release build on Github. Copy the following folders and files to your Roku project:

1. Contents of `dist/source` into your `source` directory
2. Contents of `dist/components` into your `components` directory

## Implementation

It is recommended that you instantiate Snowplow and add it to the global scope. In this way, it will be accessible from anywhere within your SceneGraph application. You may create the instance in the `init` function of your main scene.

If you installed the package using ropm, mount the component as follows:

```brightscript
m.global.AddField("snowplow", "node", false)
m.global.snowplow = CreateObject("roSGNode", "snowplow_Snowplow")
```

If you installed the package manually, mount the component as follows:

```brightscript
m.global.AddField("snowplow", "node", false)
m.global.snowplow = CreateObject("roSGNode", "Snowplow")
```

## Initialization

Trackers are initialized by setting the `init` property with configuration of the tracker. This configuration takes the form of a `roAssociativeArray`. At its most basic, the configuration takes a Snowplow collector endpoint like so:

```brightscript
m.global.snowplow.init = {
    network: {
        collector: "http://..."
    }
}
```

To learn more about configuring how events are sent, check out [this page](/docs/sources/roku-tracker/configuration/).

## Tracking events

To track an event, simply assign its properties as a `roAssociativeArray` to a field corresponding to the event type. For instance, to track a structured event, assign the `structured` property:

```brightscript
m.global.snowplow.structured = {
    se_ca: "category",
    se_ac: "action",
    se_la: "label",
    se_pr: "property",
    se_va: 10
}
```

Visit documentation about [tracking events](/docs/sources/roku-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/roku-tracker/adding-data/).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

Check out the [example Roku channel](/docs/sources/roku-tracker/example-app/) to see the tracker used with Snowplow Micro.

---

# Tracking events with the Roku tracker
> Track self-describing events, structured events, and screen views with the Roku tracker SDK including automatic install event tracking.
> Source: https://docs.snowplow.io/docs/sources/roku-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in methods to help you track different kinds of events. The methods range from single purpose methods, such as `screenView`, to the more complex but flexible `selfDescribing`, which can be used to track any kind of user behavior. We strongly recommend using `selfDescribing` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

Tracking methods supported by the Roku Tracker:

| Method           | Event type tracked                                  |
| ---------------- | --------------------------------------------------- |
| `selfDescribing` | Custom event based on "self-describing" JSON schema |
| `structured`     | Semi-custom structured event                        |
| `screenView`     | View of screen                                      |

All the methods share common features and parameters. Every type of event can have an optional context added. A Timestamp can also be provided for all event types to override the default event timestamp. See the next page to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event tracking methods, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `selfDescribing`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `selfDescribing`

Use `selfDescribing` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

The `selfDescribing` method takes a `roAssociativeArray`. This array takes a schema name and a flat hash of event data.

Example (assumes that you mounted the Snowplow instance in `m.global.snowplow`):

```brightscript
m.global.snowplow.selfDescribing = {
    data: {
        saveId: "4321",
        level: 23,
        difficultyLevel: "HARD",
        dlContent: true
    },
    schema: "iglu:com.example_company/save_game/jsonschema/1-0-2"
}
```

## Track structured events with `structured`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category" (`se_ca`), "action" (`se_ac`), "label" (`se_la`), and "value" (`se_va`) properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument | Description                                                    | Required in event? |
| -------- | -------------------------------------------------------------- | ------------------ |
| `se_ca`  | The grouping of structured events which this action belongs to | Yes                |
| `se_ac`  | Defines the type of user interaction which this event involves | Yes                |
| `se_la`  | Often used to refer to the 'object' the action is performed on | No                 |
| `se_pr`  | Describing the 'object', or the action performed on it         | No                 |
| `se_va`  | Provides numerical data about the event                        | No                 |

Example:

```brightscript
m.global.snowplow.structured = {
    se_ca: "shop",
    se_ac: "add-to-basket",
    se_pr: "pcs",
    se_va: 2
}
```

## Track screen views with `screenView`

Use `screenView` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`; while all are optional, you must provided at least one of either `name` or `id` to create a valid event. "Name" is the human-readable screen name, and "ID" should be the unique screen ID.

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

Example:

```brightscript
m.global.snowplow.screenView = {
    id: "screen23",
    name: "HUD > Save Game"
}
```

## Automatic application install events

When [configuring the tracker](/docs/sources/roku-tracker/configuration/), use the `trackInstall` setting to opt into automatic [`application_install` tracking](/docs/events/ootb-data/mobile-lifecycle-events/#install-events).

There's no way to directly detect if the app is running as part of a fresh installation, so the tracker uses an indirect trigger. It will send an install event after it creates a first `userId` value for the [`client_session` entity](/docs/sources/roku-tracker/adding-data/#adding-user-and-platform-data-with-subject). Generating a `userId` occurs even if the `client_session` entity is not enabled. This is similar to the approach [recommended by Roku](https://community.roku.com/discussions/developer/tracking-application-install-and-version-state/188117).

```brightscript
m.global.snowplow.init = {
    trackInstall: true
}
```

> **Note:** This will track an "install" the first time the tracker runs and generates a user ID. If you are adding tracking to an existing channel or upgrading from an older SDK version, enabling this setting may create a burst of events as older installations retroactively report this event for the first time. To reduce this:
>
> 1. First release the app with Roku tracker v0.3+ and this setting **disabled**
> 2. Allow user devices time to generate their first `userId` values
> 3. Release an additional version with this setting **enabled**
> 4. Only new installations should track the event
>
> To avoid multiple releases, you could also only enable the setting conditionally based on a timestamp, so only installations after a certain date trigger the event.

---

# Add entities and other data to events with the Ruby tracker
> Add event context entities, subject data with user and platform information, page data, and custom timestamps to Ruby tracker events.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/adding-data-events/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset. Each of them involves a different class from the Ruby tracker.

1. [Event context](#event-context) using SelfDescribingJson. Attach event context, describing anything you like, in the form of self-describing JSONs.
2. [Subject](#adding-user-and-platform-data-with-subject): Include information about the user, or the platform on which the event occurred.
3. [Page](#adding-page-data-with-page): For websites, include data about the page on which the event occurred.
4. [Timestamp](#event-timestamps): Override the default event timestamp with your own timestamp.

You can attach any of these as additional arguments to the `track_x_event` methods.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same [self-describing JSON schemas](/docs/fundamentals/schemas/) as the [self-describing events](/docs/sources/ruby-tracker/tracking-events/). Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event's entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`contexts`) in your data storage. There is no limit to how many entities can be attached to one event.

The context entities were originally called "context" themselves, with the event context referred to as "contexts". This was confusing (and not great English grammar), so we changed the name. However, the name "contexts" persists in some places.

Check out our [demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) to see an example of a custom eCommerce event created using a self-describing event plus product entity. Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information - reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

**v0.7.0+:**

```ruby
# Tracking a screen view with context made up of two entities
entity1 = SnowplowTracker::SelfDescribingJson.new(
  'iglu:com.my_company/movie_poster/jsonschema/1-0-0',
  { 'movie_name' => 'Solaris',
    'poster_country' => 'JP',
    'poster_year$dt' => Date.new(1978, 1, 1) }
)
entity2 = SnowplowTracker::SelfDescribingJson.new(
  'iglu:com.my_company/customer/jsonschema/1-0-0',
  { 'p_buy' => 0.23,
    'segment' => 'young adult' }
)

tracker.track_screen_view(id: 'sci-123-abc', context: [entity1, entity2])
```

**Before v0.7.0:**

```ruby
# Tracking a screen view with context made up of two entities
entity1 = SnowplowTracker::SelfDescribingJson.new(
  'iglu:com.my_company/movie_poster/jsonschema/1-0-0',
  { 'movie_name' => 'Solaris',
    'poster_country' => 'JP',
    'poster_year$dt' => Date.new(1978, 1, 1) }
)
entity2 = SnowplowTracker::SelfDescribingJson.new(
  'iglu:com.my_company/customer/jsonschema/1-0-0',
  { 'p_buy' => 0.23,
    'segment' => 'young adult' }
)

tracker.track_screen_view(nil, 'sci-123-abc', [entity1, entity2])
```

***

> **Note:** The event context is always an array, even if you are only adding one entity.

## Adding user and platform data with Subject

Subject objects store information about the user associated with the event, such as their `user_id`, what type of device they used, or what size screen that device had. Also, they store which platform the event occurred on - e.g. server-side app, mobile, games console, etc.

Each Tracker is initialized with a Subject. This means that every event by default has the platform `srv`: server-side app. Platform is the only preset Subject parameter, which can be overriden using the `set_platform` method. All other parameters must be set manually. This can be done directly on the Subject, or, if it is associated with a Tracker, via the Tracker, which has access to all the methods of its Subject.

Your server-side code may not have access to all these parameters, or they might not be useful to you. All the Subject parameters are optional, except `platform`.

You could create and define a new Subject for every user or every event you want to track. Attaching that Subject to the `track_x_event` method call would add those properties to your event. Alternatively, you could swap the tracker-associated Subject for a more appropriate one before tracking the event, using `set_subject`.

The following table lists all the properties that can be set via Subject. These are all part of the [Snowplow Tracker Protocol](/docs/events/). Check out the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Subject.html) for the details of how to set these parameters.

| Property                                   | **Description**                                                   |
| ------------------------------------------ | ----------------------------------------------------------------- |
| `platform`                                 | The platform the app runs on                                      |
| `user_id`                                  | Unique identifier for user                                        |
| `domain_userid`                            | Cookie-based unique identifier for user                           |
| `network_userid`                           | Cookie-based unique identifier for user                           |
| `domain_sessionid`                         | Cookie-based unique identifier for a visit/session of a `user_id` |
| `domain_sessionidx`                        | Cookie-based count of separate visits/sessions from a `user_id`   |
| `user_fingerprint`                         | User identifier based on (hopefully unique) browser features      |
| `user_ipaddress`                           | User's IP address                                                 |
| `useragent`                                | User agent or browser string                                      |
| `br_lang`                                  | The device/browser language                                       |
| `os_timezone`                              | The device OS's timezone                                          |
| `dvce_screenheight` and `dvce_screenwidth` | The device screen resolution                                      |
| `br_viewwidth` and `br_viewheight`         | The browser viewport size                                         |
| `br_colordepth`                            | The browser color depth                                           |

> **Note:** The methods for defining `domain_sessionid` and `domain_sessionidx` were added in tracker version 0.7.0.

Example:

**v0.7.0+:**

> **Note:** The ability to add `Subject` objects to `track_x_event` method calls (option 1 below) was added in version 0.7.0.

```ruby
# Creating a Subject and adding user data
subject = SnowplowTracker::Subject.new
subject.set_user_id('12345').set_timezone('Europe/London')

# Different ways to add Subject properties to an event
# 1. Providing the Subject as an argument
tracker.track_page_view(page_url: 'www.example.com', subject: subject)

# 2. Swapping the tracker-associated Subject for the new one
# This will affect all subsequent events tracked
tracker.set_subject(subject).track_page_view(page_url: 'www.example.com')

# 3. Setting the user data directly on the tracker-associated Subject
# This will affect all subsequent events tracked
tracker.set_user_id('12345').set_timezone('Europe/London')
tracker.track_page_view(page_url: 'www.example.com')
```

**Before v0.7.0:**

```ruby
# Creating a Subject and adding user data
subject = SnowplowTracker::Subject.new
subject.set_user_id('12345').set_timezone('Europe/London')

# Different ways to add Subject properties to an event
# Either way will affect all subsequent events tracked
# 1. Swapping the tracker-associated Subject for the new one
tracker.set_subject(subject).track_page_view('www.example.com')

# 2. Setting the user data directly on the tracker-associated Subject
tracker.set_user_id('12345').set_timezone('Europe/London')
tracker.track_page_view('www.example.com')
```

***

### Cookie-based user properties

Several of the user properties listed in the above table are relevant only to web apps, and are intended to derive from cookies. The Snowplow cookies are explained on [this page](/docs/sources/web-trackers/cookies-and-local-storage/). The client-side Snowplow JavaScript tracker sets first-party cookies, and the event collector sets a third-party cookie. As a server-side tracker, the Ruby tracker doesn't automatically have access to cookies. However, these properties can be extremely useful, e.g. in stitching events together from individual users, or for matching server-side events with client-side events.

Read more about sharing data between client-side and server-side trackers in this [blog post](https://snowplowanalytics.com/blog/2021/11/09/the-unrivaled-power-of-joining-client-and-server-side-tracking/).

If you have implemented the Ruby tracker as well as the JavaScript tracker in your web app, you could extract the values from the cookies, and set them in your Ruby events using these Subject methods. An example of this is included for `domain_user_id` in our [demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples).

Extracting the `domain_user_id` from the `_sp_id` cookie in Rails:

```ruby
def snowplow_domain_userid
  sp_cookie = cookies.find { |key, _value| key =~ /^_sp_id/ }
  sp_cookie.last.split(".").first if sp_cookie.present?
end
```

Similarly, the `domain_session_id` and `domain_session_idx` values are saved in the first-party `_sp_ses` and `_sp_id` cookies. If you configured the JavaScript tracker with a different “cookieName” option, then these cookies will be named differently.

The `network_user_id` derives from the event collector's third-party cookie, hence the name "network" as it is set at a network level. It is the server-side user identifier. The cookie is named `sp` (Snowplow Micro pipelines call it `micro` instead). The default behavior is for the collector to provide a new cookie/network user ID for each event it receives. You can override the collector cookie's value with your own generated ID using the `set_network_user_id` method.

The user fingerprint is also a client-side concept. The JavaScript Tracker generates a fingerprint based on browser features and attaches it to all client-side events. You could develop your own fingerprints to attach to your Ruby server-side events with `set_fingerprint`.

## Adding page data with Page

If the Ruby tracker is incorporated into a website server, the events tracked will describe user activity on specific webpages. Knowing on which page an event occurred can be very valuable.

Add page URL, page title and referrer URL to any event by adding a Page object to any `track_x_event` method call. These parameters, of course, form the basis of the page view event. By directly adding page details to your other event types, you can avoid needing to JOIN onto your page view events to find out where they occurred.

Example:

```ruby
# Adding Page data to a struct event
page = SnowplowTracker::Page.new(page_url: 'http://www.example.com/register')

tracker.track_struct_event(category: 'forms',
                           action: 'start-input',
                           page: page)
```

The Page class was added in tracker version 0.7.0.

## Event timestamps

Processed Snowplow events have five different timestamps. They can have either `dvce_created_tstamp` or `true_tstamp`.

| **Timestamp name**    | **Description**                                                                                                        |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `dvce_created_tstamp` | Added during event creation                                                                                            |
| `true_tstamp`         | This can be manually set as an alternative to `dvce_created_tstamp`                                                    |
| `dvce_sent_tstamp`    | Added by the Emitter on event sending                                                                                  |
| `collector_tstamp`    | Added by the event collector                                                                                           |
| `etl_tstamp`          | Added after event enrichment during the processing pipeline                                                            |
| `derived_tstamp`      | Either a calculated value (`collector_tstamp - (dvce_sent_tstamp - dvce_created_tstamp)`) or the same as `true_tstamp` |

Overriding the default event timestamp (`dvce_created_tstamp`) can be useful in some situations. For example, if the Snowplow event refers to an action that happened previously but is only now being tracked. This can be achieved using the Ruby tracker classes DeviceTimestamp and TrueTimestamp (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Timestamp.html)).

---

# Configure how events are sent for the Ruby tracker
> Configure Ruby emitters and async emitters with collector endpoint, HTTP method, buffer size, and manual flushing for event delivery.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/configuring-how-events-are-sent/

When you [initialize your Tracker object](/docs/sources/ruby-tracker/getting-started/#tracking-design-and-initialization), you will need to provide one or more Emitter objects. Remember that we advise using the Singleton pattern, to avoid constantly recreating your objects.

There are two types of Emitter. The Emitter parent class can only send events synchronously. The AsyncEmitter subclass sends events asynchronously by default. We recommend you use the AsyncEmitter, to avoid slowing down your app.

Emitters and AsyncEmitters are initialized in the same way. An event collector endpoint is required, but there are various optional parameters that can also be set (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Emitter.html) and below table). A Tracker can have more than one associated Emitter or AsyncEmitter. These can be provided at Tracker initialization, or added later on.

Example:

**v0.7.0+:**

```ruby
emitter1 = SnowplowTracker::Emitter.new(endpoint: 'collector.example.com')
emitter2 = SnowplowTracker::AsyncEmitter.new(endpoint: 'collector2.example.com',
                                             options: { method: 'post' })
emitter3 = SnowplowTracker::AsyncEmitter.new(endpoint: 'collector.elsewhere.com',
                                             options: { protocol: 'https' })

tracker = SnowplowTracker::Tracker.new(emitters: [emitter1, emitter2])
tracker.add_emitter(emitter3)
```

**Before v0.7.0:**

```ruby
emitter1 = SnowplowTracker::Emitter.new('collector.example.com')
emitter2 = SnowplowTracker::AsyncEmitter.new('collector2.example.com',
                                             { method: 'post' })
emitter3 = SnowplowTracker::AsyncEmitter.new('collector.elsewhere.com',
                                             { protocol: 'https' })

tracker = SnowplowTracker::Tracker.new([emitter1, emitter2])
tracker.add_emitter(emitter3)
```

***

Optional Emitter settings:

| **Optional parameter** | **Description**                                                   | **Default** |
| ---------------------- | ----------------------------------------------------------------- | ----------- |
| `path`                 | Override the default path for appending to the endpoint           | n/a         |
| `protocol`             | `http` or `https`                                                 | `http`      |
| `port`                 | The port for the connection                                       | n/a         |
| `method`               | `get` or `post`                                                   | `get`       |
| `buffer_size`          | The size of the buffer, i.e. the number of events to send at once | 1           |
| `on_success`           | A method to call if events were all sent successfully             | n/a         |
| `on_failure`           | A method to call if any events did not send                       | n/a         |
| `thread_count`         | Number of threads to use (relevant to AsyncEmitters only)         | 1           |
| `logger`               | Where to log                                                      | STDERR      |

Response status codes of 2xx or 3xx status codes are considered successful.

### Manual flushing

You may want to force an emitter to send all events in its buffer, even if the buffer is not full. The `Tracker` class has a `flush` method which flushes all its emitters. It accepts one argument, `async`, which defaults to false. Unless you set `async` to `true`, the flush will be synchronous: it will block until all queued events have been sent. Of course, if you are using Emitters rather than AsyncEmitters, the flush will always be synchronous, even if `async` is set to `true`.

Example:

**v0.7.0+:**

```ruby
# Synchronous flush
tracker.flush

# Asynchronous flush
tracker.flush(async: true)
```

**Before v0.7.0:**

```ruby
# Synchronous flush
tracker.flush

# Asynchronous flush
tracker.flush(true)
```

***

### Logging

Emitters log messages about their activity and success during event sending. By default, Emitters log their activity to STDERR, using the Ruby standard library [Logger class](https://ruby-doc.org/stdlib-2.7.2/libdoc/logger/rdoc/Logger.html). Messages with a message level above INFO are logged. Advanced users might wish to alter how this logging occurs. You can change the message logging level, or, alternatively, you might want to disable logging completely, or log somewhere other than STDERR. Read how to do this in the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Emitter.html).

---

# Get started with the Ruby tracker
> Install the Ruby tracker gem, initialize trackers with emitters, configure subject properties, and implement singleton pattern for Ruby and Rails apps.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/getting-started/

The Snowplow Ruby Tracker is compatible with Ruby 2.1+, including 3.0+. To add the Snowplow Tracker to your Ruby app or gem, add this line to your Gemfile:

```ruby
gem 'snowplow-tracker', '~> 0.8.0'
```

To make the Snowplow Ruby Tracker work with as many different Ruby programs as possible, we have tried to keep external dependencies to a minimum. There are only two external dependencies currently, both of which are for development of the gem itself: [rspec](https://rspec.info/) and [webmock](https://rubygems.org/gems/webmock).

## Introduction to the tracker code

Find the Ruby tracker code [here](https://github.com/snowplow/snowplow-ruby-tracker), and the API documentation [here](https://snowplow.github.io/snowplow-ruby-tracker/index.html).

The main class of the Ruby tracker is the Tracker class. Trackers provide methods for tracking events, such as `track_page_view`. The Tracker creates an appropriate event payload out of the provided event properties. This payload is passed to one or more Emitters for sending to the event collector.

Data about users, and which platform (e.g. server-side app or mobile) the event occurred on, are managed by Subject objects. A Subject can be added to each event, and Trackers always have an associated Subject. Other information can be added to events using Page, DeviceTimestamp or TrueTimestamp objects, as well as via event context. Event context is added via SelfDescribingJson objects.

> **Note:** The Ruby tracker version 0.7.0 had some breaking changes, notably the addition of keyword arguments. Check out the tracker changelog [here](https://github.com/snowplow/snowplow-ruby-tracker/blob/master/CHANGELOG).

## Tracking design and initialization

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

We suggest implementing the Ruby tracker as a Singleton global object. This pattern is demonstrated in our [Ruby on Rails example app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples). Structuring your code in this way avoids wasting bandwidth or processing on reinitializing Trackers and Emitters for every page load or event sent.

Note that the [Rails demo](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) has both the Ruby and JavaScript trackers implemented. This combination of client- and server-side tracking can be highly effective and powerful. For example, you can discover how much effect adblockers are having on your tracking, by comparing the amount of client-side and server-side page view events you collect. It also allows you to track events in the most appropriate way for the event type. Check out this [blog post](https://snowplowanalytics.com/blog/2021/11/09/the-unrivaled-power-of-joining-client-and-server-side-tracking/) for more discussion about tracking client- and server-side.

The Tracker must be initialized with an Emitter or the subclass AsyncEmitter (or an array of Emitters/AsyncEmitters). The only required argument for an Emitter/AsyncEmitter is the endpoint, i.e. the address of the event collector.

Initializing a tracker instance:

**v0.7.0+:**

```ruby
require 'snowplow-tracker'

emitter = SnowplowTracker::Emitter.new(endpoint: 'collector.example.com')
tracker = SnowplowTracker::Tracker.new(emitters: emitter)
# or
tracker = SnowplowTracker::Tracker.new(emitters: [emitter])
```

**Before v0.7.0:**

```ruby
require 'snowplow-tracker'

emitter = SnowplowTracker::Emitter.new('collector.example.com')
tracker = SnowplowTracker::Tracker.new(emitter)
# or
tracker = SnowplowTracker::Tracker.new([emitter])
```

***

This Tracker will send events via GET to `http://collector.example.com/i`. To use other settings, such as POST or HTTPS, see "[Configuring how events are sent](/docs/sources/ruby-tracker/configuring-how-events-are-sent/)".

> **Warning:** Do not include the protocol (HTTP or HTTPS) in your collector address. It will be added automatically.

At initialization, two Tracker parameters can be set which will be added to all events. The first is the Tracker `namespace`. This is especially useful to distinguish between events from different Trackers, if more than one is being used. The second user-set Tracker property is the `app_id`. This is the unique identifier for the site or application, and is particularly useful for distinguishing between events when Snowplow tracking has been implemented in multiple apps.

A Tracker is always associated with a Subject. It will be generated automatically if one is not provided during initialization. It can be swapped out later for another Subject using `set_subject`.

The final initialization parameter is a setting for the base64-encoding of any JSONs in the event payload. The default is for JSONs to be encoded. Once the Tracker has been instantiated, it is not possible to change this setting.

Initializing a tracker instance with all possible settings used:

**v0.7.0+:**

```ruby
require 'snowplow-tracker'

SnowplowTracker::Tracker.new(emitters: SnowplowTracker::Emitter.new(endpoint: 'collector.example.com'),
                             subject: SnowplowTracker::Subject.new,
                             namespace: 'tracker_no_encode',
                             app_id: 'rails_main',
                             encode_base64: false
                             )
```

**Before v0.7.0:**

```ruby
require 'snowplow-tracker'

SnowplowTracker::Tracker.new(SnowplowTracker::Emitter.new('collector.example.com'),
                             SnowplowTracker::Subject.new,
                             'tracker_no_encode',
                             'rails_main',
                             false
                             )
```

***

You can create as many Tracker instances as you like within your app; each one is completely sandboxed. For example, you may wish to send certain events to a different collector endpoint (using a different Emitter), or with a different JSON base64-encoding choice. Make sure you set different Tracker namespaces when using multiple instances.

## Testing your tracking

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

Check out our [Ruby tracker Rails demo](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples) and [Snowplow Micro examples repo](https://github.com/snowplow-incubator/snowplow-micro-examples) for two examples of automated webapp testing using Snowplow Micro and the testing framework [Cypress](https://www.cypress.io/). Any end-to-end testing framework can be used.

---

# Ruby tracker SDK
> Track events in Ruby applications, gems, and Rails web apps with the Snowplow Ruby tracker SDK compatible with Ruby 2.1+ and 3.0+.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/

The Snowplow Ruby Tracker allows you to track Snowplow events in your Ruby applications and gems and Ruby on Rails web applications.

The Ruby tracker is compatible with Ruby 2.1+, including Ruby 3.0+. The current tracker version is 0.8.0.

These pages will help you in setting up and using the Ruby tracker. There are more technical details in the [API docs.](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker.html)

As a server-side tracker, the Ruby tracker pairs very well with the JavaScript client-side tracker, if you are tracking events in a website. We've created a [simple demo Rails app](https://github.com/snowplow-industry-solutions/snowplow-ruby-tracker-examples), which incorporates both trackers, and also uses the Singleton pattern that we suggest for the Ruby tracker.

At its simplest, the Ruby tracker is used by calling a Tracker object's `track_x_event` method, such as `track_page_view`, which will send the event payload created to the Snowplow event collector.

There are three main classes which the Ruby Tracker uses: trackers, emitters, and subjects. These are introduced in more detail on the [next page](/docs/sources/ruby-tracker/getting-started/). There are also further classes which allow you to add extra data to your events, discussed on the page '[Adding data to your events: context and more](/docs/sources/ruby-tracker/adding-data-events/)'.

---

# Track events with the Ruby tracker
> Track self-describing events, structured events, page views, screen views, and ecommerce transactions with the Ruby tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/ruby-tracker/tracking-events/

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your websites and apps.

We provide several built-in methods to help you track different kinds of events. The `track_x_event` methods range from single purpose methods, such as `track_page_view`, to the more complex but flexible `track_self_describing_event`, which can be used to track any kind of user behavior. We strongly recommend using `track_self_describing_event` for your tracking, as it allows you to design custom event types to match your business requirements. This [post on our blog](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/), "Re-thinking the structure of event data" might be informative here.

Tracking methods supported by the Ruby Tracker:

| **Method**                    | **Event type tracked**                               |
| ----------------------------- | ---------------------------------------------------- |
| `track_self_describing_event` | Custom event based on "self-describing" JSON schema  |
| `track_struct_event`          | Semi-custom structured event                         |
| `track_page_view`             | View of web page                                     |
| `track_screen_view`           | View of screen                                       |
| `track_ecommerce_transaction` | eCommerce transaction (and items in the transaction) |

All the `track_x_event` methods share common features and parameters. Every type of event can have an optional context, Subject, and Page added. A Timestamp can also be provided for all event types to override the default event timestamp. See [the next page](/docs/sources/ruby-tracker/adding-data-events/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event tracking methods, followed by the out-of-the-box event types. Note that you can also design and create your own page view, screen view, or eCommerce events, using `track_self_describing_event`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

### Track self-describing events with `track_self_describing_event`

Use `track_self_describing_event` to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the `data` value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

This method was previously called `track_unstruct_event`, as a counterpoint to `track_struct_event`. This name is misleading and we are in the process of depreciating it. An `unstruct` event requires a schema ruleset and therefore can be considered more structured than a `struct` event. However, this method still produces events labelled `unstruct`, as changing the event name in the Tracker Protocol would be a breaking change.

The `track_self_describing_event` method has one required argument, which must be a SelfDescribingJson object (see [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/SelfDescribingJson.html)). This class takes a schema name and a flat hash of event data. The keys of the hash can be either strings or Ruby symbols.

Example:

**v0.7.0+:**

```ruby
self_desc_json = SnowplowTracker::SelfDescribingJson.new(
  "iglu:com.example_company/save_game/jsonschema/1-0-2",
  {
    "saveId" => "4321",
    "level" => 23,
    "difficultyLevel" => "HARD",
    "dlContent" => true
  }
)

tracker.track_self_describing_event(event_json: self_desc_json)
```

**Before v0.7.0:**

```ruby
self_desc_json = SnowplowTracker::SelfDescribingJson.new(
  "iglu:com.example_company/save_game/jsonschema/1-0-2",
  {
    "saveId" => "4321",
    "level" => 23,
    "difficultyLevel" => "HARD",
    "dlContent" => true
  }
)

tracker.track_self_describing_event(self_desc_json)
```

***

You can track anything you want using this method, as long as you can describe it in a self-describing JSON schema.

### Track structured events with `track_struct_event`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| **Argument** | **Description**                                                  | **Required in event?** |
| ------------ | ---------------------------------------------------------------- | ---------------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes                    |
| `action`     | Defines the type of user interaction which this event involves   | Yes                    |
| `label`      | Often used to refer to the 'object' the action is performed on   | No                     |
| `property`   | Describing the 'object', or the action performed on it           | No                     |
| `value`      | Provides numerical data about the event                          | No                     |

Example:

**v0.7.0+:**

```ruby
tracker.track_struct_event(category: 'shop',
                           action: 'add-to-basket',
                           property: 'pcs',
                           value: 2)
```

**Before v0.7.0:**

```ruby
tracker.track_struct_event('shop', 'add-to-basket', nil, 'pcs', 2)
```

***

### Track page views with `track_page_view`

This is a simple, single-use method for tracking a user viewing a page within your app. You can record the page URL, the page title, and the referrer URL. Only the page URL is strictly required.

As a server-side language, your Ruby code won't automatically have access to the page title. This is one small reason why page views are easier to track client-side. Conversely, server-side page view tracking is more accurate, as it is not blocked by adblockers. It can be useful to compare counts from client- and server-side page views to see how much effect adblockers are having.

Example:

**v0.7.0+:**

```ruby
tracker.track_page_view(page_url: 'www.example.com',
                        page_title: 'example',
                        referrer: 'www.referrer.com')
```

**Before v0.7.0:**

```ruby
tracker.track_page_view('www.example.com', 'example', 'www.referrer.com')
```

***

### Track screen views with `track_screen_view`

Use `track_screen_view` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name` and `id`; while both are optional, you must provided at least one of them to create a valid event. "Name" is the human-readable screen name, and "ID" should be the unique screen ID.

This method creates an `unstruct` event, by creating a SelfDescribingJson and calling `track_self_describing_event`. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the name and/or ID which you provide. That schema is hosted on the schema repository [Iglu Central](http://iglucentral.com/), and so will always be available to your pipeline.

Example:

**v0.7.0+:**

```ruby
tracker.track_screen_view(name: 'HUD > Save Game',
                          id: 'screen23')
```

**Before v0.7.0:**

```ruby
tracker.track_screen_view('HUD > Save Game', 'screen23')
```

***

### Track eCommerce transactions with `track-ecommerce-transaction`

Use this out-of-the-box method to track an ecommerce transaction. This method is unique compared to the other `track_x_event` methods, as it sends multiple multiple events: one `transaction` event, and one `transaction_item` event for each item in the transaction. This is a legacy design; if we were creating a new eCommerce tracking method now, we would attach information about each item as event context entities to a single transaction event.

The arguments for this method are a "transaction" hash, and an array of "item" hashes. These hashes are strict about which keys are allowed and required. Check out the [API docs](https://snowplow.github.io/snowplow-ruby-tracker/SnowplowTracker/Tracker.html#track_ecommerce_transaction-instance_method) for the full details of the allowed properties. You can also read there about how additional event properties (context, Subject, Page and Timestamp) are handled for eCommerce events.

Broadly, the "transaction" hash contains the information about the order as a whole: the order ID, the value of the transaction including tax and/or shipping, as well as geographic information about its origin, or the currency in which the order was placed.

The "item" hash records each item's unique SKU identifier, value, and how many were purchased, as well as any further information which might be useful, such as its human-readable item name.

Example:

**v0.7.0+:**

```ruby
transaction = {
  order_id: '12345',
  total_value: 35,
  city: 'London',
  country: 'UK',
  currency: 'GBP'
}
item1 = {
  sku: 'pbz0026',
  price: 20,
  quantity: 1,
  category: 'film'
}
item2 = {
  sku: 'pbz0038',
  price: 15,
  quantity: 1,
  name: 'red shoes'
}
tracker.track_ecommerce_transaction(transaction: transaction,
                                    items: [item1, item2])
```

**Before v0.7.0:**

> **Note:** Older versions of the Ruby tracker have a bug in `track_ecommerce_transaction`: hashes with symbols for keys are not accepted. The event will fail silently. You must use only string keys. This was fixed in version 0.7.0.

```ruby
transaction = {
  'order_id' => '12345',
  'total_value' => 35,
  'city' => 'London',
  'country' => 'UK',
  'currency' => 'GBP'
}
item1 = {
  'sku' => 'pbz0026',
  'price' => 20,
  'quantity' => 1,
  'category' => 'film'
}
item2 = {
  'sku' => 'pbz0038',
  'price' => 15,
  'quantity' => 1,
  'name' => 'red shoes'
}
tracker.track_ecommerce_transaction(transaction, [item1, item2])
```

***

This will fire three events: one for the transaction as a whole, and one for each item. The `order_id` and `currency` fields in the `transaction` argument will also be attached to each of the item events.

---

# Add entities and other data to events with the Rust tracker
> Add event context entities and subject data with user and platform information to Rust tracker events using self-describing JSON schemas.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/adding-data/

There are multiple ways to add extra data to your tracked events, adding richness and value to your dataset:

1. Event context using self-describing data: Attach event context, describing anything you like, in the form of self-describing JSONs.
2. Subject: Include information about the user, or the platform on which the event occurred.

## Event context

Event context is an incredibly powerful aspect of Snowplow tracking, which allows you to create very rich data. It is based on the same self-describing JSON schemas as the self-describing events. Using event context, you can add any details you like to your events, as long as you can describe them in a self-describing JSON schema.

Each schema will describe a single "entity". All of an event’s entities together form the event context. The event context will be sent as one field of the event, finally ending up in one column (`context`) in your data storage. There is no limit to how many entities can be attached to one event.

Note that context can be added to any event type, not just self-describing events. This means that even a simple event type like a page view can hold complex and extensive information – reducing the chances of data loss and the amount of modeling (JOINs etc.) needed in modeling, while increasing the value of each event, and the sophistication of the possible use cases.

The entities you provide are validated against their schemas as the event is processed (during the enrich phase). If there is a mistake or mismatch, the event is processed as a Bad Event.

Once defined, an entity can be attached to any kind of event. This is also an important point; it means your tracking is as DRY as possible. Using the same "user" or "image" or "search result" (etc.) entities throughout your tracking reduces error, and again makes the data easier to model.

Example:

```rust
// Tracking a Self-Describing event with context entity
let self_describing_event = SelfDescribingEvent::builder()
    .schema("iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0")
    .data(json!({"name": "test", "id": "something else"}))
    .build()?;

let event_context = Some(vec![SelfDescribingJson::new(
    "iglu:org.schema/WebPage/jsonschema/1-0-0",
    json!({"keywords": ["tester"]}),
)]);

let self_desc_event_id = tracker.track(self_describing_event, event_context)?;
```

## Adding user and platform data with Subject

Subject information describes the user and device associated with the event. This information can be entered during tracker initialization by passing a `Subject` instance to the `Snowplow::create_tracker` method. All of the information is optional.

Some subject information is filled automatically by the tracker, such as the platform of the user, timezone, and language.

Please refer to the section on subject configuration on the [Configuration page](/docs/sources/rust-tracker/initialization-and-configuration/) to learn more.

---

# Get started with the Rust tracker
> Install the Rust tracker from crates.io, initialize with namespace and collector URL, and track screen view events with Tokio runtime.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/getting-started/

The following steps will guide you through setting up the Rust tracker in your project and tracking a simple event.

## Installation

Add the snowplow\_tracker as a dependency in Cargo.toml inside your Rust application:

```toml
[dependencies]
snowplow_tracker = 0.2.0
```

Use the package APIs in your code:

```rust
use snowplow_tracker::Snowplow;
```

## Initialization

Instantiate a tracker using the `Snowplow::create_tracker` function. The function takes three required arguments: `namespace`, `app_id`, `collector_url`, and one optional argument, `subject`. Tracker `namespace` identifies the tracker instance; you may create multiple trackers with different namespaces. The `app_id` identifies your app. The `collector_url` is the URI of the Snowplow collector to send the events to. `subject` allows for an optional Subject to be attached to the tracker, which will be sent with all events.

```rust
let mut tracker = Snowplow::create_tracker("ns", "app_id", "https://...", None);
```

There are additional optional arguments to configure the tracker. To learn more about configuring how events are sent, check out [this page](/docs/sources/rust-tracker/initialization-and-configuration/).

## Tracking events

To track events, simply instantiate their respective types and pass them to the tracker.track method with optional context entities. Please refer to the documentation for specification of event properties.

```rust
// Tracking a Screen View event
let screen_view_event = ScreenViewEvent::builder()
    .id(Uuid::new_v4())
    .name("a screen view")
    .previous_name("previous name")
    .build()?;

let screen_view_event_id = tracker.track(screen_view_event, None)?;
```

## Safely closing the emitter

The emitter spawns its own thread to send events using a Tokio runtime. To allow the tracker to drop, the emitter must be closed. This can be done by calling `Tracker.close_emitter()`.

```rust
let mut tracker = Snowplow::create_tracker("ns", "app_id", "https://...", None);
// track some events...
tracker.close_emitter();
```

Visit documentation about [tracking events](/docs/sources/rust-tracker/tracking-events/) to learn about other supported event types. You may also want to read about [adding more data to tracked events](/docs/sources/rust-tracker/adding-data/).

## Logging

The [log](https://crates.io/crates/log) crate is used for logging. To enable logging, you must add a logger to your application, such as [env\_logger](https://crates.io/crates/env_logger).

## Testing

Testing that your event tracking is properly configured can be as important as testing the other aspects of your app. It confirms that you are generating the events you expect.

We recommend using [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and debugging. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/), and even use it for [automated tests](/docs/testing/snowplow-micro/automated-testing/).

---

# Rust tracker SDK
> Add event tracking to Rust applications using the Snowplow Rust tracker SDK with async event sending and retry policies.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/

The Snowplow Rust Tracker allows you to add analytics to your Rust apps.

The tracker is published on crates.io as [snowplow\_tracker](https://crates.io/crates/snowplow_tracker).

> **Warning:** If you are upgrading from 0.1.0 to 0.2.0, there are a couple of changes required to make to your code:
>
> - [`Tracker.track`](/docs/sources/rust-tracker/getting-started/#tracking-events) is no longer an async function
> - The Emitter must be safely closed to allow the tracker to drop, as it spawns a thread to send events. This can be done by calling `Tracker.close_emitter()`.

---

# Initialize and configure the Rust tracker
> Configure Rust tracker with subject information, event store, HTTP client, batch emitter, and retry policies for async event delivery.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/initialization-and-configuration/

The `Snowplow` module provides a single method to initialize and configure a new tracker, the `Snowplow::create_tracker` method. It accepts configuration parameters for the tracker and returns a `Tracker` instance.

```rust
use snowplow_tracker::Subject;
let subject = Subject::builder().language("en-gb").build()?;
let tracker = Snowplow::create_tracker("ns", "app_id", "https://...", Some(subject));
```

The method returns a `Tracker` instance. This can be later used for tracking events, or accessing tracker properties.

The required attributes of the `Snowplow::create_tracker` method are `namespace` used to identify the tracker, `app_id` to identify your app, and the Snowplow collector `endpoint`. Additionally, one can pass in a `Subject` to provide additional context about the application environment.

| Attribute   | Type      | Description                                                                |
| ----------- | --------- | -------------------------------------------------------------------------- |
| `namespace` | `&str`    | Tracker namespace to identify the tracker.                                 |
| `app_id`    | `&str`    | Application identifier.                                                    |
| `endpoint`  | `&str`    | URI for the Snowplow collector endpoint.                                   |
| `subject`   | `Subject` | Subject information about tracked user and device that is added to events. |

## Configuration of subject information: `Subject`

Subject information are persistent and global information about the tracked device or user. They apply to all events and are assigned as event properties.

The configured attributes are mapped to Snowplow event properties described in the [Snowplow Tracker Protocol](/docs/events/). They are mapped as follows:

| Attribute         | Event Property   |
| ----------------- | ---------------- |
| `user_id`         | `uid`            |
| `network_user_id` | `network_userid` |
| `domain_user_id`  | `domain_userid`  |
| `user_agent`      | `useragent`      |
| `ip_address`      | `user_ipaddress` |
| `timezone`        | `os_timezone`    |
| `language`        | `lang`           |

## Configuring the Tracker

If you're looking for more control of your tracker, most components can be configured using their respective builder functions, or created from scratch by implementing the relevant trait.

## Event Store

An event store is defined by the `EventStore` trait. The default event store is the `InMemoryEventStore`. A custom event store can be created by implementing the `EventStore` trait, and passing it to the `BatchEmitter` builder.

### In Memory Event Store

The default event store for the tracker. This event store that keeps events in memory and does not persist them to disk. As `tracker.track` is called, events are added to the event store. When the event store has enough events to create an `EventBatch`, the events will be removed from the store, and sent to the collector.

The default capacity for the `InMemoryEventStore` is 10,000 events, and the default batch size is 50 events. An attempt to add an event to the store when it is full will result in the event being dropped, so choosing the right capacity is important. The capacity and batch size can be configured using the builder, for example, if we wanted to increase the capacity to 100,000 events and the batch size to 75 events:

```rust
let event_store = InMemoryEventStore::builder()
    .capacity(100_000)
    .batch_size(75)
    .build()?;
```

> **Warning:** Events in the `InMemoryEventStore` will be lost if the process is unexpectedly terminated.

## HTTP Client

A HTTP client is defined by the `HttpClient` trait. The default tracker HTTP client is the `ReqwestClient`, but, as with the Event Store, any implementation of the `HttpClient` trait can be passed to to the `BatchEmitter` builder:

```rust
let custom_http_client = MyCustomHttpClient::new();
let emitter = BatchEmitter::builder()
    .collector_url("http://localhost:9090")
    .http_client(custom_http_client)
    .build()?;
```

> **Note:** This is an async trait. The [async-trait](https://github.com/dtolnay/async-trait) crate is used until async traits are supported in stable Rust.

### Reqwest Client

The default HTTP client for the tracker. This client uses the [`reqwest`](https://github.com/seanmonstar/reqwest) crate to asynchronously send HTTP requests to the collector. The client only supports sending POST requests.

## Emitter

An emitter is defined by the `Emitter` trait, with the default emitter being the `BatchEmitter`. Any implementation of the `Emitter` trait can be passed to `Tracker::new`:

```rust
let custom_emitter = MyCustomEmitter::new();
let mut tracker = Tracker::new("ns", "app_id", custom_emitter, None);
```

### Batch Emitter

The BatchEmitter sends events in batches. This is more efficient than sending event requests singly, as only one set of POST headers is required for a number of events. The event collector receives the request and separates out the individual event payloads. Although the Snowplow Tracker interface is sync, the BatchEmitter uses [Tokio](https://github.com/tokio-rs/tokio) to asynchronously send events to the collector.

#### Retry Behavior

Failed requests are retried with exponentially increasing delays between subsequent retry requests. This ensures that the Collector is not overwhelmed with too many requests and also saves resources on the client by reducing the number of requests during failures. The number of times a request is retried is determined by the Retry Policy.

#### Retry Policy

A retry policy can be passed into the Emitter to define how many times a request should be retried.

The available policies are:

- `RetryForever` A RetryPolicy with no cap on maximum of attempts to send an event to the collector. This policy might appear attractive where it is critical to avoid data loss because it never deliberately drops events. However, it could cause a backlog of events in the buffered queue if the collector is unavailable for too long.
- `MaxAttempts(max: u32)` A RetryPolicy which drops events after failing to contact the collector within a fixed number of attempts. This policy can smooth over short outages of connection to the collector. Events will be dropped only if the collector is unreachable for a relatively long span of time. Dropping events can be a safety mechanism against a growing backlog of unsent events.
- `NoRetry` A RetryPolicy that drops events immediately after a failed attempt to send to the collector.

The default retry policy on the `BatchEmitter` is `RetryPolicy::MaxAttempts(10)`, but can be configured using the builder, for example if we wanted to disable retries:

```rust
let emitter = BatchEmitter::builder()
    .collector_url("http://localhost:9090")
    .retry_policy(RetryPolicy::NoRetry)
    .build()?;
```

#### Safely Stopping the Emitter

The `BatchEmitter` can be safely stopped using the `tracker.close_emitter()` method. This will stop the emitter from accepting new events, and will wait for any in-flight requests to complete before returning. This is useful as the emitter runs in a separate thread, and the main thread needs to wait for the emitter to finish sending events before exiting.

---

# Track events with the Rust tracker
> Track self-describing events, structured events, screen views, and timing events with the Rust tracker SDK using builder patterns.
> Source: https://docs.snowplow.io/docs/sources/rust-tracker/tracking-events/

Designing how and what to track in your app is an important decision. Check out our docs about tracking design [here](/docs/event-studio/).

Snowplow has been built to enable you to track a wide range of events that occur when users interact with your apps.

We provide several built-in events to help you track different kinds of events. When instantiated, their objects can be passed to the `tracker.track()` method to send events to the Snowplow collector. The events range from single purpose ones, such as `ScreenViewEvent`, to the more complex but flexible `SelfDescribingEvent`, which can be used to track any kind of user behavior. We strongly recommend using `SelfDescribingEvent` for your tracking, as it allows you to design custom event types to match your business requirements. [This post](https://snowplowanalytics.com/blog/2020/01/24/re-thinking-the-structure-of-event-data/) on our blog, "Re-thinking the structure of event data" might be informative here.

Event classes supported by the Rust Tracker:

| Method                | Event type tracked                                          |
| --------------------- | ----------------------------------------------------------- |
| `SelfDescribingEvent` | Custom event based on "self-describing" JSON schema         |
| `StructuredEvent`     | Semi-custom structured event                                |
| `ScreenViewEvent`     | View of a screen in the app                                 |
| `TimingEvent`         | User timing events such as how long resources take to load. |

All the methods share common features and parameters. Every type of event can have an optional context added. See the [next page](/docs/sources/rust-tracker/adding-data/) to learn about adding extra data to events. It's important to understand how event context works, as it is one of the most powerful Snowplow features. Adding event context is a way to add depth, richness and value to all of your events.

Snowplow events are all processed into the same format, regardless of the event type (and regardless of the tracker language used). Read about the different properties and fields of events in the [Snowplow Tracker Protocol](/docs/events/).

We will first discuss the custom event types, followed by the out-of-the-box event types. Note that you can also design and create your own page view, or screen view, using `SelfDescribingEvent`, to fit your business needs better. The out-of-the-box event types are provided so you can get started with generating event data quickly.

## Track self-describing events with `SelfDescribingEvent`

Use the `SelfDescribingEvent` type to track a custom event. This is the most advanced and powerful tracking method, which requires a certain amount of planning and infrastructure.

Self-describing events are based around "self-describing" (self-referential) JSONs, which are a specific kind of [JSON schema](http://json-schema.org/). A unique schema can be designed for each type of event that you want to track. This allows you to track the specific things that are important to you, in a way that is defined by you.

This is particularly useful when:

- You want to track event types which are proprietary/specific to your business
- You want to track events which have unpredictable or frequently changing properties

A self-describing JSON has two keys, `schema` and `data`. The `schema` value should point to a valid self-describing JSON schema. They are called self-describing because the schema will specify the fields allowed in the data value. Read more about how schemas are used with Snowplow [here](/docs/fundamentals/schemas/).

After events have been collected by the event collector, they are validated to ensure that the properties match the self-describing JSONs. Mistakes (e.g. extra fields, or incorrect types) will result in events being processed as Bad Events. This means that only high-quality, valid events arrive in your data storage or real-time stream.

> **Note:** Your schemas must be accessible to your pipeline to allow this validation. See [Managing data structures](/docs/event-studio/data-structures/) for information on how to create and update schemas.

Example (assumes that `tracker` is a tracker instance created using `Snowplow::create_tracker`):

```rust
// Tracking a Self-Describing event with context entity
let self_describing_event = SelfDescribingEvent::builder()
    .schema("iglu:com.snowplowanalytics.snowplow/screen_view/jsonschema/1-0-0")
    .data(json!({"name": "test", "id": "something else"}))
    .build()?;

let event_context = Some(vec![SelfDescribingJson::new(
    "iglu:org.schema/WebPage/jsonschema/1-0-0",
    json!({"keywords": ["tester"]}),
)]);

let self_desc_event_id = tracker.track(self_describing_event, event_context)?;
```

## Track structured events with `StructuredEvent`

This method provides a halfway-house between tracking fully user-defined self-describing events and out-of-the box predefined events. This event type can be used to track many types of user activity, as it is somewhat customizable. "Struct" events closely mirror the structure of Google Analytics events, with "category", "action", "label", and "value" properties.

As these fields are fairly arbitrary, we recommend following the advice in this table how to define structured events. It's important to be consistent throughout the business about how each field is used.

| Argument   | Description                                                    | Required in event? |
| ---------- | -------------------------------------------------------------- | ------------------ |
| `category` | The grouping of structured events which this action belongs to | Yes                |
| `action`   | Defines the type of user interaction which this event involves | Yes                |
| `label`    | Often used to refer to the 'object' the action is performed on | No                 |
| `property` | Describing the 'object', or the action performed on it         | No                 |
| `value`    | Provides numerical data about the event                        | No                 |

Example:

```rust
// Tracking a Structured event
let structured_event = StructuredEvent::builder()
    .category("shop")
    .action("add-to-basket")
    .label("Add To Basket")
    .property("pcs")
    .value(2.0)
    .build()?;

let struct_event_id = tracker.track(structured_event, None)?;
```

## Track screen views with `ScreenViewEvent`

Use `ScreenViewEvent` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not webpages. The arguments are `name`, `id`, `type`, and `transitionType`. The `name` and `id` properties are required. "Name" is the human-readable screen name, and "ID" should be the unique screen ID (UUID v4).

This method creates an unstruct event, by creating and tracking a self-describing event. The schema ID for this is "iglu:com.snowplowanalytics.snowplow/screen\_view/jsonschema/1-0-0", and the data field will contain the parameters which you provide. That schema is hosted on the schema repository Iglu Central, and so will always be available to your pipeline.

| Argument         | Description                                                 | Required in event? |
| ---------------- | ----------------------------------------------------------- | ------------------ |
| `name`           | The name of the screen viewed.                              | Yes                |
| `id`             | The id (UUID v4) of screen that was viewed.                 | Yes                |
| `type`           | The type of screen that was viewed.                         | No                 |
| `previousName`   | The name of the previous screen that was viewed.            | No                 |
| `previousType`   | The type of screen that was viewed.                         | No                 |
| `previousId`     | The id (UUID v4) of the previous screen that was viewed.    | No                 |
| `transitionType` | The type of transition that led to the screen being viewed. | No                 |

Example:

```rust
// Tracking a Screen View event
let screen_view_event = ScreenViewEvent::builder()
    .id(Uuid::new_v4())
    .name("a screen view")
    .previous_name("previous name")
    .build()?;

let screen_view_event_id = tracker.track(screen_view_event, None)?;
```

## Track timing events with `TimingEvent`

Use the `TimingEvent` type to track user timing events such as how long resources take to load. These events take a timing `category`, the `variable` being measured, and the `TimingEvent` time measurement. An optional `label` can be added to further identify the timing event

| Argument   | Description                                              | Required in event? |
| ---------- | -------------------------------------------------------- | ------------------ |
| `category` | Defines the timing category.                             | Yes                |
| `variable` | Defines the timing variable measured.                    | Yes                |
| `timing`   | Represents the time.                                     | Yes                |
| `label`    | An optional string to further identify the timing event. | No                 |

Example:

```rust
let timing_event = TimingEvent::builder()
    .category("fetch_resource")
    .variable("map_loaded")
    .timing(1423)
    .label("Time to fetch map resource")
    .build()?;

let timing_event_id = tracker.track(timing_event, None)?;
```

---

# Scala tracker SDK
> Track events in Scala applications and servers with subjects, emitters, and async event sending using the Snowplow Scala tracker SDK.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/

The Snowplow Scala Tracker allows you to track Snowplow events in your Scala apps and servers. The tracker should be straightforward to use if you are comfortable with Scala development.

There are three main classes which the Scala Tracker uses: subjects, emitters, and trackers.

A subject represents a single user whose events are tracked, and holds data specific to that user.

A tracker always has one active subject at a time associated with it. The default subject only has "platform=server" configured, but you can replace it with a subject containing more data. The tracker constructs events with that subject and sends them to one or more emitters, which sends them on to a Snowplow collector.

---

# Initialize the Scala tracker
> Initialize Scala tracker with ID or http4s emitters, configure buffer size, retry policies, event queue policies, and add callbacks for event sending.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/initialization/

Assuming you have completed the Scala Tracker Setup, you are ready to initialize the Scala Tracker.

## Emitters

Each tracker instance must be initialized with an Emitter which is responsible for firing events to a Collector. We offer two different modules of emitter: the http4s-emitter and the id-emitter

### ID Emitters

The emitters in the "id" module have a simpler developer interface, because return types are not wrapped in functional events. Use these emitters if your application code is not using any other typeclasses from the cats ecosystem.

The below code:

- Creates a non-blocking emitter `emitter1` which sends events to "mycollector.com" on the default port 80
- Creates a blocking emitter `emitter2` which sends events to “myothercollector.com” on port 8080
- Creates a tracker which can be used to send events to all emitters

```scala
import scala.concurrent.ExecutionContext.Implicits.global

import com.snowplowanalytics.snowplow.scalatracker.Tracker
import com.snowplowanalytics.snowplow.scalatracker.Emitter._
import com.snowplowanalytics.snowplow.scalatracker.emitters.id._
import cats.data.NonEmptyList

val emitter1 = AsyncEmitter.createAndStart(EndpointParams("mycollector.com"))
val emitter2 = SyncEmitter(EndpointParams("myothercollector.com", port = Some(8080)))
val tracker = new Tracker(NonEmptyList.of(emitter1, emitter2), "mytrackername", "myapplicationid")
```

When using this emitter, the methods on the tracker all return `Unit`, so there is no requirement to flatMap over effect types.

```scala
tracker.trackPageView("http://example.com") // returns Unit
tracker.trackTransaction("order1234", 42.0) // returns Unit
```

The `SyncEmitter` blocks the whole thread when it sends events to the collector. The `AsyncEmitter` sends requests asynchronously from the tracker's main thread of execution, but in doing so it blocks a thread on the provided execution for each http request. The blocking calls are wrapped in scala's [blocking construct](https://www.scala-lang.org/api/current/scala/concurrent/index.html#blocking%5BT%5D\(body:=%3ET\):T), which is respected by the global execution context.

We reccomend using `AsyncEmitter` if possible as to not block threads, only using `SyncEmitter` if you have a specific need for sync in your setup.

### Http4s Emitter

Backed by a [Http4s](https://http4s.org/) client, this emitter captures actions in the context of a functional effect type, such as a cats IO, ZIO or Monix Task. This is the recommended emitter if you are familiar with functional programming and the cats ecosystem of type classes.

```scala
import cats.data.NonEmptyList
import cats.effect.std.Random
import cats.effect.{IO, Resource}
import org.http4s.ember.client.EmberClientBuilder

import com.snowplowanalytics.snowplow.scalatracker._
import com.snowplowanalytics.snowplow.scalatracker.Emitter._
import com.snowplowanalytics.snowplow.scalatracker.emitters.http4s._

val resource: Resource[IO, Tracker[IO]] = for {
  client <- EmberClientBuilder.default[IO].build
  rng <- Resource.eval(Random.scalaUtilRandom[IO])
  emitter1 <- Http4sEmitter.build[IO](EndpointParams("mycollector.com"), client)(implicitly, rng)
  emitter2 <- Http4sEmitter.build[IO](EndpointParams("myothercollector.com", port = Some(8080)), client)(implicitly, rng)
} yield new Tracker(NonEmptyList.of(emitter1, emitter2), "mytrackername", "myapplicationid")

resource.use { tracker =>
  // Use the tracker inside this block to initialize and run your application
  MyApp.run(tracker)
}
```

The above code:

- Creates a Http4s client from the EmberClientBuilder. Http4s offers other variants of the client, but Ember is the recommended default choice.
- Creates a Random instance, required to randomize the delay on request retries
- Creates an emitter `emitter1` which sends events to "mycollector.com" on the default port 80
- Creates a second emitter `emitter2` which sends events to “myothercollector.com” on port 8080
- Creates a tracker which can be used to send events to all emitters

Your application might then use the tracker by flatMapping over the functional effect:

```scala
for {
  _ <- tracker.trackPageView("http://example.com")
  _ <- tracker.trackTransaction("order1234", 42.0)
} yield ()
```

## Subject

You can configure a subject with extra data and attach it to the tracker so that the data will be attached to every event:

```scala
val subject = new Subject()
  .setUserId("user-00035")
  .setPlatform(Desktop)
val tracker = Tracker(emitters, ns, appId).setSubject(subject)
```

Alternatively, you can set the subject for each event individually.

## Buffer Configuration

Emitters use a buffer, so they can send larger payloads comprising several events, instead of sending each event immediately. You can choose the behavior of your emitter's buffering by passing in a `BufferConfig` during initialisation. For example:

```scala
val emitter = Http4sEmitter.build[IO](endpoint, bufferConfig = PayloadSize(40000))
```

The available configs are:

- `EventsCardinality(size: Int)` Configures the emitter to buffer events and send batched payloads comprising a fixed number of events
- `PayloadSize(bytes: Int)` Configures the emitter to buffer events and send batched payloads of a minimum size in bytes
- `NoBuffering` Configures the emitter to send events immediately to the collector, without buffering for larger batches
- `OneOf(left: BufferConfig, right: BufferConfig)` Configures the emitter to buffer events until either of the inner BufferConfigs say the queue is full

## Retry Policies

Emitters can be configured to retry sending events to the collector if the initial attempt fails. For example:

```scala
val emitter = Http4sEmitter.build[IO](endpoint, retryPolicy = MaxAttempts(10))
```

The available policies are:

- `RetryForever` A RetryPolicy with no cap on maximum of attempts to send an event to the collector. This policy might appear attractive where it is critical to avoid data loss because it never deliberately drops events. However, it could cause a backlog of events in the buffered queue if the collector is unavailable for too long. This RetryPolicy could be paired with an `EventQueuePolicy` that manages the behavior of a large backlog.
- `MaxAttempts(max: Int)` A RetryPolicy which drops events after failing to contact the collector within a fixed number of attempts. This policy can smooth over short outages of connection to the collector. Events will be dropped only if the collector is unreachable for a relatively long span of time. Dropping events can be a safety mechanism against a growing backlog of unsent events.
- `NoRetry` A RetryPolicy that drops events immediately after a failed attempt to send to the collector.

## Event Queue Policy

An `EventQueuePolicy` becomes important when the queue of pending events grows to an unexpectedly large number; for example, if the collector is unreachable for a long period of time

Picking an EventQueuePolicy is an opportunity to protect against excessive heap usage by limiting the maximum size of the queue

An EventQueuePolicy can be paired with an appropriate `RetryPolicy`, which controls dropping events when they cannot be sent

For example:

```scala
val emitter = new Http4sEmitter(endpoint, queuePolicy = IgnoreWhenFull(100))
```

The available policies are:

- `UnboundedQueue`. An `EventQueuePolicy` with no upper bound on the number pending events in the emitter's queue. This policy never deliberately drops events, but it comes at the expense of potentially high heap usage if the collector is unavailable for a long period of time.
- `BlockwhenFull(limit: Int)`. An `EventQueuePolicy` that blocks the tracker's thread until the queue of pending events falls below a threshold. This policy never deliberately drops events, but it comes at the expense of blocking threads if the collector is unavailable for long period of time.
- `IgnoreWhenFull(limit: Int)`. An `EventQueuePolicy` that silently drops new events when the queue of pending events exceeds a threshold.
- `ErrorWhenFull(limit: Int)`. An `EventQueuePolicy` that raises an exception when the queue of pending events exceeds a threshold.

## Global contexts

You can configure your tracker to add a context to every event sent:

```scala
val tracker = Tracker(emitters, namespace, appId).addContext(selfDescribingJson)
```

There is also syntax for adding EC2 or GCE contexts automatically:

#### EC2 Context

Amazon [Elastic Cloud](https://aws.amazon.com/ec2/) can provide basic information about instance running your app. You can add this informational as additional custom context to all sent events by enabling it in Tracker after initializaiton of your tracker:

```scala
import com.snowplowanalytics.snowplow.scalatracker.metadata._
val tracker = Tracker(emitters, ns, appId).enableEc2Context()
```

#### Google Compute Engine Metadata context

Google [Cloud Compute Engine](https://cloud.google.com/compute) can provide basic information about instance running your app. You can add this informational as additional custom context to all sent events by enabling it in Tracker after initializaiton of your tracker:

```scala
import com.snowplowanalytics.snowplow.scalatracker.metadata._
val tracker = Tracker(emitters, ns, appId).enableGceContext()
```

This will add [`iglu:com.google.cloud.gce/instance_metadata/jsonschema/1-0-0`](https://github.com/snowplow/iglu-central/blob/152c90a72d5888460985ea43605afb5252180b10/schemas/com.google.cloud.gce/instance_metadata/jsonschema/1-0-0) context to all your events

## Callbacks

All emitters supplied with Scala Tracker support callbacks invoked after every sent event (or batch of events) whether it was successful or not. This feature particularly useful for checking collector unavailability and tracker debugging.

Callbacks should have following signature:

```scala
type Callback = (Emitter.EndpointParams, Emitter.Request, Emitter.Result) => Unit
```

- `EndpointParams` is collector configuration attached to emitter
- `Request` is raw collector's payload, which can be either `GET` or `POST` and holding number of undertaken attempts
- `Result` is processed collector's response or failure reason. You'll want to pattern-match it to either no-op or notify DevOps about non-working collector

To add a callback to `AsyncBatchEmitter` you can use following approach:

```scala
import com.snowplowanalytics.snowplow.scalatracker.Emitter._

def emitterCallback(params: EndpointParams, req: Request, res: Result): Unit = {
  res match {
    case Result.Success(_) => ()
    case Result.Failure(code) =>
      devopsIncident(s"Scala Tracker got unexpected HTTP code $code from ${params.getUri}")
    case Result.TrackerFailure(exception) =>
      devopsIncident(s"Scala Tracker failed to reach ${params.getUri} with following exception $exception after ${req.attempt} attempt")
    case Result.RetriesExceeded(failure) =>
      devopsIncident(s"Scala Tracker has stopped trying to deliver payload after following failure: $failure")
      savePayload(req)      // can be investigated and sent afterwards
  }
}

val emitter = AsyncEmitter.createAndStart(endpointParams, callback = Some(emitterCallback))
```

The async emitter will perform callbacks asynchronously in its `ExecutionContext`.

---

# Track events with the Scala tracker
> Track self-describing events, structured events, page views, errors, and ecommerce transactions with the Scala tracker using custom contexts and timestamps.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/sending-events/

This page lists the available methods for sending events to Snowplow.

## `trackSelfDescribingEvent`

Use `trackSelfDescribingEvent` to track a custom Self-describing events (previously known as Unstructured Events) which consists of a name and an unstructured set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

You can use its alias `trackUnstructEvent`.

| **Argument**        | **Description**                                    | **Required?** | **Type**                   |
| ------------------- | -------------------------------------------------- | ------------- | -------------------------- |
| `unstructuredEvent` | Self-describing JSON containing unstructured event | Yes           | `SelfDescribingJson`       |
| `contexts`          | List of custom contexts for the event              | No            | `List[SelfDescribingJson]` |
| `timestamp`         | Device created timestamp or true timestamp         | No            | `Option[Timestamp]`        |

Create a Snowplow unstructured event [self-describing JSON](https://github.com/snowplow/iglu/wiki/Self-describing-JSONs):

```scala
import com.snowplowanalytics.iglu.core.{SchemaKey, SchemaVer}
import io.circe.Json

val productViewEvent = SelfDescribingJson(
  SchemaKey("com.acme", "product_view", "jsonschema", SchemaVer(1,0,0)),
  Json.obj(
    "sku" := "0000013"
  )
)
```

Send it using the `trackSelfDescribingEvent` tracker method:

```scala
tracker.trackSelfDescribingEvent(productViewEvent)
```

You can attach any number of custom contexts to an event:

```scala
val pageTypeContext = SelfDescribingJson(
  SchemaKey("com.acme", "page_type", "jsonschema", SchemaVer(1,0,0)),
  Json.obj(
    "type" := "promotional",
    "backgroundColor" := "red"
  )
)

val userContext = SelfDescribingJson(
  SchemaKey("com.acme", "user", "jsonschema", SchemaVer(1,0,0)),
  Json.obj(
    "userType" := "tester"
  )
)

t.trackSelfDescribingEvent(productViewEvent, List(pageTypeContext, userContext))
```

## `trackStructEvent`

Use `trackStructEvent` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required).

| **Argument** | **Description**                                                  | **Required?** | **Type**                   |
| ------------ | ---------------------------------------------------------------- | ------------- | -------------------------- |
| `category`   | The grouping of structured events which this `action` belongs to | Yes           | `String`                   |
| `action`     | Defines the type of user interaction which this event involves   | Yes           | `String`                   |
| `label`      | A string to provide additional dimensions to the event data      | No            | `Option[String]`           |
| `property`   | A string describing the object or the action performed on it     | No            | `Option[String]`           |
| `value`      | A value to provide numerical data about the event                | No            | `Option[Double]`           |
| `contexts`   | List of custom contexts for the event                            | No            | `List[SelfDescribingJson]` |
| `timestamp`  | Device created timestamp or true timestamp                       | No            | `Option[Timestamp]`        |

Example:

```scala
val pageTypeContext = SelfDescribingJson(
  SchemaKey("com.acme", "page_type", "jsonschema", SchemaVer(1,0,0)),
  Json.obj(
    "type" := "promotional",
    "backgroundColor" := "red"
  )
)

val userContext = SelfDescribingJson(
  SchemaKey("com.acme", "user", "jsonschema", SchemaVer(1,0,0)),
  Json.obj(
    "userType" := "tester"
  )
)

t.trackStructEvent("commerce", "order", property=Some("book"), contexts=List(pageTypeContext, userContext))
```

## `trackPageView`

Use `trackPageView` to track a user viewing a page within your app. Arguments are:

| **Argument** | **Description**                      | **Required?** | **Validation**             |
| ------------ | ------------------------------------ | ------------- | -------------------------- |
| `pageUrl`    | The URL of the page                  | Yes           | `String`                   |
| `pageTitle`  | The title of the page                | No            | `Option[String]`           |
| `referrer`   | The address which linked to the page | No            | `Option[String]`           |
| `contexts`   | Custom contexts for the event        | No            | `List[SelfDescribingJson]` |
| `timestamp`  | When the pageview occurred           | No            | `Option[Timestamp]`        |

Example:

```scala
t.trackPageView("www.example.com", Some("example"), Some("www.referrer.com"))
```

## `trackError`

Use `trackError` to track exceptions raised during your app's execution. Arguments are:

| **Argument** | **Description**                  | **Required?** | **Validation**             |
| ------------ | -------------------------------- | ------------- | -------------------------- |
| `error`      | Any throwable need to be tracked | Yes           | `Throwable`                |
| `contexts`   | Custom contexts for the event    | No            | `List[SelfDescribingJson]` |
| `timestamp`  | When the pageview occurred       | No            | `Option[Timestamp]`        |

Example:

```scala
try {
  1 / 0
} catch {
  case e: java.lang.ArithmeticException =>
    t.trackError(e)
}
```

Note: this tracker should not be used to track exceptions happening in tracker itself, use [callbacks](/docs/sources/scala-tracker/initialization/#callbacks) mechanism for that.

## `trackAddToCart`

Use `trackAddToCart` to track an add-to-cart event.

## `trackRemoveFromCart`

Use `trackRemoveFromCart` to track a remove-from-cart event.

## `trackTransaction`

Use `trackTransaction` to record view of transaction. Fire a `trackTransaction` to register the transaction, and then fire `trackTransactionItem` to log specific data about the items that were part of that transaction.

## `trackTransactionItem`

To track an ecommerce transaction item. Fire a `trackTransaction` to register the transaction, and then fire `trackTransactionItem` to log specific data about the items that were part of that transaction.

## Setting timestamp

By default, Scala Tracker will generate a `dvce_created_tstamp` and add it to event payload. You also can manually set it using `timestamp` argument in all tracking methods. It should be in milliseconds since the Unix epoch:

```scala
tracker.trackSelfDescribingEvent(productViewEvent, Nil, Some(1432806619000L))
```

Beside of it, you can set `true_tstamp` if you have more reliable source about event timestamp. You can tag timstamp as "true" using class `TrueTimestamp`:

```scala
tracker.trackSelfDescribingEvent(productViewEvent, Nil, Some(Tracker.TrueTimestamp(1432806619000L)))
```

Now event will be sent with `ttm` parameter instead of `dtm`.

## Setting the subject

By default, each event is associated with the (optional) `Subject` defined on the tracker instance. You can also send a `Subject` for each individual event, which overrides the default Subject.

```scala
val subject = new Subject()
  .setUserId("user-00035")
  .setPlatform(Desktop)
t.trackPageView("www.example.com", subject = Some(subject))
```

---

# Install the Scala tracker
> Install the Snowplow Scala tracker from Maven Central or JCenter using sbt, Gradle, or Maven with support for http4s and id emitters.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/setup/

The Tracker is published to Maven Central and JCenter, which should make it easy to add it as a dependency into your own Scala app.

**sbt:**

Add the Scala Tracker to your build.sbt:

```scala
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-core" % "2.0.0"
// If you plan to use the http4s emitter with an Ember client
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-emitter-http4s" % "2.0.0"
libraryDependencies += "org.http4s" %% "http4s-ember-client" % "0.23.15"
// If you plan to use the id emitters:
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-emitter-id" % "2.0.0"
// If you plan to use EC2/GCE contexts:
libraryDependencies += "com.snowplowanalytics" %% "snowplow-scala-tracker-metadata" % "2.0.0"
```

**Gradle:**

Add our Maven repository in your `build.gradle` file:

```gradle
repositories {
  ...
  jcenter()
}
```

Then add into the same file:

```gradle
dependencies {
  ...
  // Snowplow Scala Tracker
  compile 'com.snowplowanalytics:snowplow-scala-tracker-core_2.13:2.0.0'
  // If you plan to use the http4s emitters with an Ember client
  compile 'com.snowplowanalytics:snowplow-scala-tracker-emitter-http4s_2.13:2.0.0'
  compile 'org.http4s:http4s-ember-client_2.13:0.23.15'
  // If you plan to use the id emitters
  compile 'com.snowplowanalytics:snowplow-scala-tracker-emitter-id_2.13:2.0.0'
  // If you plan to use EC2/GCE contexts:
  compile 'com.snowplowanalytics:snowplow-scala-tracker-metadata_2.13:2.0.0'
}
```

**Maven:**

Add into your project's `pom.xml`:

```maven
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-scala-tracker-core_2.13</artifactId>
  <version>2.0.0</version>
</dependency>
<!-- If you plan to use the http4s emitter with an Ember client: -->
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-scala-tracker-emitter-http4s_2.13</artifactId>
  <version>2.0.0</version>
</dependency>
<dependency>
  <groupId>org.http4s</groupId>
  <artifactId>http4s-ember-client</artifactId>
  <version>0.23.15</version>
</dependency>
<!-- If you plan to use the id emitters: -->
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-scala-tracker-emitter-id_2.13</artifactId>
  <version>2.0.0</version>
</dependency>
<!-- If you plan to use EC2/GCE contexts: -->
<dependency>
  <groupId>com.snowplowanalytics</groupId>
  <artifactId>snowplow-scala-tracker-metadata_2.13</artifactId>
  <version>2.0.0</version>
</dependency>
```

***

> **Info:** Notice a `_2.13` postfix in artifactId. This is used for Scala libraries and denotes the Scala version which the artifact (in our case `snowplow-scala-tracker`) is compiled against. It also means that this library will bring a `org.scala-lang:scala-library_2.13.x` as transitive dependency and if you're using any other Scala dependency you should keep these postfixes in accordance (`snowplow-scala-tracker` is also compiled against Scala 2.12).

---

# Configure a subject with the Scala tracker
> Configure Scala tracker subject with platform, user ID, screen resolution, viewport, timezone, language, IP address, and domain user ID properties.
> Source: https://docs.snowplow.io/docs/sources/scala-tracker/subject-methods/

A list of the methods used to add data to the Subject class.

### Set the platform with `setPlatform`

The default platform is `Server`. These are the available alternatives, all available in the package `com.snowplowanalytics.snowplow.scalatracker`:

- Server
- Web
- Mobile
- Desktop
- Tv
- Console
- InternetOfThings
- General

Example usage

```scala
subject.setPlatform(Tv)
```

### Set the user ID with `setUserId`

You can make the user ID a string of your choice:

```scala
subject.setUserId("user-000563456")
```

### Set the screen resolution with `setScreenResolution`

If your Scala code has access to the device's screen resolution, you can pass it in to Snowplow. Both numbers should be positive integers; note the order is width followed by height. Example:

```scala
subject.setScreenResolution(1366, 768)
```

### Set the viewport dimensions with `setViewport`

Similarly, you can pass the viewport dimensions in to Snowplow. Again, both numbers should be positive integers and the order is width followed by height. Example:

```scala
subject.setViewport(300, 200)
```

### Set the color depth with `setColorDepth`

If your Scala code has access to the bit depth of the device's color palette for displaying images, you can pass it in to Snowplow. The number should be a positive integer, in bits per pixel.

```scala
subject.setColorDepth(24)
```

### Setting the timezone with `setTimezone`

If your Scala code has access to the timezone of the device, you can pass it in to Snowplow:

```scala
subject.setTimezone("Europe London")
```

### Setting the language with `setLang`

You can set the language field like this:

```scala
subject.setLang("en")
```

### Setting the IP address with `setIpAddress`

If you have access to the user's IP address, you can set it like this:

```scala
subject.setIpAddresss("34.634.11.139")
```

### Setting the useragent with `setUseragent`

If you have access to the user's useragent (sometimes called "browser string"), you can set it like this:

```scala
subject.setUseragent("Mozilla/5.0 (Windows NT 5.1; rv:24.0) Gecko/20100101 Firefox/24.0")
```

### Setting the domain user ID with `setDomainUserId`

The `domain_userid` field of the Snowplow event model corresponds to the ID stored in the first party cookie set by the Snowplow JavaScript Tracker. If you want to match up server-side events with client-side events, you can set the domain user ID for server-side events like this:

```scala
subject.setDomainUserId("c7aadf5c60a5dff9")
```

### Setting the network user ID with `setNetworkUserId`

The `network_user_id` field of the Snowplow event model corresponds to the ID stored in the third party cookie set by the Snowplow Clojure Collector and Scala Stream Collector. You can set the network user ID for server-side events like this:

```scala
subject.setNetworkUserId("ecdff4d0-9175-40ac-a8bb-325c49733607")
```

---

# Additional information for the Tracking CLI
> Learn about Tracking CLI implementation details including Golang tracker backend, no buffering behavior, and collector response exit codes.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/additional-information/

There is no buffering in the Snowplow Tracking CLI - each event is sent as an individual payload whether `GET` or `POST`.

Under the hood, the app uses the [**Snowplow Golang Tracker**](https://github.com/snowplow/snowplow-golang-tracker).

The Snowplow Tracking CLI will exit once the Snowplow collector has responded. The return codes from `snowplowtrk` are as follows:

- 0 if the Snowplow collector responded with an OK status (2xx or 3xx)
- 4 if the Snowplow collector responded with a 4xx status
- 5 if the Snowplow collector responded with a 5xx status
- 1 for any other error

---

# Tracking CLI
> Send Snowplow events from the command line with the native Tracking CLI app for shell scripts and terminal sessions.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/

The Snowplow Tracking CLI is a native app to make it easy to send an event to Snowplow from the command line. Use this to embed Snowplow tracking into your shell scripts and terminal sessions.

---

# Install the Snowplow Tracking CLI
> Download Snowplow Tracking CLI binaries for Linux, macOS, and Windows, or use the Docker container for command-line event tracking.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/installation/

You can download the binary for Linux, macOS and Windows directly from GitHub:

- [**Linux AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_linux_amd64.zip)
- [**Windows AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_windows_amd64.zip)
- [**macOS AMD 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_darwin_amd64.zip)
- [**macOS ARM 64bit Binary**](https://github.com/snowplow/snowplow-tracking-cli/releases/download/0.7.0/snowplow_tracking_cli_0.7.0_darwin_arm64.zip)

There is also a [Docker Container available](https://hub.docker.com/r/snowplow/snowplow-tracking-cli) which can be used in lieu of installing the binary directly.

---

# Snowplow Tracking CLI interface
> Track events from command line with Tracking CLI using self-describing JSON or schema URI with collector endpoint, app ID, and HTTP method options.
> Source: https://docs.snowplow.io/docs/sources/snowplow-tracking-cli/usage/

The command line interface is as follows:

```bash
snowplow-tracking-cli --collector={{COLLECTOR_DOMAIN}} --appid={{APP_ID}} --method=[POST|GET] --sdjson={{SELF_DESC_JSON}}
```

or:

```bash
snowplow-tracking-cli --collector={{COLLECTOR_DOMAIN}} --appid={{APP_ID}} --method=[POST|GET] --schema={{SCHEMA_URI}} --json={{JSON}}
```

where:

- `--collector` is the domain for your Snowplow collector, e.g. `snowplow-collector.acme.com`
- `--appid` is optional (not sent if not set)
- `--method` is optional. It defaults to `GET`
- `--protocol` is optional. It defaults to `https`
- `--sdjson` is a self-describing JSON of the standard form `{ "schema": "iglu:...", "data": { ... } }`
- `--schema` is a schema URI, most likely of the form `iglu:...`
- `--json` is a (non-self-describing) JSON, of the form `{ ... }`
- `--ipaddress` is optional. It defaults to an empty string
- `--contexts` is optional. It defaults to an empty JSON array `[]`

The idea here is that you can either send in a [**self-describing JSON**](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/), or pass in the constituent parts (i.e. a regular JSON plus a schema URI) and the Snowplow Tracking CLI will construct the final self-describing JSON for you.

## Examples

```bash
snowplow-tracking-cli --collector snowplow-collector.acme.com --appid myappid --method POST --schema iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0 --json "{\"hello\":\"world\"}"
```

```bash
snowplow-tracking-cli --collector snowplow-collector.acme.com --appid myappid --method POST --sdjson "{\"schema\":\"iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0\", \"data\":{\"hello\":\"world\"}}"
```

## Usage with Docker

```bash
docker run snowplow/snowplow-tracking-cli:latest --collector snowplow-collector.acme.com --appid myappid --method POST --sdjson "{\"schema\":\"iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0\", \"data\":{\"hello\":\"world\"}}"
```

---

# Tracker maintenance classification
> Understand Snowplow tracker maintenance groups including actively maintained, maintained, early release, and unsupported status with production readiness and update schedules.
> Source: https://docs.snowplow.io/docs/sources/tracker-maintenance-classification/

Snowplow Trackers and dbt packages have been categorized into 4 maintenance groups. These groups are designed to give some expectations into how they will maintained in the future. Each tracker GitHub repository contains a "Snowplow" Badge that shows the maintenance group for that specific tracker and links back to this page.

***

Not production-ready, not actively maintained. Deprecated.

***

Pre-Version 1.0.0, may be missing core tracker features. Accepting community contributions. Snowplow Team will test and release contributions. Snowplow Team will update to fix critical or high severity vulnerabilities. Will be released less frequently.

***

Currently production ready. Maintained by the Snowplow team but released less frequently. Accepting community contributions. May not have up to date dependencies or support for latest versions of language or framework. Scanned daily for vulnerabilities, Snowplow Team will fix critical or high severity vulnerabilities promptly.

***

Currently production ready. Actively maintained by the Snowplow team, with a frequent release schedule. Accepting community contributions. Dependencies are actively updated and will support the latest versions of language or framework. Scanned daily for vulnerabilities, Snowplow Team will fix all vulnerabilities promptly.

***

| Status | Production Ready | Release Frequency         | Accepting Contributions | Dependency and Vulnerability Updates                                                                            |
| ------ | ---------------- | ------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------- |
|        | ❌                | Never                     | ❌                       | None                                                                                                            |
|        | Varies           | As required               | ✔️                      | - Updates for critical and high severity vulnerabilities                                                        |
|        | ✔️               | As required Low frequency | ✔️                      | - Daily scans - Passive updates to dependencies - Active updates for critical and high severity vulnerabilities |
|        | ✔️               | Frequent release schedule | ✔️                      | - Daily scans - Active updates to dependencies and all levels of vulnerabilities                                |

---

# Configure emitters in the Unity tracker
> Configure sync, async, and WebGL emitters to send and store Unity tracker events with customizable protocols, methods, and batch sizes.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/emitter/

The Emitter object is responsible for sending and storing all events.

We provide the following emitters available currently:

- `SyncEmitter` : Slow blocking synchronous operation, useful for testing but should not be used in production.
- `AsyncEmitter` : Fully asynchronous operation which uses the ThreadPool to perform all of its operations.
- `WebGlEmitter` : Emitter implementing the async API that is compatible with WebGL.

### Constructor

| **Argument Name** | **Description**                                             | **Required?** | **Default**                                                                                      |
| ----------------- | ----------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------ |
| `endpoint`        | The collector uri the emitter should use                    | Yes           | Null                                                                                             |
| `protocol`        | The request Protocol (HTTP or HTTPS)                        | No            | HTTP                                                                                             |
| `method`          | The HTTP Method (GET or POST)                               | No            | POST                                                                                             |
| `sendLimit`       | The amount of events to send at a time                      | No            | 500 (AsyncEmitter) / 10 (SyncEmitter) / 1 (WebGlEmitter)                                         |
| `byteLimitGet`    | The byte limit for a GET request                            | No            | 52000                                                                                            |
| `byteLimitPost`   | The byte limit for a POST request                           | No            | 52000                                                                                            |
| `eventStore`      | Implementation for storing queued events waiting to be sent | No            | `InMemoryEventStore` on tvOS, otherwise `EventStore` (on-disk [LiteDB](https://www.litedb.org/)) |

A full Emitter construction should look like the following:

```csharp
IEmitter e1 = new AsyncEmitter ("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.GET, 50, 30000, 30000, null);
```

All except `eventStore` of these variables can be altered after creation with the accompanying `emitter.SetXXX()` function. However do be aware that multiple threads will be accessing these variables so to be safe always shut the Tracker down using `StopEventTracking()` before amending anything.

**NOTE:** Be aware that when sending events via `GET` all events will be sent individually. This means that if your `sendLimit` is 500 there is the potential for 500 Threads to be spawned at the same time which can cause serious performance issues. To alleviate this concern simply drop your `sendLimit` to a more manageable range.

Some load balancers limit the maximum URL length accepted for `GET` requests, often around 16KB. The default of 52000 bytes is considerably larger than these limits, but most single event payloads will not reach this size.

---

# Track events with the Unity tracker
> Track page views, screen views, structured events, timing events, self-describing events, and ecommerce transactions in Unity applications.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/event-tracking/

You track events by building an event payload using event-specific Builder objects, and passing the result to a tracker instance's `Track()` method. Your events can have common fields configured with the respective `SetCustomContexts()`, `SetTimestamp()`, and `SetEventId()` methods, in addition to event-specific fields covered below.

## Track page views with `Track(PageView)`

You can use `Track(PageView)` to track a user viewing a web page within your app.

Arguments are:

| **Argument**     | **Description**                      | **Required?** | **Type**         |
| ---------------- | ------------------------------------ | ------------- | ---------------- |
| `pageUrl`        | The URL of the page                  | Yes           | `string`         |
| `pageTitle`      | The title of the page                | No            | `string`         |
| `referrer`       | The address which linked to the page | No            | `string`         |
| `customContexts` | Optional custom context              | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                   | No            | `long`           |
| `eventId`        | Optional custom event id             | No            | `string`         |

Examples:

```csharp
t1.Track(new PageView()
    .SetPageUrl("www.example.com")
    .SetPageTitle("example")
    .SetReferrer("www.referrer.com")
    .Build());

t1.Track(new PageView()
    .SetPageUrl("www.example.com")
    .SetPageTitle("example")
    .SetReferrer("www.referrer.com")
    .SetCustomContext(contextList)
    .SetTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

## Track screen views with `Track(MobileScreenView)`

Use `Track(MobileScreenView)` to track a user viewing a screen (or equivalent) within your app. You **must** provide a `name` property. The `id` of the screen view will be automatically assigned (as an UUID) but you may also provide it manually. Arguments are:

| **Argument**     | **Description**                                                | **Required?** | **Type**         |
| ---------------- | -------------------------------------------------------------- | ------------- | ---------------- |
| `name`           | Human-readable name for this screen                            | Yes           | `string`         |
| `id`             | Unique identifier for this screen view (can be auto-generated) | Yes           | `string`         |
| `type`           | The type of screen that was viewed e.g feed / carousel         | No            | `string`         |
| `previousName`   | The name of the previous screen                                | No            | `string`         |
| `previousId`     | The screen view ID of the previous screen view                 | No            | `string`         |
| `previousType`   | The screen type of the previous screen view                    | No            | `string`         |
| `previousType`   | The screen type of the previous screen view                    | No            | `string`         |
| `customContexts` | Optional custom context                                        | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                                             | No            | `long`           |
| `eventId`        | Optional custom event id                                       | No            | `string`         |

Examples:

```csharp
t1.Track(new MobileScreenView("HUD > Save Game")
    .SetType("dialog")
    .SetPreviousName("HUD > Menu")
    .SetPreviousId(previousScreenId)
    .SetTransitionType("pop-up")
    .Build());

t1.Track(new MobileScreenView("HUD > Save Game")
    .SetCustomContext(contextList)
    .SetTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

> **Note:** In tracker versions 0.7.0 and earlier, screen views were tracked using the `ScreenView` class. However, this class used the older schema for screen views and has been deprecated in favour of the `MobileScreenView` in the 0.8.0 release of the tracker.

## Track structured events with `Track(Structured)`

Use `Track(Structured)` to track a custom event happening in your app which fits the Google Analytics-style structure of having up to five fields (with only the first two required):

| **Argument**     | **Description**                                                  | **Required?** | **Type**         |
| ---------------- | ---------------------------------------------------------------- | ------------- | ---------------- |
| `category`       | The grouping of structured events which this `action` belongs to | Yes           | `string`         |
| `action`         | Defines the type of user interaction which this event involves   | Yes           | `string`         |
| `label`          | A string to provide additional dimensions to the event data      | No            | `string`         |
| `property`       | A string describing the object or the action performed on it     | No            | `string`         |
| `value`          | A value to provide numerical data about the event                | No            | `double`         |
| `customContexts` | Optional custom context                                          | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                                               | No            | `long`           |
| `eventId`        | Optional custom event id                                         | No            | `string`         |

Examples:

```csharp
t1.Track(new Structured()
    .SetCategory("shop")
    .SetAction("add-to-basket")
    .Build());

t1.Track(new Structured()
    .SetCategory("shop")
    .SetAction("add-to-basket")
    .SetLabel("Add To Basket")
    .SetProperty("pcs")
    .SetValue(2.00)
    .SetCustomContext(contextList)
    .SetTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

## Track timing events with `Track(Timing)`

Use `Track(Timing)` to track an event related to a custom timing.

| **Argument**     | **Description**                        | **Required?** | **Type**         |
| ---------------- | -------------------------------------- | ------------- | ---------------- |
| `category`       | The category of the timed event        | Yes           | `string`         |
| `label`          | The label of the timed event           | No            | `string`         |
| `timing`         | The timing measurement in milliseconds | Yes           | `int`            |
| `variable`       | The name of the timed event            | Yes           | `string`         |
| `customContexts` | Optional custom context                | No            | `List<IContext>` |
| `timestamp`      | Optional timestamp                     | No            | `long`           |
| `eventId`        | Optional custom event id               | No            | `string`         |

Examples:

```csharp
t1.Track(new Timing()
    .SetCategory("category")
    .SetVariable("variable")
    .SetTiming(1)
    .Build());

t1.Track(new Timing()
    .SetCategory("category")
    .SetVariable("variable")
    .SetTiming(1)
    .SetLabel("label")
    .SetCustomContext(contextList)
    .SetTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build());
```

## Track self-describing events with `Track(SelfDescribing)`

Custom self-describing events are a flexible tool that enable Snowplow users to define their own event types and send them into Snowplow.

When a user sends in a custom self-describing event, they do so as a JSON of name-value properties, that conforms to a JSON schema defined for the event earlier.

Use `Track(SelfDescribing)` to track a custom event which consists of a name and a self-describing set of properties. This is useful when:

- You want to track event types which are proprietary/specific to your business (i.e. not already part of Snowplow), or
- You want to track events which have unpredictable or frequently changing properties

The arguments are as follows:

| **Argument**     | **Description**             | **Required?** | **Type**                                    |
| ---------------- | --------------------------- | ------------- | ------------------------------------------- |
| `eventData`      | The properties of the event | Yes           | [`SelfDescribingJson`](#selfdescribingjson) |
| `customContexts` | Optional custom context     | No            | `List<IContext>`                            |
| `timestamp`      | Optional timestamp          | No            | `long`                                      |
| `eventId`        | Optional custom event id    | No            | `string`                                    |

Example event json to track:

```json
{
  "schema": "iglu:com.acme/save_game/jsonschema/1-0-0",
  "data": {
    "levelName": "Barrels o' Fun",
    "levelIndex": 23
  }
}
```

How to set it up?

```json
// Create a Dictionary of your event data
Dictionary<string, object> eventDict = new Dictionary<string, object>();
eventDict.Add("levelName", "Barrels o' Fun");
eventDict.Add("levelIndex", 23);

// Track your event with your custom event data
t1.Track(new SelfDescribing("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict)
    .Build();

// OR

t1.Track(new SelfDescribing("iglu:com.acme/save_game/jsonschema/1-0-0", eventDict)
    .SetCustomContext(contextList)
    .SetTimestamp(1423583655000)
    .SetEventId("uid-1")
    .Build();
```

For more on JSON schema, see the [blog post](https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/).

> **Note:** In versions 0.7.0 and earlier, the `Unstructured` class was used to track self-describing events. This was deprecated in version 0.8.0 in favour of the `SelfDescribing` class.

## Track ecommerce transactions with `Track(EcommerceTransaction)`

Use `Track(EcommerceTransaction)` to track an ecommerce transaction.

Arguments:

| **Argument**     | **Description**                 | **Required?** | **Type**                         |
| ---------------- | ------------------------------- | ------------- | -------------------------------- |
| `orderId`        | ID of the eCommerce transaction | Yes           | `string`                         |
| `totalValue`     | Total transaction value         | Yes           | `double`                         |
| `affiliation`    | Transaction affiliation         | No            | `string`                         |
| `taxValue`       | Transaction tax value           | No            | `double`                         |
| `shipping`       | Delivery cost charged           | No            | `double`                         |
| `city`           | Delivery address city           | No            | `string`                         |
| `state`          | Delivery address state          | No            | `string`                         |
| `country`        | Delivery address country        | No            | `string`                         |
| `currency`       | Transaction currency            | No            | `string`                         |
| `items`          | Items in the transaction        | Yes           | `List<EcommerceTransactionItem>` |
| `customContexts` | Optional custom context         | No            | `List<IContext>`                 |
| `timestamp`      | Optional timestamp              | No            | `long`                           |
| `eventId`        | Optional custom event id        | No            | `string`                         |

The `items` argument is a `List` of individual `EcommerceTransactionItem` elements representing the items in the e-commerce transaction. Note that `Track(EcommerceTransaction)` fires multiple events: one transaction event for the transaction as a whole, and one transaction item event for each element of the `items` `List`.

Each transaction item event will have the same timestamp, orderId, and currency as the main transaction event.

#### `EcommerceTransactionItem`

To instantiate a `EcommerceTransactionItem` in your code, simply use the following constructor signature:

```json
EcommerceTransactionItem item = new EcommerceTransactionItem ()
      .SetSku ("sku")
      .SetPrice (10.2)
      .SetQuantity (1)
      .SetName ("name")
      .SetCategory ("category")
      .Build ()
```

These are the fields that can appear as elements in each `EcommerceTransactionItem` element of the transaction item's `List`:

| **Field**        | **Description**          | **Required?** | **Type**         |
| ---------------- | ------------------------ | ------------- | ---------------- |
| `sku`            | Item SKU                 | Yes           | `string`         |
| `price`          | Item price               | Yes           | `double`         |
| `quantity`       | Item quantity            | Yes           | `int`            |
| `name`           | Item name                | No            | `string`         |
| `category`       | Item category            | No            | `string`         |
| `customContexts` | Optional custom context  | No            | `List<IContext>` |
| `eventId`        | Optional custom event id | No            | `string`         |

Example of tracking a transaction containing two items:

```json
// Create some Transaction Items
EcommerceTransactionItem item1 = new EcommerceTransactionItem ()
    .SetSku ("item_sku_1")
    .SetPrice (10.2)
    .SetQuantity (1)
    .SetName ("item_name_1")
    .SetCategory ("item_category")
    .Build ();

EcommerceTransactionItem item2 = new EcommerceTransactionItem()
    .SetSku("item_sku_2")
    .SetPrice(1.00)
    .SetQuantity(1)
    .SetName("item_name_2")
    .SetCategory("item_category")
    .Build();

// Add these items to a List
List<EcommerceTransactionItem> items = new List<EcommerceTransactionItem>();
items.Add(item1);
items.Add(item2);

// Now Track the Transaction by using this list of items as an argument
tracker.Track(new EcommerceTransaction()
    .SetOrderId("order_id_1")
    .SetTotalValue(300.00)
    .SetAffiliation("my_affiliate")
    .SetTaxValue(30.00)
    .SetShipping(10.00)
    .SetCity("Boston")
    .SetState("Massachusetts")
    .SetCountry("USA")
    .SetCurrency("USD")
    .SetItems(items)
    .Build());
```

## Custom Contexts

Custom contexts are Self Describing Jsons with extra descriptive information that can be optionally attached to any Snowplow event with `SetCustomContexts(...)`. We provide several builders for Snowplow custom contexts as well as a generic builder if you wish to define and send your own custom context!

For ease of development you are also able to extend the `IContext` interface or the `AbstractContext` class for your own contexts if you so wish.

All of these contexts will need to be combined into a `List<IContext>` before being attachable to Snowplow Events.

### `DesktopContext`

The following arguments can be used in a DesktopContext:

| **Field**            | **Description**                     | **Required?** | **Type** |
| -------------------- | ----------------------------------- | ------------- | -------- |
| `osType`             | The Operating System Type           | Yes           | `string` |
| `osVersion`          | The Version of the Operating System | Yes           | `string` |
| `osServicePack`      | Service Pack information            | No            | `string` |
| `osIs64Bit`          | If the OS is 32 or 64 bit           | No            | `bool`   |
| `deviceManufacturer` | Who made the device                 | No            | `string` |
| `deviceModel`        | What is the device model            | No            | `string` |
| `processorCount`     | How many cores does the device have | No            | `int`    |

An example of a DesktopContext construction:

```json
DesktopContext context = new DesktopContext ()
    .SetOsType("OS-X")
    .SetOsVersion("10.10.5")
    .SetOsServicePack("Yosemite")
    .SetOsIs64Bit(true)
    .SetDeviceManufacturer("Apple")
    .SetDeviceModel("Macbook Pro")
    .SetDeviceProcessorCount(4)
    .Build ();
```

### `MobileContext`

The following arguments can be used in a MobileContext:

| **Field**            | **Description**                     | **Required?** | **Type**      |
| -------------------- | ----------------------------------- | ------------- | ------------- |
| `osType`             | The Operating System Type           | Yes           | `string`      |
| `osVersion`          | The Version of the Operating System | Yes           | `string`      |
| `deviceManufacturer` | Who made the device                 | Yes           | `string`      |
| `deviceModel`        | What is the device model            | Yes           | `string`      |
| `carrier`            | The name of the carrier             | No            | `string`      |
| `networkType`        | The type of network                 | No            | `NetworkType` |
| `networkTechnology`  | The networks technlogy              | No            | `string`      |
| `openIdfa`           | An OpenIDFA UUID                    | No            | `string`      |
| `appleIdfa`          | An Apple IDFA UUID                  | No            | `string`      |
| `appleIdfv`          | An Apple IDFV UUID                  | No            | `string`      |
| `androidIdfa`        | An Android IDFA UUID                | No            | `string`      |

An example of a MobileContext construction:

```json
MobileContext context = new MobileContext ()
    .SetOsType("iOS")
    .SetOsVersion("9.0")
    .SetDeviceManufacturer("Apple")
    .SetDeviceModel("iPhone 6S+")
    .SetCarrier("FREE")
    .SetNetworkType(NetworkType.Mobile)
    .SetNetworkTechnology("LTE")
    .Build ();
```

### `GeoLocationContext`

The following arguments can be used in a GeoLocationContext:

| **Field**                   | **Description**            | **Required?** | **Type** |
| --------------------------- | -------------------------- | ------------- | -------- |
| `latitude`                  | The user latitude          | Yes           | `double` |
| `longitude`                 | The user longitude         | Yes           | `double` |
| `latitudeLongitudeAccuracy` | The user lat-long accuracy | No            | `double` |
| `altitude`                  | The user altitude          | No            | `double` |
| `altitudeAccuracy`          | The user alt accuracy      | No            | `double` |
| `bearing`                   | The user bearing           | No            | `double` |
| `speed`                     | The user speed             | No            | `double` |
| `timestamp`                 | A timestamp in ms          | No            | `long`   |

An example of a GeoLocationContext construction:

```json
GeoLocationContext context = new GeoLocationContext ()
    .SetLatitude(123.564)
    .SetLongitude(-12.6)
    .SetLatitudeLongitudeAccuracy(5.6)
    .SetAltitude(5.5)
    .SetAltitudeAccuracy(2.1)
    .SetBearing(3.2)
    .SetSpeed(100.2)
    .SetTimestamp(1234567890000)
    .Build ();
```

### `GenericContext`

The GenericContext is a simple builder with three functions:

- `SetSchema(string)` : Sets the Context Schema Path
- `Add(string, object)` : Adds a single key-pair value to the data packet of this context
- `AddDict(string, object)` : Adds a dictionary of key-pair values to the data packet

You must set a schema string or a RuntimeException will be thrown.

An example of a GenericContext construction:

```json
GenericContext context = new GenericContext()
    .SetSchema("iglu:com.acme/acme_context/jsonschema/1-0-0")
    .Add("context", "custom")
    .Build();
```

## SelfDescribingJson

A `SelfDescribingJson` is used as a wrapper around a `Dictionary<string, object>`. After creating the Dictionary you want to wrap you can create a `SelfDescribingJson` using the following:

```json
// Data as a Dictionary
Dictionary<string, object> data = new Dictionary<string, object>();
data.Add("Event", "Data")

// We then create a new SelfDescribingJson
SelfDescribingJson json = new SelfDescribingJson("iglu:com.acme/example/jsonschema/1-0-0", data);
```

This object is now ready to be Tracked within a SelfDescribing Event.

You can create a SelfDescribingJson with the following arguments:

| **Argument** | **Description**                           | **Required?** | **Type**                    |
| ------------ | ----------------------------------------- | ------------- | --------------------------- |
| `schema`     | JsonSchema that describes the data        | Yes           | `string`                    |
| `data`       | Data that will be validated by the schema | No            | `Dictionary<string,object`> |

---

# Unity tracker SDK
> Track Snowplow events from Unity games and apps across multiple platforms including iOS, Android, WebGL, and desktop.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/

The Snowplow Unity Tracker allows you to track Snowplow events from your Unity games and apps. The current tracker version is 0.8.1.

### Snowplow Icebreaker Demo

To see the Tracker in action you can download our demonstration game Snowplow Icebreaker from [Github](https://github.com/snowplow/snowplow-unity-tracker) and loading the SnowplowTracker.Demo folder into Unity 2022.3. You will need to set a valid collector endpoint on the Main Menu Page to see the events being generated.

---

# Initialize the Unity tracker
> Import required libraries and create tracker instances with emitters in Unity projects across platforms including WebGL.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/initialization/

Assuming you have completed the Unity Tracker Setup for your project, you are now ready to initialize the Unity Tracker.

### Importing the library

Add the following `using` lines to the top of your `.cs` scripts to access the Tracker:

```csharp
using SnowplowTracker;
using SnowplowTracker.Emitters;
using SnowplowTracker.Enums;
using SnowplowTracker.Events;
```

You should now be able to setup the Tracker!

### Creating a tracker

To instantiate a Tracker in your code (can be global or local to the process being tracked) simply instantiate the Tracker interface with the following:

```csharp
// Create Emitter – to support WebGL, make sure to use the WebGlEmitter
IEmitter emitter = Application.platform == RuntimePlatform.WebGLPlayer
        ? new WebGlEmitter("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.POST)
        : new AsyncEmitter("com.collector.acme", HttpProtocol.HTTPS, HttpMethod.POST);

// Create a Tracker instance with the Emitter
Tracker tracker = new Tracker(emitter, "Namespace", "AppId");

// Start the Tracker
tracker.StartEventTracking();
```

---

# Track session data with the Unity tracker
> Manage user session data with configurable foreground and background timeouts across Unity platforms with persistent storage.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/session/

The Session object is responsible for maintaining persistent data around user sessions over the life-time of an application.

Session information is persisted using file storage on most platforms except for tvOS and WebGL. On those two platforms, session information is stored in [`PlayerPrefs`](https://docs.unity3d.com/ScriptReference/PlayerPrefs.html).

### Constructor

| **Argument Name**   | **Description**                                        | **Required?**      | **Default**              |
| ------------------- | ------------------------------------------------------ | ------------------ | ------------------------ |
| `sessionPath`       | The file path to store the persistent session state in | Yes (accepts null) | "snowplow\_session.dict" |
| `foregroundTimeout` | The time until a session expires in focus              | No                 | 600 (s)                  |
| `backgroundTimeout` | The time until a session expires in back               | No                 | 300 (s)                  |
| `checkInterval`     | How often to validate the session timeout              | No                 | 15 (s)                   |

A full Session construction should look like the following:

```csharp
Session session = new Session ("snowplow_session_state.json", 1200, 600, 30);
```

A minimal construction would be:

```csharp
Session session = new Session (null);
```

The timeouts refer to the length of time the session remains active after the last event is sent. As long as events are sent within this limit the session will not timeout.

### Functions

Except `sessionPath`, all of the constructor variables can be altered/retrieved after creation with the accompanying `session.SetXXX()`/`session.GetXXX()` functions.

#### `SetBackground(bool)`

Will set whether or not the application is in the background. It is up to the developer to set this metric if they wish to have a different timeout for foreground and background.

---

# Install the Unity tracker
> Install the Unity tracker package for Unity 2022.3+ supporting Windows, macOS, Linux, iOS, Android, tvOS, and WebGL platforms.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/setup/

The Snowplow Unity Tracker has been built and tested using Unity 2022.3 on Windows, Linux, OSX, iOS, Android, tvOS, and WebGL. It is built using .NET Standard 2.0 and requires at least Unity 2018.1.

It is not currently compatible with the WebPlayer.

## Dependencies

There are several dependencies that are required to use the Unity Tracker currently. These are available in [this folder](https://github.com/snowplow/snowplow-unity-tracker/tree/master/Resources).

- `UnityEngine.dll` : This is included as a way of setting up development externally to the Unity playground; you do not need to include this.
- `Newtonsoft.Json.dll` : This is included as a way of setting up development externally to the Unity playground; you do not need to include this as it is bundled with Unity 2020+ or should be installed via [Newtonsoft.Json-for-Unity package](https://github.com/jilleJr/Newtonsoft.Json-for-Unity).

All the other dependencies are required for proper Tracker function but they are included inside the `unitypackage`.

## Download

The Unity Tracker package is published to the Unity Trackers [Github releases](https://github.com/snowplow/snowplow-unity-tracker/releases). Once downloaded simply add the `SnowplowTracker.unitypackage` to your project. This should insert all of the required dll files into your `Assets/Plugins/` folder.

---

# Configure a subject with the Unity tracker
> Configure subject properties including user ID, screen resolution, viewport, timezone, and language to attach user context to all tracked events.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/subject/

You may have additional information about your application's environment, current user and so on, which you want to send to Snowplow with each event.

The Subject class has a set of `Set...()` methods to attach extra data relating to the user to all tracked events:

- `SetUserId`
- `SetScreenResolution`
- `SetViewport`
- `SetColorDepth`
- `SetTimezone`
- `SetLanguage`
- `SetIpAddress`
- `SetUseragent`
- `SetNetworkUserId`
- `SetDomainUserId`

Here are some examples:

```csharp
Subject s1 = new Subject();
s1.SetUserId("Kevin Gleason");
s1.SetLanguage("en-gb");
s1.SetScreenResolution(1920, 1080);
```

After that, you can add your Subject to your Tracker like so:

```csharp
Tracker t1 = new Tracker(..., s1)

// OR

t1.SetSubject(s1);
```

### Set user ID with `SetUserId`

You can set the user ID to any string:

```csharp
s1.SetUserId( "{{USER ID}}" )
```

Example:

```csharp
s1.SetUserId("alexd")
```

### Set screen resolution with `SetScreenResolution`

If your C# code has access to the device's screen resolution, then you can pass this in to Snowplow too:

```csharp
t1.SetScreenResolution( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
t1.SetScreenResolution(1366, 768)
```

### Set viewport dimensions with `SetViewport`

If your C# code has access to the viewport dimensions, then you can pass this in to Snowplow too:

```csharp
s.SetViewport( {{WIDTH}}, {{HEIGHT}} )
```

Both numbers should be positive integers; note the order is width followed by height. Example:

```csharp
s.SetViewport(300, 200)
```

### Set color depth with `SetColorDepth`

If your C# code has access to the bit depth of the device's color palette for displaying images, then you can pass this in to Snowplow too:

```csharp
s.SetColorDepth( {{BITS PER PIXEL}} )
```

The number should be a positive integer, in bits per pixel. Example:

```csharp
s.SetColorDepth(32)
```

### Set timezone with `SetTimezone`

This method lets you pass a user's timezone in to Snowplow:

```csharp
s.SetTimezone( {{TIMEZONE}} )
```

The timezone should be a string:

```csharp
s.SetTimezone("Europe/London")
```

### Set the language with `SetLanguage`

This method lets you pass a user's language in to Snowplow:

```csharp
s.SetLanguage( {{LANGUAGE}} )
```

The language should be a string:

```csharp
s.SetLanguage('en')
```

### `SetIpAddress`

This method lets you pass a user's IP Address in to Snowplow:

```csharp
SetIpAddress( {{IP ADDRESS}} )
```

The IP address should be a string:

```csharp
subj.SetIpAddress("127.0.0.1");
```

### `SetUseragent`

This method lets you pass a useragent in to Snowplow:

```csharp
SetUseragent( {{USERAGENT}} )
```

The useragent should be a string:

```csharp
subj.SetUseragent("Agent Smith");
```

### `SetNetworkUserId`

This method lets you pass a Network User ID in to Snowplow:

```csharp
SetNetworkUserId( {{NUID}} )
```

The network user id should be a string:

```csharp
subj.SetNetworkUserId("network-id");
```

### `SetDomainUserId`

This method lets you pass a Domain User ID in to Snowplow:

```csharp
SetDomainUserId( {{DUID}} )
```

The domain user id should be a string:

```csharp
subj.SetDomainUserId("domain-id");
```

---

# Configure a tracker with the Unity tracker
> Configure the Unity tracker instance to coordinate event tracking, manage sessions, and control background processing with customizable platform settings.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/tracker/

The Tracker object is responsible for co-ordinating the saving and sending of events as well as managing the optional Session object.

### Constructor

| **Argument Name**                                 | **Description**                                                          | **Required?** | **Default** |
| ------------------------------------------------- | ------------------------------------------------------------------------ | ------------- | ----------- |
| [`emitter`](/docs/sources/unity-tracker/emitter/) | The Emitter object you create                                            | Yes           | Null        |
| `trackerNamespace`                                | The name of the tracker instance                                         | Yes           | Null        |
| `appId`                                           | The application ID                                                       | Yes           | Null        |
| [`subject`](/docs/sources/unity-tracker/subject/) | The Subject that defines a user                                          | No            | Null        |
| [`session`](/docs/sources/unity-tracker/session/) | The Session object you create                                            | No            | Null        |
| `platform`                                        | The device the Tracker is running on                                     | No            | Mobile      |
| `base64Encoded`                                   | If we [base 64 encode](https://en.wikipedia.org/wiki/Base64) json values | No            | True        |

A full Tracker construction should look like the following:

```csharp
IEmitter e1 = new AsyncEmitter ("com.collector.acme")
Subject subject = new Subject();
Session session = new Session(null);
Tracker t1 = new Tracker(e1, "Namespace", "AppId", subject, session, DevicePlatforms.Desktop, true);
```

All of these variables can be altered after creation with the accompanying `tracker.SetXXX()` function.

### Functions

The Tracker also contains several critical functions that must be used to start Tracking.

#### `StartEventTracking()`

This function must be called before any events will start being stored or sent. This is due to the fact that we do not want to start any background processing from the constructors so it is left up to the developer to choose when to start everything up.

This function:

- Starts the background emitter thread
- Starts the background event processor thread
- Starts the background session check timer (Optional)

Once this is run everything should be in place for asynchronous event tracking.

#### `StopEventTracking()`

If you need to temporarily halt the Tracker from tracking events you can run this function. This will bring all event processing, sending and collection to a halt and nothing will be started again until `StartEventTracking()` is fired again.

#### `Track(IEvent)`

This is the function used for Tracking all events.

```csharp
tracker.Track(IEvent newEvent);
```

---

# Unity tracker utilities
> Use logging utilities and thread-safe queue implementations for debugging and managing Unity tracker operations.
> Source: https://docs.snowplow.io/docs/sources/unity-tracker/utilities/

The Tracker also provides several extra utilities that can be used.

## Log

There is a custom logging wrapper which allows you to control the level of Tracker logging that occurs. You can set the level via:

```csharp
Log.SetLogLevel({0,1,2 or 3});
```

- `0` : Turns of all logging
- `1` : Error only
- `2` : Error and Debug
- `3` : Error, Debug and Verbose

## 8.2 ConcurrentQueue

Due to the .NET 2.0 limitations in earlier versions of the Snowplow Unity Tracker we have had to implement our own ThreadSafe queue which can be found in the following package:

- `SnowplowTracker.Collections`

---

# Opt-outs and anonymous tracking on web
> Capture Snowplow web events without user or session identifiers, without IP addresses and without setting any cookies.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/anonymous-tracking/

The Snowplow web tracker allows you to track events anonymously. It enables anonymizing various user and session identifiers to support user privacy in case consent for tracking the identifiers is not given. This means that no user identifiers are sent to the Snowplow event collector. By default, anonymous tracking is not enabled.

Anonymous tracking can be configured on initialization, but can be reset later. You may wish to toggle this functionality on or off during a page visit, for example when a user accepts a cookie banner you may not want to disable anonymous tracking, or when a user logs in to your site.

For information about tracking user consent interactions and GDPR basis for processing, check out [this page](/docs/sources/web-trackers/tracking-events/consent-gdpr/). Read more about anonymous tracking in the [overview page](/docs/events/anonymous-tracking/).

On web, the following [user and session identifiers](/docs/events/ootb-data/user-and-session-identification/) can be anonymized:

- Client-side user identifiers:

  - `user_id`, a business identifier provided by you
  - `domain_userid`, set automatically by the tracker from the cookies
  - `userId` in the [session](/docs/sources/web-trackers/tracking-events/session/) entity, set automatically by the tracker to the same value as `domain_userid`
  - `tabId` in the [browser](/docs/sources/web-trackers/tracking-events/browsers/) entity, set automatically by the tracker

- Client-side session identifiers:

  - `domain_sessionid` and `domain_sessionidx`, set automatically by the tracker from the cookies
  - `previousSessionId` in the session entity, set automatically by the tracker

- Server-side user identifiers:
  - `network_userid` and `user_ipaddress`, set by the [Collector](/docs/pipeline/collector/)

## Configuring anonymous tracking

There are several levels to the anonymization depending on which of the three categories are affected. Set this using the [initialization configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### 1. Full client-side anonymization

```javascript
anonymousTracking: true,
stateStorageStrategy: 'cookieAndLocalStorage'
```

This mode will no longer track any user identifiers or session information. The tracker won't add the session entity to events, even if [session tracking](/docs/sources/web-trackers/tracking-events/session/) is enabled.

| Identifier          | Location in event | Included in event?  |
| ------------------- | ----------------- | ------------------- |
| `user_id`           | Atomic            | ❌                   |
| `domain_userid`     | Atomic            | ❌                   |
| `domain_sessionid`  | Atomic            | ❌                   |
| `domain_sessionidx` | Atomic            | ❌                   |
| `userId`            | Session entity    | ❌ no session entity |
| `sessionId`         | Session entity    | ❌ no session entity |
| `previousSessionId` | Session entity    | ❌ no session entity |
| `tabId`             | Browser entity    | ❌                   |
| `network_userid`    | Atomic            | ✅                   |
| `user_ipaddress`    | Atomic            | ✅                   |

Similar in behavior to setting `stateStorageStrategy: 'none'`, as it will store no values in cookies or localStorage, however by using `anonymousTracking` you can toggle this behavior on and off (useful for allowing events to be sent without user identifiers until cookie banners have been accepted).

Setting `stateStorageStrategy` to `cookieAndLocalStorage` or `localStorage` also allows for event buffering to continue working whilst not sending user information when `anonymousTracking` is enabled.

Anonymous tracking can be toggled on and off.

### 2. Client-side anonymization with session

```javascript
anonymousTracking: { withSessionTracking: true },
stateStorageStrategy: 'cookieAndLocalStorage'
```

This mode will continue to track session information but will track no user identifiers.

| Identifier          | Location in event | Included in event? |
| ------------------- | ----------------- | ------------------ |
| `user_id`           | Atomic            | ❌                  |
| `domain_userid`     | Atomic            | ❌                  |
| `domain_sessionid`  | Atomic            | ✅                  |
| `domain_sessionidx` | Atomic            | ✅                  |
| `userId`            | Session entity    | ❌ set to null UUID |
| `sessionId`         | Session entity    | ✅                  |
| `previousSessionId` | Session entity    | ❌                  |
| `tabId`             | Browser entity    | ❌                  |
| `network_userid`    | Atomic            | ✅                  |
| `user_ipaddress`    | Atomic            | ✅                  |

To achieve this, the tracker will use Cookies or local storage. For session tracking, `stateStorageStrategy` must be either `cookieAndLocalStorage` (default), `localStorage` or `cookie`. If this feature is enabled and the storage strategy is not appropriate, then full anonymous tracking will occur.

The Snowplow JavaScript Tracker performs sessionization client side. This allows anonymous session tracking to be done using client side storage without sending any user identifier fields to the collector.

### 3. Full anonymization/cookieless

```javascript
anonymousTracking: { withServerAnonymisation: true },
stateStorageStrategy: 'none',
eventMethod: 'post'
```

This mode will no longer track any user identifiers or session information, and will additionally prevent the Snowplow Collector from generating a `network_userid` cookie and capturing the users IP address. The same behavior described for above for Client side Anonymous tracking also applies.

| Identifier          | Location in event | Included in event?  |
| ------------------- | ----------------- | ------------------- |
| `user_id`           | Atomic            | ❌                   |
| `domain_userid`     | Atomic            | ❌                   |
| `domain_sessionid`  | Atomic            | ❌                   |
| `domain_sessionidx` | Atomic            | ❌                   |
| `userId`            | Session entity    | ❌ no session entity |
| `sessionId`         | Session entity    | ❌ no session entity |
| `previousSessionId` | Session entity    | ❌ no session entity |
| `tabId`             | Browser entity    | ❌                   |
| `network_userid`    | Atomic            | ❌                   |
| `user_ipaddress`    | Atomic            | ❌                   |

Setting `stateStorageStrategy` to `cookieAndLocalStorage` or `localStorage` also allows for event buffering to continue working whilst not sending user information when `anonymousTracking` is enabled. However for an experience that doesn't use any browser storage (cookieless), set `stateStorageStrategy` to `none`. This can be later toggled on, once a user accepts a cookie policy.

Server Anonymization requires the Snowplow Stream Collector v2.1.0+. Using a lower version will cause events to fail to send until Server Anonymization is disabled.

## Toggling anonymous tracking

You may wish to toggle this functionality on or off during a page visit, for example when a user accepts a cookie banner you may not want to disable anonymous tracking, or when a user logs in to your site.

### Enable anonymous tracking

> **Note:** Enabling Anonymous tracking will clear all current user, session and page data from events sent to the collector. Although not sent in requests to collector, existing user and session identifiers will not be removed from cookies or local storage. See below for information on how to clear user data.

If you wish to enable Anonymous Tracking, you can call:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking');
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';

enableAnonymousTracking();
```

***

which will enable client side anonymous tracking.

For full, cookieless anonymization, including anonymizing data within the Snowplow Collector (cookies and ip address), enable server anonymization too:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', {
  options: { withServerAnonymisation: true }
});
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';

enableAnonymousTracking({
  options: { withServerAnonymisation: true }
});
```

***

Server Anonymization requires the Snowplow Stream Collector v2.1.0+. Using a lower version will cause events to fail to send until Server Anonymization is disabled.

If you want to enable anonymous tracking with session tracking, then you can use:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', {
  options: { withSessionTracking: true }
});
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';

enableAnonymousTracking({
  options: { withSessionTracking: true }
});
```

***

From v3.1.0 it's also possible to change the `stateStorageStrategy` when enabling Anonymous Tracking, allowing you to switch off storage when turning anonymous tracking on:

**JavaScript (tag):**

```javascript
snowplow('enableAnonymousTracking', { options: {}, stateStorageStrategy: 'none' }); // Available from v3.1.0
```

**Browser (npm):**

```javascript
import { enableAnonymousTracking } from '@snowplow/browser-tracker';

enableAnonymousTracking({
  options: {},
  stateStorageStrategy: 'none'
});
```

***

### Disable anonymous tracking

To do this you can call the following methods:

**JavaScript (tag):**

```javascript
snowplow('disableAnonymousTracking');
```

or, if you wish to also adjust the `stateStorageStrategy` when enabling:

```javascript
snowplow('disableAnonymousTracking', {
  stateStorageStrategy: 'cookieAndLocalStorage'
});
```

**Browser (npm):**

```javascript
import { disableAnonymousTracking } from '@snowplow/browser-tracker';

disableAnonymousTracking();
```

or, if you wish to also adjust the `stateStorageStrategy` when enabling:

```javascript
import { disableAnonymousTracking } from '@snowplow/browser-tracker';

disableAnonymousTracking({
  stateStorageStrategy: 'cookieAndLocalStorage'
});
```

***

> **Note:** If configuring the tracker with `stateStorageStrategy: 'localStorage'` and anonymous tracking using `withSessionTracking: true`, then if you change to a `stateStorageStrategy` which prefer cookies such as `cookie` or `cookieAndLocalStorage` then the session identifiers will reset. To maintain session identifiers, ensure you use the same `stateStorageStrategy`.

### Clear user data

If you wish to clear all the cookies (except for the Collector `sp` cookie) and local storage values which contain user data when switching on anonymous tracking, or triggered by other actions on your site, you can call the following:

**JavaScript (tag):**

```javascript
snowplow('clearUserData');
```

**Browser (npm):**

```javascript
clearUserData();
```

***

From v3.1, this will also clear in memory session and user identifiers too. This ensures all possible identifiers are cleared and even if tracking is resumed you will see new session and user identifiers. If you'd like to preserve the in-memory session and user identifiers, for future events should you continue tracking after clearing the cookies, you can do so:

**JavaScript (tag):**

```javascript
snowplow('clearUserData', { preserveSession: true, preserveUser: true });
```

**Browser (npm):**

```javascript
clearUserData({ preserveSession: true, preserveUser: true });
```

***

## Respecting Do Not Track

Most browsers have a Do Not Track option which allows users to express a preference not to be tracked. You can respect that preference by setting the `respectDoNotTrack` field of the [initialization configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `true`. This prevents cookies from being sent and events from being fired.

## Opt-out cookie

Similar in function to Do Not Track, it's possible to set an [opt-out cookie](/docs/sources/web-trackers/cookies-and-local-storage/#opt-out-cookie) to prevent all events being tracked.

---

# Configure how events are sent by the web trackers
> Configure request methods, payload sizes, retries, and custom headers for event transmission to the collector.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/configuring-how-events-sent/

It's possible to add a fine-grained configuration for exactly how the web tracker should track and send events.

## Configuring the request

### Base64 encoding

Context entities and custom self-describing events can be encoded into Base64 to ensure that no data is lost or corrupted. You can turn encoding on or off using the `encodeBase64` field of the [tracker configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

This setting is enabled by default for GET requests. For POST requests, it is disabled by default as it doesn't provide the same benefits and can result in larger payload size and is more difficult to debug.

### Number of events per request

The default `bufferSize` is 1, i.e. an event will be processed into a single request and sent as soon as it is tracked. This can be increased to send events in batches, using the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

The `bufferSize` property is only relevant when making POST requests, [see below](#post-support).

### Maximum payload size

**POST requests**

Because the Snowplow Stream Collector can have a maximum request size, the Tracker limits POST requests to 40000 bytes. If the combined size of the events in `localStorage` is greater than this limit, they will be split into multiple POST requests. You can override this default using a `maxPostBytes` in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

**GET requests**

By default, there is no limit on the maximum size of GET requests – the tracker will add to queue and try to emit all GET requests irrespective of their size. However (since version 3.4), there is an optional `maxGetBytes` parameter which serves two purposes:

1. It prevents requests over the threshold in bytes to be added to event queue and retried in case sending them is not successful.
2. It sends events over the threshold as individual POST requests (same as for `maxPostBytes`).

The size of GET requests is calculated for their full GET request URL.

**Collector limit**

The Snowplow Stream Collector cannot process requests bigger than 1MB because that is the maximum size of a Kinesis record.

### Custom request headers

From v3.2.0, you are able to set custom headers with an `eventMethod: "post"` and `eventMethod: "get"` (Except for IE9). This functionality should only be used in the case where a Proxy or other Collector type is being used which allows for custom headers to be set on the request.

> **Warning:** Adding additional headers without returning the appropriate CORS Headers on the OPTIONS request will cause events to fail to send.

```javascript
customHeaders: {
  'Content-Language': 'de-DE, en-CA',
}
```

Set this in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### Disabling sending credentials with requests

The `credentials` option in the tracker configuration controls whether cookies are sent with requests to a Snowplow Collector running on a different domain. The default setting is `include` to always include the cookies. The setting translates to the `credentials` option in the fetch API, see [the documentation here](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#including_credentials).

```json
credentials: 'omit'
```

Before v4, this configuration was provided using the `withCredentials` flag.

### Sent timestamp for GET events

By default, the tracker sends POST requests. If you update the `eventMethod` to `get`, the tracker will send events using GET requests. These events will contain a `stm` [timestamp parameter](/docs/events/timestamps/) recording when the event was sent.

If you need to make requests as small as possible, you can omit the `stm` parameter by setting the `useStm` [configuration option](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `false`. This option was introduced in version 2.10.1. It's a niche parameter that you should only use if you have a specific reason to do so.

POST requests always include the sent timestamp.

## Network protocol and method

### Setting the request protocol

Normally the protocol (http or https) used by the Tracker to send events to a collector is the same as the protocol of the current page. You can force the tracker to use https by prefixing the collector endpoint with the protocol. For example:

**JavaScript (tag):**

```javascript
newTracker('sp', 'https://{{collector_url_here}}', {
  appId: 'my-app-id'
}
```

**Browser (npm):**

```javascript
newTracker('sp', 'https://{{collector_url_here}}', {
  appId: 'my-app-id'
}
```

***

### GET support

By default, events are sent by GET. This can be changed using the `eventMethod` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### POST support

If you set the `eventMethod` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) to `post`, the tracker will send events using POST requests rather than GET requests. In browsers which do not support cross-origin XMLHttpRequests (e.g. IE9), the tracker will fall back to using GET.

`eventMethod` defaults to `post`. `get` can be used for GET requests.

The main advantage of POST requests is that they circumvent Internet Explorer’s maximum URL length of 2083 characters by storing the event data in the body of the request rather than the querystring.

You can also batch events sent by POST by setting a numeric `bufferSize` field in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This is the number of events to buffer before sending them all in a single POST. If the user navigates away from the page while the buffer is only partially full, the tracker will attempt to send all stored events immediately, but this often doesn’t happen before the page unloads. Normally the tracker will store unsent events in `localStorage`, meaning that unsent events will be resent when the user next visits a page on the same domain. The `bufferSize` defaults to 1, meaning events are sent as soon as they are created.

We recommend leaving the `bufferSize` as the default value of 1. This ensure that events are sent as they are created, and reduces the chance of events being unsent and left in local storage, if a user closes their browser before a flush can occur (which happens on page visibility changing).

If you have set `bufferSize` to greater than 1, you can flush the buffer using the `flushBuffer` method:

**JavaScript (tag):**

```javascript
snowplow('flushBuffer');
```

**Browser (npm):**

```javascript
flushBuffer();
```

***

For instance, if you wish to send several events at once, you might make the API calls to create the events and store them and then and call `flushBuffer` afterwards to ensure they are all sent before the user leaves the page.

Note that if `localStorage` is inaccessible or you are not using it to store data, the buffer size will always be 1 to prevent losing events when the user leaves the page.

### `keepalive` option for Collector requests

The keepalive feature in the [fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) indicates that the request should be allowed to outlive the webpage that initiated it. It enables requests to the Snowplow Collector to complete even if the page is closed or navigated away from.

This option is disabled by default.

> **Note:** Browsers put a limit on keepalive requests of 64KB. You may control the size of the individual POST requests using the `maxPostBytes` option. But in case of multiple keepalive requests in parallel (may happen in case of multiple trackers), the limit is shared between them and may cause the requests to fail.

```ts
let tracker = newTracker("...", "...", {
    // ...
    keepalive: true
});
```

### Custom POST path

The POST path that is used to send POST requests to a collector can be changed with the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) value `postPath`.

`postPath` defaults to the standard path: `/com.snowplowanalytics.snowplow/tp2`

> **Warning:** While modifying `postPath` is generally recommended as a way to help avoid various ad-blocking tools when capturing events, it is also non-standard behavior that does not conform to the default Collector protocol.
>
> You must update the [Collector configuration](/docs/pipeline/collector/) to support the new path _before_ you send events to it with this setting. Otherwise, events will not be received by the Collector, or in some cases will be collected but will fail validation.
>
> Make sure that requests are supported by your Collector configuration or redirected to the Collector at the correct endpoint (normally this is `/com.snowplowanalytics.snowplow/tp2`).

## Retries

Unsuccessful requests are retried by default: the tracker retries on all 3xx, 4xx, and 5xx status codes except for 400, 401, 403, 410, and 422. Events in failed requests that are not retried are lost.

Starting with version 3.17 of the tracker, it is also possible to completely disable retry functionality, using the `retryFailedRequests` boolean option in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This option takes precedence over `retryStatusCodes` and `dontRetryStatusCodes` (see below).

### Connection timeout

When events are sent using POST or GET, they are given 5 seconds to complete by default. GET requests having a timeout is only available in 2.15.0.

`_connectionTimeout_: 5000`

This value is configurable when initialising the tracker ([configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/)) and is specified in milliseconds. The value specified here will effect both POST and GET requests.

**Warning:** Setting this value too low may prevent events from successfully sending to your collector or the tracker may retry to send events that have already arrived at the collector, as the tracker will assume the request failed on timeout, leading to duplicate events in the warehouse. **We recommend 5000 milliseconds as the minimum value and 10000 as the maximum value.**

### Custom retry HTTP codes

The tracker provides a retry functionality that sends the same events repeatedly in case GET or POST requests to the Collector fail. This may happen due to connection issues or a non-successful HTTP status code in Collector response.

Prior to version 3.5 of the tracker, requests receiving all 4xx and 5xx HTTP status codes in Collector response were retried. Since version 3.5, the behavior changed and became customizable:

By default, the tracker retries on all 3xx, 4xx, and 5xx status codes except for 400, 401, 403, 410, and 422. The set of status codes for which events should be retried or not is customizable. You can make use of the `retryStatusCodes` and `dontRetryStatusCodes` lists to specify them ([configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/)). Retry behavior can only be configured for non-successful status codes (i.e., >= 300).

```json
retryStatusCodes: [403], // override default behavior and retry on 403
dontRetryStatusCodes: [418] // force retry on 418
```

Please note that not retrying sending events to the Collector means that the events will be dropped when they fail to be sent. Take caution when choosing the `dontRetryStatusCodes`.

### Event queue behavior

The `useLocalStorage` [configuration option](/docs/sources/web-trackers/tracker-setup/initialization-options/) refers to event queue behavior. It was introduced in version 4.0.0. If you set it to `false`, events will be stored in memory only. If the page closes before events are sent, they'll be lost.

This is a **risky** setting that you should only use if you have a specific reason, such as compliance or privacy.

Previous versions of the tracker had a `useLocalStorage` option that referred to state storage. In version 3 [we deprecated this option](/docs/sources/web-trackers/migration-guides/v2-to-v3-migration-guide/) in favor of `stateStorageStrategy`. If you're upgrading from version 2 and were using `useLocalStorage`, switch to `stateStorageStrategy: 'localStorage'` if don't want to store state in cookies, or use the default `cookieAndLocalStorage` behavior.

## Callbacks

Provide callbacks within the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

### `onRequestSuccess` callback

> **Note:** Available from v3.18.1

The `onRequestSuccess` option allows you to supply a callback function to be executed whenever a request is successfully sent to the collector. In practice this means any request which returns a `2xx` status code will trigger this callback.

The callback's signature is: `(data: EventBatch) => void` where `EventBatch` can be either:

- `Record<string, unknown>[]` for POST requests
- `string[]` for GET requests

### `onRequestFailure` callback

> **Note:** Available from v3.18.1

The `onRequestFailure` option allows you to supply a callback function to be executed whenever a request fails to be sent to the collector. This is the inverse of the `onRequestSuccess` callback, so any non `2xx` status code will trigger this callback.

The callback's signature is: `(data: RequestFailure) => void` where `RequestFailure` is:

```ts
export type RequestFailure = {
  /** The batch of events that failed to send */
  events: EventBatch;
  /** The status code of the failed request */
  status?: number;
  /** The error message of the failed request */
  message?: string;
  /** Whether the tracker will retry the request */
  willRetry: boolean;
};
```

The format of `EventBatch` is the same as the `onRequestSuccess` callback.

## Custom tracker components and configuration

### Custom event store

Before events are sent to the Snowplow Collector, they are queued in an event store. This ensures that events can be batched together and retried in case of network failures.

By default, the tracker uses the browser local storage for the event store implementation. It is possible to use an in memory event store or provide a custom event store implementation by setting the `eventStore` configuration option.

```ts
import { newInMemoryEventStore } from '@snowplow/tracker-core';

let tracker = newTracker("...", "...", {
    // ...
    eventStore: newInMemoryEventStore({
        maxSize: 1000
    })
});
```

### Custom fetch function

The tracker provides an option to override the [fetch function](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) used to make requests to the Snowplow Collector. It may be useful in unit tests to test that requests are made without actually making them. You may also want to wrap the fetch function to modify or monitor the requests.

```ts
let tracker = newTracker("...", "...", {
    // ...
    customFetch: (input: RequestInfo, init?: RequestInit) => fetch(input, init)
});
```

### Synchronous cookie writes

Since v4 of the tracker, the tracker updates the user and session information in cookies asynchronously from the track calls (e.g., `trackPageView`). The cookies are updated within 10ms of the track call and the read cookie values are cached for 50ms. This strategy makes the track function take less time and avoid blocking the main thread of the app.

It might be desirable to update cookies synchronously. This can be useful for testing purposes to ensure that the cookies are written before the test continues. It also has the benefit of making sure that the cookie is correctly set before session information is used in events. The downside is that it is slower and blocks the main thread.

To configure synchronous cookie updates, use the `synchronousCookieWrite` configuration option:

```ts
let tracker = newTracker("...", "...", {
    // ...
    synchronousCookieWrite: true
});
```

---

# Configure cookie and storage settings for the web trackers
> Configure cookie names, domains, lifetimes, and storage strategies for optimal first-party tracking.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/

Snowplow allows for a highly configurable cookie set up. This allows for you to create optimal first party tracking in a privacy-first world, including [anonymous](/docs/sources/web-trackers/anonymous-tracking/) and cookieless tracking.

## Flow charts

In the PDF below you'll find a flow chart to help you with your cookie configuration, guiding you through the configuration options for both your [Snowplow Collector](/docs/api-reference/stream-collector/) and the [Snowplow JavaScript Tracker](/docs/sources/web-trackers/).

- [Cookie configuration for Snowplow CDI](/assets/config-calculator-snowplow-bdp.pdf)
- [Cookie configuration for Snowplow Self-Hosted](/assets/config-calculator-snowplow-ce.pdf)

## Cookie name

Set the cookie name for the tracker instance using the `cookieName` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). The default is `_sp_`. Snowplow uses two cookies, a domain cookie and a session cookie. In the default case, their names are `_sp_id` and `_sp_ses` respectively. If you are upgrading from an earlier version of Snowplow, you should use the default cookie name so that the cookies set by the earlier version are still remembered. Otherwise you should provide a new name to prevent clashes with other Snowplow users on the same page.

Once set, you can retrieve a cookie name thanks to the `getCookieName(basename)` method where basename is `id` or `ses` for the domain and session cookie respectively. As an example, you can retrieve the complete name of the domain cookie with `getCookieName('id')`.

## Cookie domain

If your website spans multiple subdomains e.g.

- `www.mysite.com`
- `blog.mysite.com`
- `application.mysite.com`

You will want to track user behavior across all those subdomains, rather than within each individually. As a result, it is important that the domain for your first party cookies is set to `.mysite.com` rather than `www.mysite.com`. By doing so, any values that are stored on the cookie on one of subdomain will be accessible on all the others.

Although it's possible to set this manually, we recommend that you enable automatic discovery and setting of the root domain, using the optional `discoverRootDomain` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). If it is set to `true` (the default), the tracker automatically discovers and sets the `configCookieDomain` value to the root domain.

> **Note:** If you have been setting this manually please note that the automatic detection does not prepend a `.` to the domain. For example a root domain of `.mydomain.com` would become `mydomain.com`. This is because the library we use for setting cookies doesn't care about the difference.
>
> **This will then result in a different domain hash, so we recommend that if you have been setting this manually with a leading `.` to continue to do so manually.**

To set the domain manually, use the `cookieDomain` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This will override the `discoverRootDomain` setting.

> **Warning:** In earlier versions, `discoverRootDomain` was disabled by default, and enabling it would cause it to ignore the `cookieDomain` setting, if set.
>
> In v4 this is reversed: `discoverRootDomain` is now enabled by default, but will be ignored if `cookieDomain` is set. When migrating from older versions, you may need to remove your `cookieDomain` configuration to maintain the earlier behavior.
>
> If you want the earlier default behavior where no domain at all is used, leave `cookieDomain` unset and explicitly set `discoverRootDomain` to `false`.

> **Warning:** Changing the cookie domain will reset all existing cookies. As a result, it might be a major one-time disruption to data analytics because all visitors to the website will receive a new `domain_userid`.

## Cookie lifetime and duration

Whenever a tracker is initialized on your domain, it will set domain-specific visitor's cookies. By default, these cookies will be active for 2 years. You can change this duration using the `cookieLifetime` [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) parameter or `setVisitorCookieTimeout` method.

**JavaScript (tag):**

```javascript
snowplow('newTracker', 'cf', '{{COLLECTOR_URL}}', {
  cookieLifetime: 86400 * 31,
});
```

or

```javascript
snowplow('setVisitorCookieTimeout', 86400 * 30);  // 30 days
```

**Browser (npm):**

```javascript
newTracker('cf', '{{COLLECTOR_URL}}', {
  cookieLifetime: 86400 * 31,
});
```

or

```javascript
setVisitorCookieTimeout(86400 * 30);  // 30 days
```

***

If `cookieLifetime` is set to `0`, the cookie will expire at the end of the session (when the browser closes). If set to `-1`, the first-party cookies will be disabled.

Whenever an event fires, the Tracker creates a session cookie. If the cookie didn't previously exist, the Tracker interprets this as the start of a new session.

By default the session cookie expires after 30 minutes. This means that a user leaving the site and returning in under 30 minutes does not change the session. You can override this default by setting `sessionCookieTimeout` to a duration (in seconds) in the configuration object. For example,

```javascript
{
  ...
  sessionCookieTimeout: 3600
  ...
}
```

would set the session cookie lifespan to an hour.

## Cookie samesite and secure attributes

Set the cookie samesite attribute for the tracker instance using the `cookieSameSite` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). The default is `None` for backward compatibility reasons, however `Lax` is likely a better option for most use cases given the reasons below. Valid values are `Strict`, `Lax`, `None` or `null`. `null` will not set the SameSite attribute.

It is recommended to set either `None` or `Lax`. You must use `None` if using the tracker in a third-party iframe. `Lax` is good in all other cases and must be used if not setting Secure to true.

> **Note:** It's been noted that Safari 12 doesn't persist cookies with `SameSite: None` as expected which can lead to rotation of the `domain_userid` from users using this browser. You should switch to `cookieSameSite: 'Lax'` in your tracker configuration to solve this, unless you are tracking inside a third-party iframe.

Set the cookie secure attribute for the tracker instance using the `cookieSecure` field of the configuration object. The default is `true`. Valid values are `true` or `false`.

It is recommended to set this to `true`. This must be set to `false` if using the tracker on non-secure HTTP.

## Storage strategy

Three strategies are made available to store the Tracker’s state: cookies, local storage or no storage at all. You can set the strategy with the help of the `stateStorageStrategy` parameter in the configuration object to `cookieAndLocalStorage` (the default), `cookie`, `localStorage` or `none` respectively.

When choosing local storage, the Tracker will additionally store events in local storage before sending them so that they can be recovered if the user leaves the page before they are sent.

## Local storage queue size

Because most browsers limit local storage to around 5mb per site, you may want to limit the number of events the tracker will queue in local storage if they fail to send. The default is a max queue size of 1000, but you may wish to reduce this if your web application also makes use local storage. To do so, you should set the optional `maxLocalStorageQueueSize` field of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/) is set to your desired value (e.g. 500).

---

# Cookie lifetime extension service
> How to create and use a Cookie Extension service, previously known as ID service, to mitigate against ITP by increasing cookie lifetimes.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/cookies-and-local-storage/cookie-extension/

Use a cookie lifetime extension service to extend the lifetime of your cookies.

Browsers, including iOS browsers (Safari, Chrome, Firefox) and desktop Safari, have limited the lifetime of some cookies, which limits the effectiveness of tracking a customer journey where the user is not regularly returning to your website.

As of Safari 16.4, released in April 2023, Safari sets the [lifetime of server-set first-party cookies](https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense) to a maximum of 7 days in the following cases:

1. The server setting the cookie is behind a CNAME that resolves (at any point) to a host that is third-party to the website the user is currently browsing.
2. The server setting the cookie is set with A/AAAA records that resolve to an IP address (IP4 or IP6) where the first half of the address does not match the first half of the IP address for the server on the website the user is currently browsing. (e.g. 203.52.1.2 and 203.52.56.22 are okay, 201.55.1.2 is not).

By using a cookie lifetime extension service, you can extend the lifetime of browser identifiers, and therefore maintain high-quality knowledge of your customer journeys.

## Service functionality

The responsibilities of this service are:

1. Create a unique identifier (UUID v4) for this browser, set it in a cookie and return it in a `Set-Cookie` response header on a domain accessible by the service at all times.
2. Increase the expiry for the cookie used as the `network_userid` identifier (by default, a cookie named `sp`, configured via `collector.config.name`) which should have the same value as the first cookie.

On this page, we've called the new unique identifier cookie `spCookieExtensionService`.

### Business logic

The Cookie Extension service code should include the following logic:

- If the Cookie Extension service new identifier cookie already exists on the request, then it should re-set the cookies with the same values and updated expiration for both the `spCookieExtensionService` and `sp` cookies.
- If `spCookieExtensionService` does not exist, but the Collector's `sp` cookie does exist, then the it should set `spCookieExtensionService value = sp cookie value`. _This will make sure we keep old identifiers in place and not lose any data._
- If `spCookieExtensionService` and `sp` are both missing, then it generates a new ID in the `spCookieExtensionService` and `sp` cookies with the same unique identifier generation algorithm with the Snowplow pipeline, currently UUID v4.
- The HTTP response should have a 200 OK status code but any additional payload is not necessary.

### Example code

Here are two examples of code for Cookie Extension service API endpoints.

**Next.js TypeScript:**

```ts
loading...
```

[View on GitHub](https://github.com/snowplow-industry-solutions/cookie-extension-service-examples/blob/main/examples/typescript/Next.js/api-route.ts)

**PHP:**

```php
loading...
```

[View on GitHub](https://github.com/snowplow-industry-solutions/cookie-extension-service-examples/blob/main/examples/php/wordpress/api-route.php)

***

## Deploy the service

You will need to deploy and execute your Cookie Extension service code from the same IP space that serves the main web document of your application. This is probably the web application system, or the CDN in front of the application.

This code has minimal functionality and could be:

- An **API endpoint** that is part of the main web application.
- **A function running in an edge worker.** Almost all modern CDNs allow the addition of 'middleware' code which runs on requests at edge nodes in front of the client's application domain.
- **A custom middleware based on the customer's framework**, e.g. ExpressJS middleware, Next.js middleware, Play custom action etc. that can run on every document request.
- **A low-footprint application with a single endpoint**, e.g. a Go web server.

```mermaid
sequenceDiagram
    autonumber
    participant Service as Cookie Extension service endpoint
    participant Tracking as Snowplow tracking code
    participant Collector as Snowplow Collector
    Tracking->>Service: Request an ID
    note over Service: Cookie Extension service code
    Service->>Tracking: OK response
    note over Tracking,Service: Set-Cookie: sp=...<br/>Set-Cookie: spCookieExtensionService=...
    loop Normal flow
        Tracking->>Collector: Event tracking
        note over Tracking,Collector: Cookie: sp=...#59; spCookieExtensionService=...
    end
```

## Configure the tracker

> **Note:** Before version 4.5.0 of the tracker this attribute was available as `idService`.

When the Cookie Extension service has been deployed on a system with the same resolved IP as the main document, the tracker can then be configured to orchestrate the required Cookie Extension service API calls.

This process is opt-in by using the `cookieExtensionService` option during tracker initialization:

**JavaScript (tag):**

```tsx
window.snowplow("newTracker", "sp", "{{collector_url_here}}", {
  /* ...Rest of the tracker options */
  cookieExtensionService: "/cookie-extension-service-endpoint"
});
```

**Browser (npm):**

```tsx
newTracker('sp1', 'c.customer.com', {
  cookieExtensionService: "/cookie-extension-service-endpoint",
  /* ...Rest of the tracker options */
 });
```

***

When the tracker detects this option it will send an HTTP request during initialization on this endpoint to have the service set the required identifiers before sending any event.

---

# Get cookie values to use in web tracking
> Retrieve domain user ID, session ID, and other cookie values from the tracker for use in your application.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/cookies-and-local-storage/getting-cookie-values/

You can retrieve cookie values to use elsewhere in your application, e.g. to send them to other systems, or to include them in server-side requests.

If you have access to the tracker instance, use the built-in getter methods. If you need to read cookie values from code that runs independently of the tracker, you can parse the cookie directly.

## Retrieving cookie properties from the tracker

It's possible to retrieve cookie properties for use in your code using a [tracker callback](/docs/sources/web-trackers/tracking-events/#get-event-properties).

### All `_sp_id` cookie values

The `getDomainUserInfo` method returns all the information stored in `_sp_id` first-party cookie in an array:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var domainUserInfo = sp.getDomainUserInfo();
 console.log(domainUserInfo);
})
```

**Browser (npm):**

```javascript
const domainUserInfo = sp.getDomainUserInfo();
console.log(domainUserInfo);
```

***

The `domainUserInfo` variable will contain an array with 11 elements:

1. A string set to `'1'` if this is the user's first session and `'0'` otherwise
2. The domain user ID
3. The timestamp at which the cookie was created
4. The number of times the user has visited the site
5. The timestamp for the current visit
6. The timestamp of the last visit
7. The session id
8. ID of the previous session (since version 3.5)
9. ID of the first event in the current session (since version 3.5)
10. Device created timestamp of the first event in the current session (since version 3.5)
11. Index of the last event in the session (used to inspect order of events) (since version 3.5)

### Cookie name

The `getCookieName` method returns the complete cookie name for the domain or session cookie:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var cookieName = sp.getCookieName('id');
 console.log(cookieName);
})
```

**Browser (npm):**

```javascript
const cookieName = sp.getCookieName('id');
console.log(cookieName);
```

***

The argument corresponds to the basename of the cookie: `'id'` for the domain cookie, `'ses'` for the session cookie.

### Domain user ID

The `getDomainUserId` method returns the user ID stored in the first-party cookie:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var domainUserId = sp.getDomainUserId();
 console.log(domainUserId);
})
```

**Browser (npm):**

```javascript
const domainUserId = sp.getDomainUserId();
console.log(domainUserId);
```

***

### Domain session index

The `getDomainSessionIndex` method returns the session index stored in the first-party cookie:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var domainSessionIndex = sp.getDomainSessionIndex();
 console.log(domainSessionIndex);
})
```

**Browser (npm):**

```javascript
const domainSessionIndex = sp.getDomainSessionIndex();
console.log(domainSessionIndex);
```

***

## Get cookie values without tracker access

You can use the following function to extract the Domain User Information from the ID cookie:

```javascript
/*
* Function to extract the Snowplow Domain User Information from the first-party cookie set by the Snowplow JavaScript Tracker
*
* @param string cookieName (optional) The value used for "cookieName" in the tracker constructor argmap
* (leave blank if you did not set a custom cookie name)
*
* @return string or bool The ID string if the cookie exists or false if the cookie has not been set yet
*/
function getSnowplowDuid(cookieName) {
  var cookieName = cookieName || '_sp_';
  var matcher = new RegExp(cookieName + 'id\\.[a-f0-9]+=([^;]+);?');
  var match = document.cookie.match(matcher);
  var split = match[1].split('.');
  if (match && match[1]) {
    return {
      'domain_userid': split[0],
      'domain_sessionidx': split[2],
      'domain_sessionid': split[5]
    }
  } else {
    return false;
  }
}
```

If you set a custom `cookieName` field in the argmap, pass that name into the function; otherwise call the function without arguments. Note that if the function is called before the cookie exists (i.e. when the user is visiting the page for the first time and `sp.js` has not yet loaded) if will return `false`.

---

# Cookies and local storage for the web trackers
> Understand how the web trackers use cookies and local storage to persist user and session information.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/cookies-and-local-storage/

Unless you have enabled `respectDoNotTrack` during initialization, the tracker will persist information on the client. By default, the Snowplow JavaScript and Browser Tracker make use of Cookies and local storage. The behavior of each of these cookies and local storage keys are described here.

## Cookies

By default, information will be stored both in cookies and local storage, i.e. `stateStorageStrategy: 'cookieAndLocalStorage'` in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This setting prefers cookie storage for `_sp_id` and `_sp_ses`.

Alternatively, you can specify `localStorage` to have the state stored only in local storage or `cookie` to only use cookies. Finally, you can set `stateStorageStrategy` to `none` in order not to store anything client-side. You may also leverage [`anonymousTracking`](/docs/sources/web-trackers/anonymous-tracking/) to control when values are stored in cookies or local storage.

The stored state takes the form of two first party cookies: the session cookie and the ID cookie. By default their names are prefixed with `_sp_`, but you can change this using the `cookieName` field in the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). Their names are suffixed with a hash of the current domain, so the full cookie names might look something like `_sp_ses.4209` and `_sp_id.4209`.

| Cookie name | Expires                                                            | Description                                                                                                                                       |
| ----------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_sp_id`    | 2 years or `cookieLifetime` set on tracker initialization          | Stores user information that is created when a user first visits a site and updated on subsequent visits.                                         |
| `_sp_ses`   | 30 minutes or `sessionCookieTimeout` set on tracker initialization | Used to identify if the user is in an active session on a site or if this is a new session for a user (i.e. cookie doesn't exist or has expired). |
| `sp`        | 1 year or `collector.cookie.expiration` set in collector config    | Stores a server-side collector generated unique identifier for a user that is sent with all subsequent tracking event events.                     |

### Domain ID cookie `_sp_id`

This cookie is called `_sp_id.{{DOMAIN HASH}}` by default. It is used to persist information about a user’s activity on the domain between sessions. It contains the following information:

- An ID for the user (`domainUserId`) based on a v4 (random) UUID. Generated by the [uuid](https://www.npmjs.com/package/uuid) library.
- The timestamp of the user’s first visit
- How many times the user has visited the domain
- The timestamp of the current visit
- The timestamp of the last visit
- The ID of the current session
- ID of the previous session (since version 3.5)
- ID of the first event in the current session (since version 3.5)
- Device created timestamp of the first event in the current session (since version 3.5)
- Index of the last event in the session (used to inspect order of events) (since version 3.5)

`_sp_id` is stored in the format: `{domainUserId}.{createdTime}.{visitCount}.{nowTime}.{lastVisitTime}.{sessionId}.{previousSessionId}.{firstEventId}.{firstEventTsInMs}.{eventIndex}`. Please note that the last 4 parts of the cookie (`previousSessionId`, `firstEventId`, `firstEventTsInMs`, `eventIndex`) are only available since version 3.5 of the tracker.

### Session cookie `_sp_ses`

Called `_sp_ses.{{DOMAIN HASH}}` by default, the only purpose of this cookie is to differentiate between different visits. Whenever an event is fired, the session cookie is set to expire in 30 minutes. (This value can be altered using `setSessionCookieTimeout`)

If no session cookie is already present when an event fires, the tracker treats this as an indication that long enough has passed since the user last visited that this session should be treated as a new session rather than a continuation of the previous session. The `visitCount` (how many times the user has visited) is increased by one and the `lastVisitTs` (the timestamp for the last session) is updated.

Note: A new session can be started at any time by calling the function `newSession`.

When using [anonymous tracking](/docs/sources/web-trackers/anonymous-tracking/) with session (`anonymousTracking: { withSessionTracking: true }`; available from v2.15.0+) this key will contain a _salt_ value which is used to stitch page views into a session. The value is never sent to the collector.

### Collector cookie `sp`

There is a third sort of Snowplow-related cookie: the cookie set by the Collector, independently of the JavaScript Tracker. The Collector cookie is called “sp”. It is either a first or third-party cookie, depending on the collector URL (it can be used as a first party cookie is the collector is on the same domain as the site), used to track users over multiple domains.

This cookie can be disabled by setting `collector.cookie.enabled` to false (See [here](/docs/api-reference/stream-collector/configure/) for more information).

### Opt-out cookie

A fourth cookie can be set that overrides all other settings.

It is possible to set an opt-out cookie in order not to track anything, similarly to Do Not Track, through `setOptOutCookie('opt-out');` where ‘opt-out’ is the name of your opt-out cookie. If this cookie is set, cookies won’t be stored and events won’t be fired.

## Local storage

Local storage will only be used if `stateStorageStrategy` is set to `localStorage` or `cookieAndLocalStorage` (default). Both the ID and session cookies listed above can be stored in local storage rather than as cookies by setting `stateStorageStrategy` to `localStorage`. Local storage can be disabled by setting `stateStorageStrategy` to `cookie` or `none`.

| Storage key                                  | Description                                                                                                                                                                                                                                                                        |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `snowplowOutQueue_{namespace}_post2`         | Used to store a cache of unsent events. This is used to reduce the chance of events to be lost due to page navigation and events not being set to the collector before the navigation event occurs. Where GET requests are used, this key will end in `_get` rather than `_post2`. |
| `snowplowOutQueue_{namespace}_post2.expires` | Used to match the concept of cookie expiry within local storage. This ensures a consistent behavior between cookie and local storage. Where GET requests are used, this key will end in `_get` rather than `_post2`.                                                               |

## Mapping values to tracker protocol

The values stored in the cookies listed above are mapped into the [tracker protocol](/docs/fundamentals/canonical-event/) when events are sent to a Snowplow Collector.

The below table shows which parameters the cookie values map to:

| Request Parameter | Event Parameter     | Cookie Value          |
| ----------------- | ------------------- | --------------------- |
| `duid`            | `domain_userid`     | `_sp_id.domainUserId` |
| `nuid`            | `network_userid`    | `sp`                  |
| `vid`             | `domain_sessionidx` | `_sp_id.visitCount`   |
| `sid`             | `domain_sessionid`  | `_sp_id.sessionId`    |

---

# Cross-domain tracking on web
> Track users across multiple domains by decorating links with user and session identifiers.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/cross-domain-tracking/

Configure [cross-domain tracking](/docs/events/cross-navigation/) using the `crossDomainLinker` and `useExtendedCrossDomainLinker` fields of the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

## Choose which links to decorate

Provide a callback for the `crossDomainLinker` option to configure which links to decorate. It should be a function taking one argument, the link element, that returns `true` if the tracker should decorate the link element and `false` otherwise.

We recommend specifying a subset of links to decorate, to avoid decorating unintended links. For example, links within the same domain, or "links" that are actually email addresses or telephone numbers.

This function would only decorate those links whose destination is `http://acme.de/` or whose HTML ID is `crossDomainLink`:

**JavaScript (tag):**

```javascript
snowplow('newTracker', 'namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function (linkElement) {
    return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
  }
});
```

**Browser (npm):**

```javascript
newTracker('namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function (linkElement) {
    return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
  }
});
```

***

If you want to decorate every link to the domain `github.com`:

```javascript
{
  crossDomainLinker: function (linkElement) {
    return /^https:\/\/github\.com/.test(linkElement.href);
  }
}
```

### Check hostname

To only decorate links that are on a different domain to the current page, check the hostname.

```javascript
{
  crossDomainLinker: function (linkElement) {
    return linkElement.hostname !== location.hostname;
  }
}
```

If you have a shared configuration deployed across multiple sites, you may want to check both:

```javascript
{
  crossDomainLinker: function (linkElement) {
    var networkSites = [
      "mysite.com",
      "sistersite.com"
    ];
    return linkElement.hostname !== location.hostname && networkSites.indexOf(linkElement.hostname) > -1;
  }
}
```

Here, link decoration will only occur when links are to different sites within the list.

## Update event listeners

When the tracker loads it doesn't immediately decorate links. Instead, it adds event listeners to links which decorate them when a user clicks on them or navigates to them using the keyboard. This ensures that the timestamp added to the querystring is fresh.

If further links get added to the page after the tracker has loaded, you can use the tracker's `crossDomainLinker` method to add listeners again. Listeners won't be added to links which already have them, so it's good practice to use the same linker function every call.

**JavaScript (tag):**

```javascript
snowplow('crossDomainLinker', function (linkElement) {
  return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
});
```

**Browser (npm):**

```javascript
crossDomainLinker(function (linkElement) {
  return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
});
```

***

## Configure extended format properties

To decorate links with extended `_sp` format properties, set `useExtendedCrossDomainLinker`:

**JavaScript (tag):**

```javascript
snowplow('newTracker', 'namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function(defineWhichLinksToDecorate) { },
  useExtendedCrossDomainLinker: true,
});
```

**Browser (npm):**

```javascript
newTracker('namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function(defineWhichLinksToDecorate) { },
  useExtendedCrossDomainLinker: true,
});
```

***

This will add `sessionId` and `sourceId` to the querystring.

To include further properties:

**JavaScript (tag):**

```javascript
snowplow('newTracker', 'namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function(defineWhichLinksToDecorate) { },
  useExtendedCrossDomainLinker: {
    userId: true;
    sessionId: true;
    sourceId: true;
    sourcePlatform: true;
    reason: true; // can also be a callback
  },
});
```

**Browser (npm):**

```javascript
newTracker('namespace', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax',
  crossDomainLinker: function(defineWhichLinksToDecorate) { },
  useExtendedCrossDomainLinker: {
    userId: true;
    sessionId: true;
    sourceId: true;
    sourcePlatform: true;
    reason: true; // can also be a callback
  },
});
```

***

> **Note:** If your `userId` values contains PII, check that you're not leaking them to third-party sites in your link decoration.

Provide a callback to `reason` in the form `((evt: Event) => string)` to add custom information to the querystring. If set to `true`, the link text will be used.

## Shared page URLs and user IDs

If a user clicks a cross-site link, and the tracker decorates the URL with their `domain_userid`, and then they share that URL, other users will also have the parameters when they visit the shared page. This could cause problems if you are using `domain_userid` as a user identifier.

You can choose to hide the parameter after the first event, e.g. a page view, fires on the destination page. Because the pipeline populates dedicated event fields with the parameter, you won't need additional modeling to stitch the IDs. If the user copies the URL after the tracking code has fired, it may not include the `_sp` parameter and get copied to other users they share it with.

You can do this using the [History API](https://developer.mozilla.org/en-US/docs/Web/API/History/replaceState).

**JavaScript (tag):**

The URL-updating code runs in a [callback](/docs/sources/web-trackers/tracking-events/#get-event-properties) to ensure it doesn't run before the page view event can capture the original URL.

```javascript
snowplow('trackPageView'); // page URL is https://example.com/?example=123&_sp=6de9024e-17b9-4026-bd4d-efec50ae84cb.1680681134458
snowplow(function(){
  if (/[?&]_sp=/.test(window.location.href)) {
    history.replaceState(history.state, "", window.location.href.replace(/&?_sp=[^&#]+/, "")); // page URL is now https://example.com/?example=123
  }
});
```

**Browser (npm):**

```javascript
trackPageView(); // page URL is https://example.com/?example=123&_sp=6de9024e-17b9-4026-bd4d-efec50ae84cb.1680681134458
if (/[?&]_sp=/.test(window.location.href)) {
  history.replaceState(history.state, "", window.location.href.replace(/&?_sp=[^&#]+/, "")); // page URL is now https://example.com/?example=123
}
```

***

This approach can work with other parameters, e.g. `utm_source`, but it's not recommended unless you are sure other systems have also already captured the parameter values.

Some systems may consider the URL update a single page application page change, and automatically fire additional page view events with this implementation. Test this technique to ensure compatibility with your existing vendors.

---

# Global context for web trackers
> Define context entities once and automatically attach them to all tracked events using global context.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/

**Global context** (also known as global entities) lets you define your own contexts once (e.g. on tracker initialization) and then have this context sent with every single event subsequently recorded on the page. This saves having to manually build and send the context array with every single event fired.

Here is an example that adds a global context entity to all subsequently tracked events:

**JavaScript (tag):**

```javascript
// Create a context entity and add it to global context
let contextEntity = {
  schema: 'iglu:com.acme/user_context/jsonschema/1-0-0',
  data: { userid: 1234, name: 'John Doe' }
};
window.snowplow('addGlobalContexts', [contextEntity]);

// The global context will be added to this page view event as well
window.snowplow('trackPageView');
```

**Browser (npm):**

```javascript
import { addGlobalContexts } from "@snowplow/browser-tracker";

// Create a context entity and add it to global context
let contextEntity = {
  schema: 'iglu:com.acme/user_context/jsonschema/1-0-0',
  data: { userid: 1234, name: 'John Doe' }
};
addGlobalContexts([contextEntity]);

// The global context will be added to this page view event as well
trackPageView();
```

***

You may not want to add the same context entity to all tracked events. There are several ways to make global context dynamic:

1. Using a **[context generator](#context-generators)** that generates context entities on the fly when events are sent.
2. Using **[filter functions](#filter-functions)** that filter which events to add global context entities to.
3. Using **[rulesets](#rulesets)** that define rules which types of events to accept or reject when generating global context entities.

### Context generators

Generating context on-the-fly is accomplished with **context generators**. A context generator is a callback that will be evaluated with an optional argument that contains useful information.

A context generator can return a single context entity, an array of context entities, or `undefined` (to add no context for that event).

A sample context generator that conditionally generates a context entity could look like this:

**JavaScript (tag):**

```javascript
const contextGenerator = (args) => {
    if (args.eventType == 'pv') {
        return {
            schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
            data: { test: 1 },
        };
    }
};
window.snowplow('addGlobalContexts', [contextGenerator]);
```

**Browser (npm):**

```javascript
const contextGenerator = (args) => {
    if (args.eventType == 'pv') {
        return {
            schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
            data: { test: 1 },
        };
    }
};
addGlobalContexts([contextGenerator]);
```

***

The optional input is an associative array that contains three elements:

- `event` : self-describing JSON
- `eventType` : string
- `eventSchema` : string (schema URI)

Keep in mind that the arguments `eventType` and `eventSchema` are data found in `event`. `eventType` and `eventSchema` are provided for convenience, so that simple tasks don't require users to search through the event payload.

#### `eventType`

This argument is a string taken from the event payload field, `e`.

`eventType` takes the following values:

| Type                           | `e`       |
| ------------------------------ | --------- |
| Pageview tracking              | pv        |
| Page pings                     | pp        |
| Link click                     | ue        |
| Ad impression tracking         | ue        |
| Ecommerce transaction tracking | tr and ti |
| Custom structured event        | se        |
| Custom self describing event   | ue        |

Further information about the event payload can be found in the [tracker protocol documentation](/docs/events/).

#### `eventSchema`

Users should be aware of the behavior of the argument `eventSchema`. Since 'first-class events' (e.g. structured events, transactions, pageviews, etc.) lack a proper schema (their event type is determined by the `e` field), callbacks will be provided the upper-level schema that defines the payload of all events:

`iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4`

For self describing events, `eventSchema` will be the schema that describes the self describing event, not the event payload. Again, this behavior isn't necessarily uniform, but provides more utility to differentiate events.

### Conditional context providers

We can augment context primitives by allowing them to be sent conditionally. While it's possible to define this functionality within context generators (with conditional logic), conditional context providers simplify common ways of sending contexts that follow certain rules.

The general form is an array of two objects:

`[conditional part, context primitive or [array of primitives]]`

The conditional part is standardized into two options:

- a filter function
- a schema ruleset

### Filter functions

Filter functions take the standard callback arguments defined for context generators, but instead of returning a Self Describing JSON, return a boolean value. As should be expected: `true` will attach the context part, `false` will not attach the context part.

### Example

**JavaScript (tag):**

```javascript
// A filter that will only attach contexts to structured events
function structuredEventFilter(args) {
  return args['eventType'] === 'se';
}
var globalContextDefinition = [structuredEventFilter, contextEntityToBeAdded];
window.snowplow('addGlobalContexts', [globalContextDefinition]);
```

**Browser (npm):**

```javascript
// A filter that will only attach contexts to structured events
function structuredEventFilter(args) {
  return args['eventType'] === 'se';
}
var globalContextDefinition = [structuredEventFilter, contextEntityToBeAdded];
addGlobalContexts([globalContextDefinition]);
```

***

### Rulesets

Rulesets define when to attach context primitives based on the event schema. This follows the standard behavior for all callbacks (the schema used to evaluate is the same provided in `eventSchema`, namely the payload schema for "first-class events" and otherwise the schema found within the self describing event).

Here's the specific structure of a ruleset, it's an object with certain optional rules that take the form of fields, each holding an array of strings:

```json
{
    accept: [],
    reject: []
}
```

The rules are evaluated as follows:

- The context is attached only if at least one `accept` rule matches the event schema **and** no `reject` rules match.
- If you specify only `reject` rules (no `accept`), the context is never attached.
- Both `accept` and `reject` can be a single rule string or an array of rule strings.

Some examples, take note that wild-card matching URI path components is defined with an asterisk, `*`, in place of the component:

```javascript
// Only attaches contexts to this one schema
var ruleSetAcceptOne = {
  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0']
};

// Only attaches contexts to these schemas
var ruleSetAcceptTwo = {
  accept: ['iglu:com.mailchimp/cleaned_email/jsonschema/1-0-0',
  'iglu:com.mailchimp/subscribe/jsonschema/1-0-0']
};

// Only attaches contexts to schemas with mailchimp vendor
var ruleSetAcceptVendor = {
  accept: ['iglu:com.mailchimp/*/jsonschema/*-*-*']
};

// Only attaches contexts to schemas that aren't mailchimp vendor
var ruleSetRejectVendor = {
  reject: ['iglu:com.mailchimp/*/jsonschema/*-*-*']
};

// Only attach to Snowplow first class events
var ruleSet = {
  accept: ['iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4']
};
```

You can add a global context entity with a ruleset as follows:

**JavaScript (tag):**

```javascript
var globalContextDefinition = [ruleSet, contextEntityToAdd];
window.snowplow('addGlobalContexts', [globalContextDefinition]);
```

**Browser (npm):**

```javascript
var globalContextDefinition = [ruleSet, contextEntityToAdd];
addGlobalContexts([globalContextDefinition]);
```

***

### Rule requirements

All rules and schemas follow a standard form:

`protocol:vendor/event_name/format/version`

And rules must meet some requirements to be considered valid:

- Two parts are invariant: protocol and format. They are always `iglu` and `jsonschema` respectively.
  - Wildcards can therefore be used only in `vendor`, `event_name` and `version`.

- Version matching must be specified like so: `*-*-*`, where any part of the versioning can be defined, e.g. `1-*-*`, but only sequential parts are to be wildcarded, e.g. `1-*-1` is invalid but `1-*-*` is valid.

- Vendors require the first two "larger parts":
  - `com.acme.*`

- Vendors cannot be defined with non-wildcarded parts between wildcarded parts:

  - `com.acme.*.marketing.*` is invalid
  - `com.acme.*.*` is valid

### Named global entities

From version 4 onwards, you can alternatively supply the definitions as a mapping object. This associates each global entity with a "name" or "tag" that can be used to reference it in future calls.

For example, in the below we associate a global entity definition with the name `example`, and can then update or remove it by name in later calls.

**JavaScript (tag):**

```javascript
// add the entity defined by `globalContextDefinition` with the name "example"
var globalContextDefinition = [ruleSet, contextEntityToAdd];
window.snowplow('addGlobalContexts', { example: globalContextDefinition });

// update the global entity named "example"; this replaces the entity from `globalContextDefinition`
var globalContextDefinition2 = [ruleSet, differentContextEntity];
window.snowplow('addGlobalContexts', { example: globalContextDefinition2 });

// remove the global entity named "example"; it will no longer appear on subsequent events
window.snowplow('removeGlobalContexts', ["example"]);
```

**Browser (npm):**

```javascript
// add the entity defined by `globalContextDefinition` with the name "example"
var globalContextDefinition = [ruleSet, contextEntityToAdd];
addGlobalContexts({ example: globalContextDefinition });

// update the global entity named "example"; this replaces the entity from `globalContextDefinition`
var globalContextDefinition2 = [ruleSet, differentContextEntity];
addGlobalContexts({ example: globalContextDefinition2 });

// remove the global entity named "example"; it will no longer appear on subsequent events
removeGlobalContexts(["example"]);
```

***

Names can be whatever makes sense to you, but each name can only be associated with a single entity definition at a time (additions with the same name will replace the previous value). The value for the named entity can be the same as unnamed global entities defined with the array method, including generator functions, filter functions, and rulesets and are processed equivalently.

### Global contexts methods

These are the standard methods to add and remove global context entities:

#### `addGlobalContexts`

To add global contexts:

**JavaScript (tag):**

```javascript
// unnamed global entities
snowplow('addGlobalContexts', [array of global contexts])

// named global entities
snowplow('addGlobalContexts', {
  name: globalContext1,
  name2: globalContext2,
})
```

**Browser (npm):**

```javascript
// unnamed global entities
addGlobalContexts([array of global contexts])

// named global entities
addGlobalContexts({
  name: globalContext1,
  name2: globalContext2,
})
```

***

#### `removeGlobalContexts`

To remove a global context:

**JavaScript (tag):**

```javascript
// unnamed global entities
snowplow('removeGlobalContexts', [array of global contexts])

// named global entities
snowplow('removeGlobalContexts', ["name", "name2"]) // array of entity names
```

**Browser (npm):**

```javascript
// unnamed global entities
removeGlobalContexts([array of global contexts])

// named global entities
removeGlobalContexts(["name", "name2"]) // array of entity names
```

***

Unnamed global context entities are matched as follows:

- **Objects** (context entities, filter providers, ruleset providers): Compared by JSON serialization, so the structure must be identical.
- **Functions** (context generators): Compared by reference. You must pass the exact same function instance that was originally added.

For example:

**JavaScript (tag):**

```javascript
var entity = {
  schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
  data: { test: 1 },
};
window.snowplow('addGlobalContexts', [entity]); // add a global context
window.snowplow('removeGlobalContexts', [entity]); // remove the global context
```

**Browser (npm):**

```javascript
var entity = {
  schema: 'iglu:com.acme.marketing/some_event/jsonschema/1-0-0',
  data: { test: 1 },
};
addGlobalContexts([entity]); // add a global context
removeGlobalContexts([entity]); // remove the global context
```

***

#### `clearGlobalContexts`

To remove all named and unnamed global context entities:

**JavaScript (tag):**

```javascript
snowplow('clearGlobalContexts');
```

**Browser (npm):**

```javascript
clearGlobalContexts();
```

***

---

# Track custom events on web
> Track self-describing events and attach custom entities using JSON schemas for flexible data capture.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/custom-tracking-using-schemas/

The specific data you need might not be covered by the JavaScript tracker's built-in event types and entities. In this case, you can use Snowplow’s [self-describing events](/docs/fundamentals/events/#self-describing-events) and custom [entities](/docs/fundamentals/entities/) to track [customised data](/docs/events/custom-events/).

## Track a custom event (self-describing)

To track a custom self-describing event, use the `trackSelfDescribingEvent` method. For example:

**JavaScript (tag):**

```javascript
snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/1-0-0',
    data: {
        productId: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        availableSince: new Date(2013,3,7)
    }
  }
});
```

**Browser (npm):**

```javascript
import { trackSelfDescribingEvent } from '@snowplow/browser-tracker';

trackSelfDescribingEvent({
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/1-0-0',
    data: {
        productId: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        availableSince: new Date(2013,3,7)
    }
  }
});
```

***

The second argument or event property, depending on tracker, is a [self-describing JSON](http://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/). It has two fields:

- A `data` field, containing the properties of the event
- A `schema` field, containing the location of the [JSON schema](http://json-schema.org/) against which the `data` field should be validated.

Like all `trackX` methods, `trackSelfDescribingEvent` can also be passed an array of custom entities as an additional parameter.

## Track a custom entity

Custom context can be used to augment any standard Snowplow event type, including self-describing events, with additional data. We refer to this custom context as [entities](/docs/fundamentals/entities/).

The context is an array of entities. More than one entity (of either different or the same type) can be attached to an event. The `context` argument (if it is provided at all) should be a non-empty array.

As with self-describing events, if you want to create your own custom context, you will need to [create a corresponding schema](/docs/event-studio/data-structures/). Snowplow uses the schema to validate that the JSON containing the context properties is well-formed.

> **Tip:** Custom entities can be added as an extra argument to most Snowplow's `trackX()` methods, e.g. `trackPageView` or `trackLinkClick`.
>
> Not all plugins support adding custom entities; please refer to the corresponding plugin documentation for details.

Here are two examples of schema for custom entities. One describes a screen:

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.example",
        "name": "screen",
        "format": "jsonschema",
        "version": "1-0-1"
    },
    "type": "object",
    "properties": {
        "screenType": {
            "type": "string"
        },
        "lastUpdated": {
            "type": "string"
        }
    }
    "required": ["screenType", "lastUpdated"],
    "additionalProperties": false
}
```

and the other describes a user on that screen:

```json
{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "self": {
        "vendor": "com.example",
        "name": "user",
        "format": "jsonschema",
        "version": "2-0-0"
    },
    "type": "object",
    "properties": {
        "user": {
            "type": "string"
        }
    }
    "required": ["user"],
    "additionalProperties": false
}
```

**Important:** Even if only one custom entity is being attached to an event, it still needs to be wrapped in an array.

Tracking a **page view** with both of these example entities attached:

**JavaScript (tag):**

```javascript
snowplow('trackPageView', {
  context: [{
    schema: "iglu:com.example_company/page/jsonschema/1-2-1",
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  },
  {
    schema: "iglu:com.example_company/user/jsonschema/2-0-0",
    data: {
      userType: 'tester'
    }
  }]
});
```

**Browser (npm):**

```javascript
trackPageView({
  context: [{
    schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  },
  {
    schema: 'iglu:com.example_company/user/jsonschema/2-0-0',
    data: {
      userType: 'tester'
    }
  }]
});
```

***

Tracking a **self describing event** with both of these entities attached:

**JavaScript (tag):**

```javascript
snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.example_company/product_viewed/jsonschema/1-0-1',
    data: {
      productId: '12345',
      price: 10.99
    }
  },
  context: [{
    schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  },
  {
    schema: "iglu:com.example_company/user/jsonschema/2-0-0",
    data: {
      userType: 'tester'
    }
  }]
});
```

**Browser (npm):**

```javascript
trackSelfDescribingEvent({
  event: {
    schema: 'iglu:com.example_company/product_viewed/jsonschema/1-0-1',
    data: {
      productId: '12345',
      price: 10.99
    }
  },
  context: [{
    schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  },
  {
    schema: "iglu:com.example_company/user/jsonschema/2-0-0",
    data: {
      userType: 'tester'
    }
  }]
});
```

***

## Track a structured event

We recommend using custom self-describing events instead of structured events, as the provide more better data governance and easier modeling.

There are five parameters that can be associated with each structured event. Only the first two are required:

| Name       | Required? | Description                                                                                                                                                      | Type    |
| ---------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `Category` | Yes       | The name you supply for the group of objects you want to track e.g. 'media', 'ecomm'.                                                                            | String  |
| `Action`   | Yes       | Defines the type of user interaction for the web object e.g. 'play-video', add-to-basket'.                                                                       | String  |
| `Label`    | No        | Identifies the specific object being actioned e.g. ID of the video being played, or the SKU or the product added to basket.                                      | String? |
| `Property` | No        | Describing the object or the action performed on it. This might be the quantity of an item added to basket.                                                      | String? |
| `Value`    | No        | Quantify or further describe the user action. This might be the price of an item added to basket, or the starting time of the video where play was just pressed. | Float?  |

An example of tracking a user listening to a music mix:

**JavaScript (tag):**

```javascript
snowplow('trackStructEvent', {
  category: 'Mixes',
  action: 'Play',
  label: 'MrC/fabric-0503-mix',
  property: '',
  value: 0.0
});
```

**Browser (npm):**

```javascript
import { trackStructEvent } from '@snowplow/browser-tracker';

trackStructEvent({
  category: 'Mixes',
  action: 'Play',
  label: 'MrC/fabric-0503-mix',
  value: 0.0
});
```

***

---

# Web tracker SDKs
> Track user behavior on websites using the JavaScript tracker or Browser tracker with flexible plugins and tracking methods.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/

Snowplow provides two web trackers to use depending how you wish to add analytics to your webapp.

The **JavaScript tracker** is loaded via tag: add code snippets to your website or Tag Manager solution. Some plugins are included in the standard tag, but we also provide customization options.

The **Browser tracker** is available via `npm` (`@snowplow/browser-tracker`) and can be directly bundled into your application. It supports core tracking methods out of the box and can be extended through plugins (`@snowplow/browser-plugin-*`). This tracker is often used when natively integrating tracking into React, Angular and Vue applications.

As the API is similar, we have combined the documentation for both trackers. We've marked the sections which are only relevant to one tracker or another.

The JavaScript and Browser tracker, along with the [Node.js](/docs/sources/node-js-tracker/) tracker, are part of [one monorepo](https://github.com/snowplow/snowplow-javascript-tracker).

The current version is 4.6.8.

---

# Migration guides for the web trackers
> Upgrade to newer versions of the web trackers with step-by-step migration guides.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/migration-guides/

This section contains information to help you upgrade to newer versions of the web trackers.

---

# Migrating from web trackers v2 to v3
> Migrate from JavaScript tracker v2 to v3 with the new tracking API, plugins, and configuration changes.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/migration-guides/v2-to-v3-migration-guide/

A number of features which have been deprecated for a number of v2.x releases have been removed in v3 and a new API for `track` methods has been introduced which is a major breaking change.

> **Note:** These code snippets are for the Javascript (tag) tracker, but the same changes apply for the Browser (npm) tracker.

## New CDN Locations

The JavaScript Tracker is now available in new locations on third party CDN providers. CDN providers are useful for getting tracking up and running quickly, particularly when testing a new release. We still strongly recommend renaming `sp.js` and hosting the file yourself but if you'd like to access the tracker from a CDN, then is available on [jsDelivr](https://www.jsdelivr.com/package/npm/@snowplow/javascript-tracker?path=dist) and [unpkg](https://unpkg.com/browse/@snowplow/javascript-tracker@latest/dist/).

## New tracking API (BREAKING CHANGE)

The track methods now accept an Object which contains the event data, instead of the parameter list in v2. As an example, here is what a Page View event used to look like:

```javascript
window.snowplow('trackPageView', 'My Title', [ // Set page title; add page context
  {
    schema: "iglu:org.schema/WebPage/jsonschema/1-0-0",
    data: {
      keywords: ['tester']
    }
  }
]);
```

Starting in v3, this looks like:

```javascript
window.snowplow('trackPageView', {
  title: 'My Title',
  context: [
    {
      schema: 'iglu:org.schema/WebPage/jsonschema/1-0-0',
      data: {
        keywords: ['tester'],
      },
    },
  ],
});
```

This improves the readability of the tracking code, removing the need for comments that are often used to describe what each property represents. It also allows the tracking functions to be easily extended in the future.

All functions that previously accepted multiple arguments now accept an Object instead. Functions which have a single parameter, such as the setters, have remained as they are as they will never have multiple parameters. For example:

```javascript
window.snowplow('setUserId', 'alex-d-123');
```

You can find the complete list of all the functions and the new named arguments in the [v3 documentation](/docs/sources/web-trackers/).

## Renamed Properties (BREAKING CHANGE)

All references to `whitelist` and `blacklist` have been removed in Link Click tracking and Form tracking. They have been replaced with `allowlist` and `denylist`.

For example, in v2 you would specify:

```javascript
window.snowplow('enableLinkClickTracking',
  { blacklist: ['exclude'] },
  true,
  true,
});
```

In v3 this is now:

```javascript
window.snowplow('enableLinkClickTracking', {
  options: { denylist: ['exclude'] },
  pseudoClicks: true,
  trackContent: true,
});
```

## Removed Tracking

Three sets of properties will no longer be collected in the default configuration. Although in the case of two of them, they can still be loaded as additional plugins (See "Whats New?").

`Parrable` opt-out cookie tracking is no longer available. This specific Parrable cookie is no longer part of Parrable.

Tracking Browser features, from the `navigator.mimeTypes` is no longer included as this is a [deprecated API](https://developer.mozilla.org/en-US/docs/Web/API/NavigatorPlugins/mimeTypes). If you still wish to track this, you can load this as a plugin.

Optimizely Classic tracking is no longer included by default. Optimizely now defaults all installations and has migrated the majority of installations to Optimizely X, which is still included by default. Optimizely Classic tracking can be loaded as a plugin.

## Removed Configuration Options

This isn't a breaking change as the tracker will ignore unrecognised properties during initialisation but you may want to take this opportunity to remove them. The properties below were all part of the configuration Object passed to a `newTracker` call.

`pageUnloadTimer` has been removed. The tracker no longer attempts to block when unloading the page. Instead it will send events on `visibilityChange` to `hidden` and make one final attempt to flush any unsent events in `beforeUnload`.

`forceSecureTracker` and `forceUnsecureTracker` have been removed. You can now force the protocol of the collector endpoint when initialising the tracker, or leave it off and the tracker will use the same as the current site. e.g. `window.snowplow('newTracker', 'sp', 'collector.mywebsite.com', {})` will use the same protocol whereas `window.snowplow('newTracker', 'sp', 'https://collector.mywebsite.com', {})` will force the events to be sent over HTTPS.

`useLocalStorage` and `useCookies` have been removed in favour of `stateStorageStrategy` which can be one of four values: `'cookieAndLocalStorage'` (default), `'cookie'`, `'localStorage'` or `'none'`.

`contexts.parrable` is no longer available. The Parrable feature this tracked is no longer part of Parrable.

`post` has been removed. Set `eventMethod` instead, the available options are: `'post'` (default), `beacon` or `get`.

`skippedBrowserFeatures` is removed as `sp.js` will no longer collect `mimeTypes` browser features without the use of an additional Plugin (See Plugins for more information).

## Removed Functions

Some deprecated functions have also been removed. This shouldn't cause any errors if you do call them, although you will see warnings in your developer tools console.

The following have been removed:

`trackUnstructEvent` has been removed in favour of `trackSelfDescribingEvent`. They behave in exactly the same way, `trackUnstructEvent` was an alias for `trackSelfDescribingEvent`.

`window._snaq` has been removed. All calls to the tracker should be made via `window.snowplow` now (or your own global if you have configured the Tag parameters).

Legacy `window.Snowplow` (note the capital S) and the containing functions (`getTrackerCf`, `getTrackerUrl` and `getAsyncTracker`) have been removed.

`setCountPreRendered` and the prerendered visibility checks, as this is [a deprecated API](https://developer.mozilla.org/en-US/docs/Web/API/Document/visibilityState#browser_compatibility).

User Fingerprinting was removed in v2.13 but now the remaining method stubs `enableUserFingerprint` and `getUserFingerprint` have been completely removed.

The following functions are removed and should be set during Tracker initialisatoin: `setAppId`, `setCookieNamePrefix`, `setCookieDomain`, `setSessionCookieTimeout`, `setUserFingerprintSeed`, `respectDoNotTrack`, `setPlatform`, `encodeBase64` and `setCollectorCf`.

## Whats New?

There are also a number of new features which you might want to now start to take advantage of. Additionally the full `sp.js` in 3.0.0 is smaller than 2.17.3 and `sp.lite.js` is even smaller! Read on...

#### Plugins

You can now load Plugins which extend the JavaScript Tracker with new Contexts and new API functions. As an example, you might want to continue tracking the `mimeTypes` which have been dropped by default in v3. To do that you would write the following:

```javascript
window.snowplow('newTracker', 'sp', {});
window.snowplow('addPlugin', 'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-browser-features@3.0.0/dist/index.umd.min.js', ['snowplowBrowserFeatures', 'BrowserFeaturesPlugin']);
window.snowplow('trackPageView');
```

This will queue all track events until the plugin is loaded, or 5000 milliseconds, whichever is sooner. Just we recommend with `sp.js` we suggest you rename and host the plugin JavaScript files yourself.

#### sp.lite.js

We also now publish a much smaller version of the JavaScript Tracker, called `sp.lite.js`. You can find it on the CDNs and in the GitHub Releases just like the regular `sp.js`. This has a limited featureset but is roughly half the size of `sp.js`.

Included in `sp.lite.js` is: Page View, Self Describing and Structured Event tracking as well as Activity Tracking and Anonymous Tracking. All other features can be loaded as separate plugins. So if you wanted sp.lite.js with Form tracking, you could do this:

```html
<script type="text/javascript" async=1>
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[]; p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments) };p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1; n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","https://cdn.jsdelivr.net/npm/@snowplow/javascript-tracker@3.0.0-beta.4/dist/sp.lite.js","snowplow"));

window.snowplow('newTracker', 'sp', {});
window.snowplow('addPlugin', 'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-form-tracking@3.0.0/dist/index.umd.min.js', ['snowplowFormTracking', 'FormTrackingPlugin']);
window.snowplow('enableFormTracking');
</script>
```

#### Build your own sp.js

It's now easier than ever to build your own `sp.js` to include _just_ the features you want!

To do this, you'll need to install [git](https://git-scm.com/) and [Node.js](https://nodejs.org/en/) 10, 12 or 14 (at the time of writing) then open a Terminal or Command Prompt and run the following:

```bash
$ git clone https://github.com/snowplow/snowplow-javascript-tracker.git
$ npm install -g @microsoft/rush
$ rush update
```

Then open the file `/trackers/javascript-tracker/tracker.config.ts` in your favourite text editor and flip the `true` values to `false` for features which you don't require.

Then run:

```bash
$ rush build
```

Once complete (it might take a minute or two), you'll find your brand new `sp.js` at: `/trackers/javascript-tracker/dist/sp.js` along with a new sourcemap `sp.js.map` which we suggest hosting together for a better developer experience.

---

# Migrating from web trackers v3 to v4
> Upgrade to JavaScript tracker v4 with changes to cookie handling, async behavior, and plugin updates.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/migration-guides/v3-to-v4-migration-guide/

Version 4 is a major release with several breaking changes to defaults and behavior. We have tried to keep the APIs compatible with v3 where possible to reduce code changes required.

> **Note:** These code snippets are for the Javascript (tag) tracker, but the same changes apply for the Browser (npm) tracker.

## Behavior changes

### `cookieDomain` and `discoverRootDomain` handling

`discoverRootDomain` is a setting to enable automatic detection of the closest-to-root domain possible to set cookies on. This is important to allow consistent Domain User ID and Domain Session ID values between pages on different subdomains of the same domain. Without this setting, the root domain value needs to be set explicitly with the `cookieDomain` setting in order to be shared correctly between subdomains. If neither are set, the cookie gets set against the current domain of the page and is not shared between any other subdomains.

In earlier versions, `discoverRootDomain` was disabled by default. Enabling the setting caused the SDK to ignore any configured `cookieDomain` and prefer the `discoverRootDomain`-derived value instead.

In v4, `discoverRootDomain` is now enabled by default, but will defer to the value in `cookieDomain` if it is set. This means previously correct configurations may now use the incorrect domain for updating cookies, and you may need to update your `newTracker` code to account for this change.

- If you previously had `discoverRootDomain: true`, you can now remove this setting and should remove any `cookieDomain` configuration.
- If you previously had `cookieDomain` configured and not `discoverRootDomain`, no change is required.
- If you previously had neither `cookieDomain` nor `discoverRootDomain` set, you will need to add `discoverRootDomain: false` to maintain the previous behavior.

**JavaScript (v3):**

```javascript
// v3: ignored cookieDomain, always used discoverRootDomain
window.snowplow('newTracker', 'sp', {
  cookieDomain: 'example.com',
  discoverRootDomain: true,
});

// v3: explicit cookieDomain, discoverRootDomain disabled
window.snowplow('newTracker', 'sp', {
  cookieDomain: 'example.com',
  discoverRootDomain: false, // or not set
});

// v3: no domain config
window.snowplow('newTracker', 'sp', {});
```

**JavaScript (v4):**

```javascript
// v4: remove cookieDomain, use default discoverRootDomain
window.snowplow('newTracker', 'sp', {});

// v4: explicit cookieDomain, preferred over discoverRootDomain
window.snowplow('newTracker', 'sp', {
  cookieDomain: 'example.com',
});

// v4: for no domain, explicitly opt-out of discoverRootDomain
window.snowplow('newTracker', 'sp', {
  discoverRootDomain: false,
});
```

***

### Async cookie access

Whereas in v3, the tracker updated user and session information in cookies during the track call (e.g., `trackPageView`), v4 makes this asynchronous. The cookies are updated within 10ms of the track call and the read cookie values are cached for 50ms. This strategy makes the track function take less time and avoid blocking the main thread of the app.

There is an option to keep the behavior from v3 and update cookies synchronously. This can be useful for testing purposes to ensure that the cookies are written before the test continues. It also has the benefit of making sure that the cookie is correctly set before session information is used in events. The downside is that it is slower and blocks the main thread.

[Read more about the synchronous cookie write option.](/docs/sources/web-trackers/configuring-how-events-sent/#synchronous-cookie-writes)

### Removed Beacon API, introduced `keepalive`

Version 4 removes the option to set `beacon` as the method for making requests to the Snowplow Collector. This is due to the use of the fetch API instead of the XMLHttpRequest API for making HTTP requests.

As an alternative to the Beacon API, there is a new keepalive option available from the fetch API. It indicates that the request should be allowed to outlive the webpage that initiated it. It enables requests to the Snowplow Collector to complete even if the page is closed or navigated away from.

[Read more about the option here.](/docs/sources/web-trackers/configuring-how-events-sent/#keepalive-option-for-collector-requests)

### `credentials` instead of `withCredentials`

Also related to the move to fetch instead of XMLHttpRequest, we have changed the `withCredentials` configuration option. It is now called `credentials` and has values that reflect the option in the fetch API.

[Read more about the option here.](/docs/sources/web-trackers/configuring-how-events-sent/#disabling-sending-credentials-with-requests).

### `os_timezone` detection

The JavaScript and Browser trackers will now attempt to populate the `os_timezone` field by querying the [`Intl.DateTimeFormat`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat) API in supported browsers.

Previously, this value was populated by heuristics in the [Timezone](/docs/sources/web-trackers/tracking-events/timezone-geolocation/) plugin.

The new method may produce different results and only supports more modern browsers. The Timezone plugin is still supported, but no longer bundled in `sp.js` by default. Using the plugin will override the new default value.

### Default for `encodeBase64`

The [`encodeBase64` setting](/docs/sources/web-trackers/configuring-how-events-sent/#base64-encoding) controls whether self describing JSON data in entities and self describing events are encoded with URL-safe [base64 encoding](https://en.wikipedia.org/wiki/Base64) or as JSON strings.

When sending events via the `GET` method, this is recommended because the JSON string would require URL-encoding to safely transmit, which typically results in a larger payload than with the \~33% size overhead of base64-encoded JSON data, which is inherently URL-safe.

For `POST` requests however, the overhead of base64 encoding makes the payload size larger and makes debugging more difficult as the payload isn't human-readable, and has no safety benefits.

For example:

```javascript
/* Actual JSON, length = 211 */
{"schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0","data":[{"schema":"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0","data":{"id":"0edd614e-975d-4f3a-9cd9-af123ac35068"}}]}

/* URI-encoded JSON, length = 289 */
"%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Fcontexts%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%5B%7B%22schema%22%3A%22iglu%3Acom.snowplowanalytics.snowplow%2Fweb_page%2Fjsonschema%2F1-0-0%22%2C%22data%22%3A%7B%22id%22%3A%220edd614e-975d-4f3a-9cd9-af123ac35068%22%7D%7D%5D%7D"

/* Base64-encoded JSON, length = 282 */
"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvd2ViX3BhZ2UvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsiaWQiOiIwZWRkNjE0ZS05NzVkLTRmM2EtOWNkOS1hZjEyM2FjMzUwNjgifX1dfQ"

/* JSON-string-encoded JSON, length = 229 */
"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"0edd614e-975d-4f3a-9cd9-af123ac35068\"}}]}"
```

In v3, `encodeBase64` defaulted to `true` if not explicitly configured. In v4, if not configured explicitly, the setting will now default to `false` if the [`eventMethod`](/docs/sources/web-trackers/configuring-how-events-sent/#network-protocol-and-method) is `post` (the default), but remains `true` if using the `get` method.

If your Collector is hosted behind a Web Application Firewall and custom entities/events include content that may be classified as risky (e.g. content containing HTML code) this may trigger false positives and cause the events to be rejected by the firewall; re-enabling base64 encoding should obfuscate the payload again and allow successful delivery.

Per the [tracker protocol](/docs/events/), the payload fields used for base-64-encoded data differs from plain JSON-encoded data: when upgrading to v4, plugins or tests relying on payload data being in `cx` or `ue_px` may now require updating to check the non-base64-encoded equivalents in `co` and `ue_pr`. Tracker protocol-compliant processors such as the Enricher should already support both fields and require no changes.

### Dropped support for older browsers

The support for the following browsers versions has been dropped:

- Drop IE 11 and Safari 10 support.

## Plugin changes

Plugin APIs are forwards compatible with v3 plugins, so the plugins that are no longer maintained can still be used with v4 if required. v4 plugins that don't use the new `filter` API are also backwards compatible with v3 if you only need some of the new functionality.

### Removed plugins

The following plugins are no longer maintained and are no longer available in v4. The v3 plugins should still work with v4, but will no longer be officially supported.

- [Browser Features](/docs/sources/web-trackers/previous-versions/web-trackers-v3/plugins/#browser-features-plugin-deprecated)
- [Classic Consent](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/consent-gdpr/original/)
- [Classic Ecommerce](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/ecommerce/original/)
- [Optimizely](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/optimizely/#optimizely-classic)

### Bundled plugin changes

The list of plugins included in the default `sp.js` JavaScript Tracker bundle has changed.

The following plugins are no longer included by default:

- [Client Hints](/docs/sources/web-trackers/tracking-events/client-hints/)
- [Classic Consent](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/consent-gdpr/original/)
- [Classic Ecommerce](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/ecommerce/original/)
- [Enhanced Ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/enhanced/)
- [Timezone Detection](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)
- [Optimizely](/docs/sources/web-trackers/previous-versions/web-trackers-v3/tracking-events/optimizely/#optimizely-classic)
- [Optimizely X](/docs/sources/web-trackers/tracking-events/optimizely/)
- [Performance Timing](/docs/sources/web-trackers/tracking-events/timings/#performance-timing-plugin-original)

The following plugins are now included by default:

- [Button Click Tracking](/docs/sources/web-trackers/tracking-events/button-click/)
- [Enhanced Consent](/docs/sources/web-trackers/tracking-events/consent-gdpr/)
- [Snowplow Ecommerce](/docs/sources/web-trackers/tracking-events/ecommerce/)
- [Performance Navigation Timing](/docs/sources/web-trackers/tracking-events/timings/)
- [Web Vitals](/docs/sources/web-trackers/tracking-events/web-vitals/)

To keep using old plugins, they will have to be [explicitly installed](/docs/sources/web-trackers/plugins/configuring-tracker-plugins/) using `addPlugins` or built into a custom bundle.

### Plugin behavior updates

#### Button click

- The plugin now uses `capture`-phase event listeners; button clicks that were previously untracked due to ancestor elements stopping event propagation are now more likely to track correctly
- Buttons within Custom Components and open-mode shadow trees should now be trackable but were previously ignored

#### Form tracking

- The plugin now uses document-level event listeners rather than per-element event listeners
- Because of the above, re-calling `enableFormTracking` is no longer required to detect forms dynamically added to the page
- The plugin now uses `capture`-phase event listeners; previously it relied on `target` or `bubble`-phase events
- Forms within Custom Components and open-mode shadow trees should now be trackable but were previously ignored; field focus must first occur before `change` and `submit` can be detected correctly
- Field transform functions can now return `null` values

#### GA cookies

- The [GA Cookies plugin](/docs/sources/web-trackers/tracking-events/ga-cookies/) now tracks in GA4 mode rather than Universal Analytics mode. This involves a [different schema](http://iglucentral.com/?q=cookies); the newer GA4 schema (`com.google.ga4/cookies`) is now the default, rather than the classic/Universal Analytics schema (`com.google.analytics/cookies`).
- To restore the old behavior, specify configuration for the plugin to enable UA support and disable GA4 support.

#### HTML5 media

- Plugin now wraps the Snowplow Media plugin, uses v2 Media Tracking schemas
- The v3 `enableMediaTracking` method has been removed, in its place is `startHtml5MediaTracking`
- Media tracking can now be stopped with the `endHtml5Tracking` method

#### Link click

- The plugin now uses document-level event listeners rather than per-element event listeners
- Because of the above, `refreshLinkClickTracking` is no longer required, is deprecated, and has no effect; new links will be tracked automatically
- The plugin now uses `capture`-phase event listeners; previously it relied on `target` or `bubble`-phase events
- Links within Custom Components and open-mode shadow trees should now be trackable but were previously ignored
- The manual `trackLinkClick` method can now accept an element to build a payload from, rather than requiring the called to construct the event payload
- Link click tracking now also tracks links with empty `href` attributes and assigns `about:invalid` as the `targetUrl` to avoid the event failing schema validation. Such links will also be reported as warnings in the console.

#### YouTube media

- Plugin now wraps the Snowplow Media plugin, uses v2 Media Tracking schemas
- New `startYouTubeTracking` and `endYouTubeTracking` methods to align with existing v2 plugins
- `enableYouTubeTracking` now wraps `startYouTubeTracking` for v3 compatibility; a new `disableYouTubeTracking` method exists as an equivalent for `endYouTubeTracking`
- The new events use new schemas, continue using v3 to maintain consistent tracking
- The plugin should now work correctly alongside other users of the iFrame API

---

# Configure web tracker plugins
> Install and configure plugins for the JavaScript tag and Browser npm versions of the Snowplow web tracker.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/plugins/configuring-tracker-plugins/

The Snowplow web trackers allow extension via plugins. There are a number of [out-of-the-box Snowplow plugins](/docs/sources/web-trackers/plugins/), but you can also [build your own](/docs/sources/web-trackers/plugins/creating-your-own-plugins/).

## Install plugins

The installation method depends on which version of the web tracker you are using.

**JavaScript (tag):**

The [JavaScript tracker distributions](/docs/sources/web-trackers/tracker-setup/) include a full-featured `sp.js` version with [some plugins](/docs/sources/web-trackers/plugins/), and a smaller `sp.lite.js` version with no plugins.

If the plugin you need is present in your `sp.js` version, you can use it without additional installation. Otherwise, you can either build a custom `sp.js` bundle with the required plugins included, or load plugins dynamically at runtime.

While plugins can be loaded dynamically, self-hosting the additional files adds complexity, and additional resource requests can negatively impact page performance. Consider building a custom bundle with only the plugins you need.

**Browser (npm):**

The tracker doesn't come with any plugins pre-installed.

Install plugins as npm packages. For example, to use the Button Click tracking plugin:

```bash
npm install @snowplow/browser-plugin-button-click-tracking
# or
yarn add @snowplow/browser-plugin-button-click-tracking
# or
pnpm add @snowplow/browser-plugin-button-click-tracking
```

***

## Make plugins available to the tracker

You can make plugins available to the tracker in different ways.

**JavaScript (tag):**

Bundled plugins are automatically available, and their methods are exposed directly.

You can configure bundled plugins that add entities during initialization via the [`contexts` option](/docs/sources/web-trackers/tracker-setup/initialization-options/):

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  contexts: {
    performanceNavigationTiming: true
  }
});

// Methods from bundled plugins are available directly
snowplow('enableFormTracking');
```

**Browser (npm):**

Include plugins in the `plugins` array when creating the tracker:

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { PerformanceNavigationTimingPlugin } from '@snowplow/browser-plugin-performance-navigation-timing';
import { FormTrackingPlugin, enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

newTracker('sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  plugins: [
    PerformanceNavigationTimingPlugin(),
    FormTrackingPlugin()
]
});

// For plugins that add entities, no additional configuration is needed
// Adding the plugin makes it active

// Use the methods provided by the plugins
enableFormTracking();
```

***

You can add plugins dynamically at any time. This also works for [inline plugins](/docs/sources/web-trackers/plugins/creating-your-own-plugins/#inline-plugins) that are part of your codebase rather than published separately.

**JavaScript (tag):**

Use `addPlugin` with a URL to load the plugin dynamically.

```javascript
snowplow(
  'addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-performance-navigation-timing@latest/dist/index.umd.min.js",
  ["snowplowPerformanceNavigationTiming", "PerformanceNavigationTimingPlugin"]
);
```

**Browser (npm):**

Use the `addPlugin` function:

```javascript
import { newTracker, addPlugin } from '@snowplow/browser-tracker';
import { PerformanceNavigationTimingPlugin } from '@snowplow/browser-plugin-performance-navigation-timing';

newTracker('sp', '{{collector_url_here}}', {
  appId: 'my-app-id'
});

// Later...
addPlugin({ plugin: PerformanceNavigationTimingPlugin() });
```

***

## Build a custom bundle

If you use the JavaScript (tag) tracker and want to reduce file size or include only specific plugins, you can build a custom `sp.js` bundle.

To do this, clone the [tracker repository](https://github.com/snowplow/snowplow-javascript-tracker) and update the plugins config file to include the plugins you want. Then build the tracker to create your custom `sp.js`.

You'll need git and Node.js (version 14+) installed:

```bash
git clone https://github.com/snowplow/snowplow-javascript-tracker.git
cd snowplow-javascript-tracker
npx @microsoft/rush update
```

Edit `trackers/javascript-tracker/tracker.config.ts` and set plugins to `true` or `false` based on your needs.

```ts
/* By default included plugins in sp.js */
export const adTracking = true;
export const enhancedConsent = true;
export const errorTracking = true;
export const formTracking = true;
export const gaCookies = true;
export const linkClickTracking = true;
export const performanceNavigationTiming = true;
export const siteTracking = true;
export const snowplowEcommerceTracking = true;
export const buttonClickTracking = true;
export const webVitals = true;

/* By default excluded plugins in sp.js */
export const clientHints = false;
export const mediaTracking = false;
export const optimizelyX = false;
export const youtubeTracking = false;
export const snowplowMedia = false;
export const vimeoTracking = false;
export const privacySandbox = false;
export const eventSpecifications = false;
export const geolocation = false;
export const timezone = false;
export const elementTracking = false;

/* Deprecated */
export const enhancedEcommerce = false;
export const performanceTiming = false;
```

Once you've edited the file, run the build command:

```bash
npx @microsoft/rush build
```

Your custom `sp.js` will be at `trackers/javascript-tracker/dist/sp.js`, along with a sourcemap `sp.js.map`. Host both files together for a better developer experience.

## Include custom plugins in a bundle

To include your own plugins or third-party plugins in a custom JavaScript tracker bundle, you need to make the following changes:

1. Add the plugin as a dependency in `trackers/javascript-tracker/package.json`
2. Import the plugin in `trackers/javascript-tracker/src/features.ts`
3. Add conditional logic in `features.ts` to check a feature flag for the plugin
4. Add a feature flag in both `tracker.config.ts` and `tracker.lite.config.ts`
5. Run `npx @microsoft/rush update` and `npx @microsoft/rush build`

For example, to embed the [SimpleContextTemplate](https://github.com/snowplow-incubator/snowplow-browser-plugin-simple-template) plugin:

```diff
diff --git a/trackers/javascript-tracker/package.json b/trackers/javascript-tracker/package.json
--- a/trackers/javascript-tracker/package.json
+++ b/trackers/javascript-tracker/package.json
@@ -59,7 +59,8 @@
     "@snowplow/browser-tracker-core": "workspace:*",
     "@snowplow/tracker-core": "workspace:*",
     "tslib": "^2.3.1",
-    "@snowplow/browser-plugin-enhanced-consent": "workspace:*"
+    "@snowplow/browser-plugin-enhanced-consent": "workspace:*",
+    "snowplow-browser-plugin-simple-template": "snowplow-incubator/snowplow-browser-plugin-simple-template"
   },

diff --git a/trackers/javascript-tracker/src/features.ts b/trackers/javascript-tracker/src/features.ts
--- a/trackers/javascript-tracker/src/features.ts
+++ b/trackers/javascript-tracker/src/features.ts
@@ -47,6 +47,7 @@ import * as SiteTracking from '@snowplow/browser-plugin-site-tracking';
 import * as SnowplowEcommerce from '@snowplow/browser-plugin-snowplow-ecommerce';
 import * as MediaTracking from '@snowplow/browser-plugin-media-tracking';
 import * as YouTubeTracking from '@snowplow/browser-plugin-youtube-tracking';
+import * as SimpleContext from 'snowplow-browser-plugin-simple-template';
 import * as plugins from '../tracker.config';
 import { BrowserPlugin } from '@snowplow/browser-tracker-core';
 import { JavaScriptTrackerConfiguration } from './configuration';
@@ -196,5 +197,10 @@ export function Plugins(configuration: JavaScriptTrackerConfiguration) {
     activatedPlugins.push([EnhancedConsentPlugin(), apiMethods]);
   }

+  if (plugins.simpleContext) {
+    const { SimpleContextPlugin, ...apiMethods } = SimpleContext;
+    activatedPlugins.push([SimpleContextPlugin(), apiMethods]);
+  }
+
   return activatedPlugins;
 }

diff --git a/trackers/javascript-tracker/tracker.config.ts b/trackers/javascript-tracker/tracker.config.ts
--- a/trackers/javascript-tracker/tracker.config.ts
+++ b/trackers/javascript-tracker/tracker.config.ts
@@ -48,3 +48,4 @@ export const browserFeatures = false;
 export const mediaTracking = false;
 export const youtubeTracking = false;
 export const enhancedConsent = false;
+export const simpleContext = true;

diff --git a/trackers/javascript-tracker/tracker.lite.config.ts b/trackers/javascript-tracker/tracker.lite.config.ts
--- a/trackers/javascript-tracker/tracker.lite.config.ts
+++ b/trackers/javascript-tracker/tracker.lite.config.ts
@@ -48,3 +48,4 @@ export const browserFeatures = false;
 export const mediaTracking = false;
 export const youtubeTracking = false;
 export const enhancedConsent = false;
+export const simpleContext = false;
```

With these changes, the resulting `sp.js` will include the custom plugin, while `sp.lite.js` will not.

---

# Create your own web tracker plugins
> Build custom plugins to extend the web tracker with your own context entities, events, and filtering logic.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/plugins/creating-your-own-plugins/

The Snowplow JavaScript Trackers v3 allow extension via plugins. There are a number of official Snowplow plugins, but we also encourage building your own. You can either include them directly in your codebase or tag management tool, or you could publish them to npm as public packages that the whole community can use.

## Plugin Interface

The Browser Plugins conform to a common interface which describes a number of functions you can optionally use to create your own plugin.

```javascript
interface BrowserPlugin {
 /**
 Called when the plugin is initialised during the Tracker construction
 *
 @remark
 Use to capture the specific Tracker instance for each instance of a Browser Plugin
 */
 activateBrowserPlugin?: (tracker: BrowserTracker) => void;

 /**
 Called when the plugin is initialised during the trackerCore construction
 *
 @remark
 Use to capture the specific core instance for each instance of a core plugin
 */;
 activateCorePlugin?: (core: TrackerCore) => void;

 /**
 Called just before the trackerCore callback fires
 @param payloadBuilder - The payloadBuilder which will be sent to the callback, can be modified
 */
 beforeTrack?: (payloadBuilder: PayloadBuilder) => void;

 /**
 Called just after the trackerCore callback fires
 @param payload - The final built payload
 */
 afterTrack?: (payload: Payload) => void;

 /**
 Called before the payload is sent to the callback to decide whether to send the payload or skip it
 @param payload - The final event payload, can't be modified.
 @returns True if the payload should be sent, false if it should be skipped
 */
 filter?: (payload: Payload) => boolean;

 /**
 Called when constructing the context for each event
 Useful for adding additional context to events
 */
 contexts?: () => SelfDescribingJson[];

 /**
 Passed a logger instance which can be used to send log information
 to the active logger
 */
 logger?: (logger: Logger) => void;
}
```

## Plugin Templates

If you'd like to build your own plugins and publish them on NPM then we've created two template repositories to help you get started.

### Simple Context Plugin Template

This template shows how you can add a context to every single event that is emitted from a tracker which is using this plugin. The template is written in TypeScript, so you'll need to build it before using it and/or publishing. We've also included some example Jest tests to help you write tests for your plugin.

<https://github.com/snowplow-incubator/snowplow-browser-plugin-simple-template>

The README should guide you through building, testing, publishing and using a plugin built with this template.

### Advanced Plugin Template

This template shows how you can use all the available functions of a plugin, as well as how to make the plugin only respond to certain trackers in a multi-tracker setup. The template is written in TypeScript, so you'll need to build it before using it and/or publishing. We've also included some example Jest tests to help you write tests for your plugin.

<https://github.com/snowplow-incubator/snowplow-browser-plugin-advanced-template>

The README should guide you through building, testing, publishing and using a plugin built with this template.

## Inline Plugins

You might not want to publish a package for your plugin but directly include the code in your codebase. In that case, you can pass the plugin directly into the tracker when you call `newTracker`.

### Example of adding context entities

**JavaScript (tag):**

```javascript
const myPlugin = {
  SimpleContextPlugin: function () {
    return {
      contexts: () => {
        return [
          {
            schema: 'iglu:com.acme/my_context/jsonschema/1-0-0',
            data: {
              property: 'value',
            },
          },
        ];
      },
    };
  },
  trackMyEvent: function (event) {
    // Extend the API and track something here (see advanced-template above)
    console.log(event);
  }
};

window.snowplow('addPlugin:sp1', myPlugin, 'SimpleContextPlugin');
window.snowplow('trackMyEvent', { eventProp: 'value' });
```

**Browser (npm):**

```javascript
import { addPlugin } from '@snowplow/browser-tracker';

const myPlugin = {
  contexts: () => {
    return [
      {
        schema: 'iglu:com.acme/my_context/jsonschema/1-0-0',
        data: {
          property: 'value',
        },
      },
    ];
  },
};

addPlugin(myPlugin, ['sp1'])
```

***

### Example of filtering events

The following example shows a plugin that filters the tracked events and only allows page view events – all other tracked events are discarded.

**JavaScript (tag):**

```javascript
const myPlugin = {
  SimpleFilterPlugin: function () {
    return {
      filter: (payload) => {
        return payload.e === 'pv';
      },
    };
  },
};

window.snowplow('addPlugin:sp1', myPlugin, 'SimpleFilterPlugin');
window.snowplow('trackPageView');
```

**Browser (npm):**

```javascript
import { newTracker } from '@snowplow/browser-tracker';

const myPlugin = {
  filter: (payload) => {
    return payload.e === 'pv';
  },
};

newTracker('sp1', '{{COLLECTOR_URL}}', { plugins: [myPlugin] });
```

***

---

# Web tracker plugins
> Extend the web tracker with plugins for tracking ads, ecommerce, media, forms, and more.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/plugins/

The JavaScript Tracker is based around a plugin architecture which allows new functionality to be added to the tracker. There are a number of Snowplow maintained plugins, however you are also free to [build your own](/docs/sources/web-trackers/plugins/creating-your-own-plugins/) or leverage community plugins too.

## All plugins

The following table lists the Snowplow plugins in alphabetical order. It shows what kinds of data are created using them, and their [distribution](/docs/sources/web-trackers/tracker-setup/) for the JavaScript tracker. Read more about configuring plugins [here](/docs/sources/web-trackers/plugins/configuring-tracker-plugins/) or on the individual pages.

If you're using the JavaScript tracker with the full `sp.js` distribution and your plugin is included, no further installation or initialization is required. You can use it straight away.

| Plugin                                                                                                                    | Since version  | Creates             | Tracked              | `sp.js` | `sp.lite.js` | Package name                                   |
| ------------------------------------------------------------------------------------------------------------------------- | -------------- | ------------------- | -------------------- | ------- | ------------ | ---------------------------------------------- |
| [Ads](/docs/sources/web-trackers/tracking-events/ads/)                                                                    | 3.0.0          | Events              | Manual               | ✅       | ❌            | `browser-plugin-ad-tracking`                   |
| [Button click](/docs/sources/web-trackers/tracking-events/button-click/)                                                  | 3.18.0         | Events              | Automatic            | ✅       | ❌            | `browser-plugin-button-click-tracking`         |
| [Client Hints](/docs/sources/web-trackers/tracking-events/client-hints/)                                                  | 3.0.0          | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-client-hints`                  |
| [Consent (Enhanced)](/docs/sources/web-trackers/tracking-events/consent-gdpr/)                                            | 3.8.0          | Events and entities | Manual               | ✅       | ❌            | `browser-plugin-enhanced-consent`              |
| [Debugger](/docs/sources/web-trackers/testing-debugging/)                                                                 | 3.0.0          | Other               | n/a                  | ❌       | ❌            | `browser-plugin-debugger`                      |
| [Ecommerce (Snowplow)](/docs/sources/web-trackers/tracking-events/ecommerce/)                                             | 3.8.0          | Events and entities | Manual               | ✅       | ❌            | `browser-plugin-snowplow-ecommerce`            |
| [Ecommerce (Enhanced)](/docs/sources/web-trackers/tracking-events/ecommerce/enhanced/) - Deprecated                       | 3.0.0 - 3.24.6 | Events              | Manual               | ❌       | ❌            | `browser-plugin-enhanced-ecommerce`            |
| [Errors](/docs/sources/web-trackers/tracking-events/errors/)                                                              | 3.0.0          | Events              | Manual and automatic | ✅       | ❌            | `browser-plugin-error-tracking`                |
| [Element visibility](/docs/sources/web-trackers/tracking-events/element-tracking/)                                        | 4.6.0          | Events              | Automatic            | ❌       | ❌            | `browser-plugin-element-tracking`              |
| [Event specifications](/docs/sources/web-trackers/tracking-events/event-specifications/)                                  | 3.23.0         | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-event-specifications`          |
| [Forms](/docs/sources/web-trackers/tracking-events/form-tracking/)                                                        | 3.0.0          | Events              | Automatic            | ✅       | ❌            | `browser-plugin-form-tracking`                 |
| [GA cookies](/docs/sources/web-trackers/tracking-events/ga-cookies/)                                                      | 3.0.0          | Entities            | Automatic            | ✅       | ❌            | `browser-plugin-ga-cookies`                    |
| [Geolocation](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)                                           | 3.0.0          | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-geolocation`                   |
| [Kantar Focal Meter](/docs/sources/web-trackers/tracking-events/focalmeter/)                                              | 3.16.0         | Other               | n/a                  | ❌       | ❌            | `browser-plugin-focalmeter@focalmeter_plugin`  |
| [Link click](/docs/sources/web-trackers/tracking-events/link-click/)                                                      | 3.0.0          | Events              | Automatic            | ✅       | ❌            | `browser-plugin-link-click-tracking`           |
| [Media (Snowplow)](/docs/sources/web-trackers/tracking-events/media/)                                                     | 3.12.0         | Events and entities | Manual               | ❌       | ❌            | `browser-plugin-media`                         |
| [Media (HTML)](/docs/sources/web-trackers/tracking-events/media/html5/)                                                   | 3.2.0          | Events and entities | Automatic            | ❌       | ❌            | `browser-plugin-media-tracking`                |
| [Media (Vimeo)](/docs/sources/web-trackers/tracking-events/media/vimeo/)                                                  | 3.14.0         | Events and entities | Automatic            | ❌       | ❌            | `browser-plugin-vimeo-tracking`                |
| [Media (Youtube)](/docs/sources/web-trackers/tracking-events/media/youtube/)                                              | 3.2.0          | Events and entities | Automatic            | ❌       | ❌            | `browser-plugin-youtube-tracking`              |
| [Optimizely X](/docs/sources/web-trackers/tracking-events/optimizely/)                                                    | 3.0.0          | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-optimizely-x`                  |
| [Performance navigation timing](/docs/sources/web-trackers/tracking-events/timings/)                                      | 3.10.0         | Entities            | Automatic            | ✅       | ❌            | `browser-plugin-performance-navigation-timing` |
| [Performance timing](/docs/sources/web-trackers/tracking-events/timings/#performance-timing-plugin-original) - Deprecated | 3.0.0 - 3.24.6 | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-performance-timing`            |
| [Privacy Sandbox](/docs/sources/web-trackers/tracking-events/privacy-sandbox/)                                            | 3.14.0         | Entities            | Automatic            | ❌       | ❌            | `browser-plugin-privacy-sandbox`               |
| [Screen views](/docs/sources/web-trackers/tracking-events/screen-views/)                                                  | 4.2.0          | Events              | Combination          | ❌       | ❌            | `browser-plugin-screen-tracking`               |
| [Site search](/docs/sources/web-trackers/tracking-events/site-search/) in Site plugin                                     | 3.0.0          | Events              | Manual               | ✅       | ❌            | `browser-plugin-site-tracking`                 |
| [Social media interactions](/docs/sources/web-trackers/tracking-events/social-media/) in Site plugin                      | 3.0.0          | Events              | Manual               | ✅       | ❌            | `browser-plugin-site-tracking`                 |
| [Timing](/docs/sources/web-trackers/tracking-events/timings/generic/) in Site plugin                                      | 3.0.0          | Events              | Manual               | ✅       | ❌            | `browser-plugin-site-tracking`                 |
| [Timezone](/docs/sources/web-trackers/tracking-events/timezone-geolocation/) - Legacy                                     | 3.0.0          | Other               | Automatic            | ❌       | ❌            | `browser-plugin-timezone`                      |
| [WebView](/docs/sources/web-trackers/tracking-events/webview/)                                                            | 4.3.0          | Other               | Automatic            | ❌       | ❌            | `browser-plugin-webview`                       |
| [Web vitals](/docs/sources/web-trackers/tracking-events/web-vitals/)                                                      | 3.13.0         | Events              | Automatic            | ✅       | ❌            | `browser-plugin-web-vitals`                    |

You can find the plugins code [here](https://github.com/snowplow/snowplow-javascript-tracker/tree/master/plugins) and also search for them on [npmjs.com](https://www.npmjs.com/).

---

# Quick start guide for the web trackers
> Get started quickly with the JavaScript or Browser tracker to track page views and user activity on your website.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/quick-start-guide/

Follow these instructions to quickly implement a Snowplow web tracker with default configuration, and track a page view.

**JavaScript (tag):**

Getting started with sending events using the JavaScript tracker is very similar to other web analytics vendors like Google Analytics, Adobe Analytics, etc.

The process involves the following high level steps:

- Download the latest version of the Snowplow JavaScript tracker file, `sp.js`, which can be found [here](https://github.com/snowplow/snowplow-javascript-tracker/releases).
- If you are already hosting static files somewhere on your own domain, it should just be a matter of downloading and adding the `sp.js` file. Otherwise you can follow our [guides for self hosting](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/), use another method of your choice, or leverage a [Third Party CDN](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/third-party-cdn-hosting/) (useful for evaluation or testing).
- Once you have a JS tracker available, you can add the tag snippet to your site. There are also alternative options described below for adding the tracker to your website.
- If manually inserting the tag into your website or tag management solution: Snowplow CDI customers can generate a tag snippet in Snowplow Console [here](https://console.snowplowanalytics.com/tag-generator). Other users can use and edit the standard tag [here](/docs/sources/web-trackers/tracker-setup/).

```javascript
 ;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[]; p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments) };p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1; n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));
```

- Once you’ve generated your tag add it to all the pages you’d like to track:

  - Place the tag directly into your codebase. Typically this will be placed into the `<head>` element of your page or in a similar, suitable, location if using a Single Page Application framework.
    - The tag loads asynchronously to avoid slowing down page load times. Tracking calls made before the script loads are automatically queued and processed once the tracker initializes.
  - Load the tag using a Tag Management solution such as Google Tag Manager, usually triggered on page load.

- Configure an instance of the tracker by calling `newTracker` with your desired properties.

```javascript
window.snowplow('newTracker', 'sp1', '{{collector_url}}', {
  appId: 'my-app-id'
})
```

- Then you can use the track methods to send some events. You can send a Page View event to all initialised trackers with just:

```javascript
window.snowplow('trackPageView');
```

Rather than adding the tag snippet directly, you may wish to use an alternative option for loading the JavaScript Tracker.

- Users of Google Tag Manager can use the [Snowplow Analytics Custom Template](/docs/sources/google-tag-manager/).

- Use the Snowplow Plugin in the [analytics npm package](/docs/sources/web-trackers/tracker-setup/snowplow-plugin-for-analytics-npm-package/).

**Browser (npm):**

Getting started with sending events using the Browser Tracker will be familiar for anyone who is used to installing npm packages into their web apps and is designed to work with frameworks such as React, Angular and Vue.

The process involves the following high level steps:

- Install the `@snowplow/browser-tracker` package using your preferred package manager

  - `npm install @snowplow/browser-tracker`
  - `yarn add @snowplow/browser-tracker`
  - `pnpm add @snowplow/browser-tracker`

- You can then import this library into your application

```javascript
import { newTracker, trackPageView } from "@snowplow/browser-tracker";
```

- Configure an instance of the tracker by calling `newTracker(...)` with your desired properties. This will create a module level instance of your tracker. You don't need to keep a reference to it.

```javascript
newTracker('sp1', '{{collector_url}}', {
  appId: 'my-app-id',
  plugins: [ ],
})
```

- Then you can use the track methods to send some events. You can send a Page View event to all initialised trackers with just:

```javascript
trackPageView();
```

***

## What's tracked with the default configuration?

Using just the initialization snippets above won't result in any events, unless `trackPageView` is called.

However, we recommend that [activity tracking](/docs/sources/web-trackers/tracking-events/activity-page-pings/) (page pings) is enabled immediately following initialization.

**JavaScript (tag):**

```javascript
<!-- Snowplow starts plowing -->
<script type="text/javascript">
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[];
p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments)
};p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1;
n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));

snowplow('newTracker', 'sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
});

snowplow('enableActivityTracking', {
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

// snowplow('trackPageView') can then be called

</script>
<!-- Snowplow stops plowing -->
```

**Browser (npm):**

```javascript
import {
  newTracker,
  trackPageView
} from '@snowplow/browser-tracker';

newTracker('sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
});

enableActivityTracking({
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

// trackPageView() can then be called
```

***

Adding this code to your site will cause [page ping events](/docs/sources/web-trackers/tracking-events/activity-page-pings/) to be automatically tracked and sent via POST.

The events will all have the `webPage` entity attached, containing the [page view ID](/docs/sources/web-trackers/tracking-events/page-views/).

---

# Test, debug and troubleshoot the web trackers
> Test and debug your web tracker implementation using Snowplow Inspector, Snowplow Micro, and the debugger plugin.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/testing-debugging/

To prevent data loss and bad events, it's important to test and validate your tracking implementation. Snowplow provides two main tools for this, which work well together.

## Snowplow Inspector for browsers

Use our [browser extension](/docs/testing/snowplow-inspector/) to inspect event requests in the browser Developer Tools window.

## Snowplow Micro

A [lightweight Snowplow pipeline](/docs/testing/snowplow-micro/) ideal for sending test events into. You can [deploy it through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/) via Docker.

## Debugger plugin

The debugger plugin logs detailed information about tracked events to your browser's Developer Tools console. For each event, it displays:

- Event type
- Schema information for self-describing events and entities
- All entities attached to the event
- The complete event payload sent to the Collector

The plugin uses color-coded output to distinguish between different types of information, making it easier to scan through logged events.

> **Note:** You may need to enable **Verbose** logs in your Developer Tools console, as this plugin uses `console.debug` to output results.

![Example console output from the debugger plugin](/assets/images/Screenshot-2021-03-28-at-20.08.35-ca0922b97b07198a894a3627e34ff92b.png)

### Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                  |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)            |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-debugger@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-debugger@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-debugger@latest/dist/index.umd.min.js",
  ["snowplowDebugger", "DebuggerPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-debugger`
- `yarn add @snowplow/browser-plugin-debugger`
- `pnpm add @snowplow/browser-plugin-debugger`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { DebuggerPlugin } from '@snowplow/browser-plugin-debugger';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ DebuggerPlugin() ],
});
```

***

### Configuration

The `DebuggerPlugin` function accepts an optional `logLevel` parameter that controls the verbosity of console output.

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-debugger@latest/dist/index.umd.min.js",
  ["snowplowDebugger", "DebuggerPlugin"],
  [3] // Log level: 3 = debug (default)
);
```

**Browser (npm):**

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { DebuggerPlugin } from '@snowplow/browser-plugin-debugger';
import { LOG_LEVEL } from '@snowplow/tracker-core';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ DebuggerPlugin(LOG_LEVEL.debug) ],
});
```

***

The available log levels are:

| Level   | Value | Description                                         |
| ------- | ----- | --------------------------------------------------- |
| `none`  | 0     | Disables all logging                                |
| `error` | 1     | Logs errors only                                    |
| `warn`  | 2     | Logs errors and warnings                            |
| `debug` | 3     | Logs errors, warnings, and debug messages (default) |
| `info`  | 4     | Logs all messages                                   |

---

# Create a white-label JavaScript tracker build
> Build a customized version of sp.js with your own naming to avoid ad blockers and improve tracking reliability.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/creating-a-whitelabel-build/

Every time the snowplow tracker loads on the page, a queue-like object `GlobalSnowplowNamespace` is initialised on the global window scope. This object is then used to queue commands before the tracker snippet (sp.js) has fully loaded.

In some cases, like two Snowplow instances on the same page from different sources or different tracker versions, there might be a need to separate this queue to avoid conflicts and tracking inconsistencies. To help you with this, we provide a custom build option for our JavaScript tracker that will produce loader tags (tag.js & tag.min.js) and tracker snippets (sp.js & sp.lite.js) utilising a custom namespace.

To create the _white-label_ build, as we call it, you need to follow the steps as shown below:

1. Clone the JavaScript repository `git clone git@github.com:snowplow/snowplow-javascript-tracker.git`
2. Go through the [Maintainer quick start](https://github.com/snowplow/snowplow-javascript-tracker#maintainer-quick-start) to setup the required tooling.
3. Change directory to the JavaScript tracker project `cd ./trackers/javascript-tracker`.
4. Run the custom white-label build command `rushx build --whitelabel=MyWhitelabel` replacing MyWhitelabel with the namespace you want. This will replace all references of `GlobalSnowplowNamespace` with `MyWhitelabel`.
5. If the build finished successfully, you can retrieve the updated loader files from the `tags` folder and the updated tracker files from the `dist` folder.

> **Warning:** Since the build depends on your local copy of the Snowplow JavaScript repository, make sure you keep up to date with the latest developments and updating your files and build accordingly.

---

# Host the JavaScript tracker
> Self-host sp.js on your own domain for better security, performance, and control over tracker assets.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/

We recommend self-hosting the Snowplow JavaScript tracker, `sp.js`, as it has some definite advantages over using a third-party-hosted JavaScript:

1. Hosting your own JavaScript allows you to use your own JavaScript minification and asset pipelining approach (e.g. bundling all JavaScripts into one minified JavaScript)
2. As [Douglas Crockford](https://github.com/douglascrockford) put it about third-party JavaScripts: _“it is extremely unwise to load code from servers you do not control.”_
3. Renaming `sp.js` will ensure that Snowplow continues to function in the presence of ad/content blockers, which typically block `sp.js` (see e.g. [EasyPrivacy](https://easylist-downloads.adblockplus.org/easyprivacy.txt))

Below are guides for hosting the minified `sp.js` asset on Amazon Web Services and Google Cloud Platform. This is our recommended strategy for self hosting however there are other options available to self hosting `sp.js`, you may choose to bundle it into your application directly or host with a different provider as two examples.

The latest minified version of the Snowplow JavaScript Tracker, called `sp.js`, is available from the [JavaScript Tracker GitHub releases](https://github.com/snowplow/snowplow-javascript-tracker/releases). If you don't need all the functionality included in `sp.js`, there is also a `sp.lite.js` bundle for v3 that includes [fewer plugins](/docs/sources/web-trackers/plugins/). For more custom requirements, you can [build your own v3 bundle](/docs/sources/web-trackers/plugins/configuring-tracker-plugins/) using your own selection of plugins, which is only an option if you are self-hosting.

---

# Self Hosting the JavaScript Tracker on AWS
> Host sp.js on AWS CloudFront and S3 for fast, reliable tracker delivery to your website visitors.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/self-hosting-the-javascript-tracker-aws/

## Pre-requisites

For the purposes of this guide, we are going to assume that you want to serve the standard `sp.js` from CloudFront. To accomplish this, you will need the following:

- An account with [Amazon Web Services](http://aws.amazon.com/)
- S3 and CloudFront enabled within your AWS account

## Download the JavaScript tracker file

Navigate to <https://github.com/snowplow/snowplow-javascript-tracker/releases> and download the latest version of the Snowplow JavaScript Tracker sp.js file

## gzip and rename the file

- rename `sp.js` to a random 8 character string to reduce the chance of AdBlockers preventing the script from loading e.g. `gh7rnghq.js`
- `gzip` the file to reduce the file size and reduce associated cloud storage and egress costs.

From a terminal / command prompt window, navigate to where you have downloaded the file and run:

`gzip -c sp.js > gh7rnghq.js`

**N.B.** on Windows you may need to [download the gzip binaries](http://gnuwin32.sourceforge.net/packages/gzip.htm)

We will continue referring to the file as `sp.js` throughout this guide, however where `sp.js` is mentioned we are referring to your renamed and gzipped file.

## Uploading to Amazon Web Services

### Create a bucket for the JavaScript Tracker

Navigate to [S3 within AWS](https://s3.console.aws.amazon.com/s3/home) and then create a new bucket within your Amazon S3 account to store `sp.js`.

| Option                                           | Value                                       |
| ------------------------------------------------ | ------------------------------------------- |
| **Bucket Name**                                  | For example, `[company-name]-sp-js`         |
| **AWS Region**                                   | For example, `EU (London) eu-west-2`        |
| **Block Public Access settings for this bucket** | Deselect: ****Block _all_ public access**** |

You can leave all other settings as their defaults.

Click **Create Bucket**.

### Upload the JavaScript

You want to upload the **minified** version of the Snowplow JavaScript, which is called `sp.js`, that you downloaded in an earlier step. If you haven't downloaded it yet, you can obtain the latest version of the JavaScript from [Github releases](https://github.com/snowplow/snowplow-javascript-tracker/releases).

Now you're ready to upload the JavaScript Tracker into your S3 bucket.

1. Navigate to your new bucket by clicking on your Buckets name
2. Click `Create folder` and create a new folder which represents the version of the JavaScript Tracker you are uploading. This will help with any browser caching issues as you update your tracker in the future. If you downloaded Version 3.0.0 then create a folder called `3.0.0`
3. Navigate into your new folder, click **Upload**, click **Add Files** and browse to your file
4. Your file should now be present in the **Files and folders** section
5. Click on **Additoinal upload options** to drop down to extra settings
6. In the **Access control list (ACL)** pane, you must select **Read** within the **Objects** column on **Everyone (public access)** row. You may need to confirm you understand the effects of this change within the UI.
7. Before uploading, we must now we will configure the required metadata for a gzipped file and the cache control for Cloudfront.

#### Configuring metadata

First we will set the `Content-Encoding` header.

1. Scroll down to the **Metadata** section within the **Addtional upload options.**
2. Click **Add Metadata**
3. Add Type: **System defined**, Key: **Content-Encoding**, Value: **gzip**

Now we will add the `Cache-Control` header.

1. Click **Add Metadata**
2. Add Type: **System defined**, Key: **Cache-Control**, Value: **max-age=315360000**
3. Click **Edit metadata**

This sets your items to expire in 10 years, that is 10x365x24x60x60 = 315,360,000.

We recommend that you set the `Cache-Control max-age` property on the file. This property determines _both_ how long Cloudfront caches `sp.js` in its edge locations, and crucially, how long individual browsers cache `sp.js` before repinging Cloudfront for a fresh copy. By setting a long expiration date, you can reduce the number of browser requests for `sp.js`, which can significantly decrease your Cloudfront costs.

The only disadvantage of a long expiration is that you need to find a way to _force_ end users to fetch a fresh copy of `sp.js` when you upgrade to a newer version. However as you have created a versioned folder, this is easily managed by saving your new version to a new folder in your S3 bucket, and updating your Snowplow tags to point to the new version.

Your metadata should now look something like this:

| Type           | Key             | Value                  |
| -------------- | --------------- | ---------------------- |
| System defined | Cache-Control   | max-age=315360000      |
| System defined | Content-Type    | application/javascript |
| System defined | Cotent-Encoding | gzip                   |

Now scroll to the bottom and click **Upload** to upload the JavaScript file into your bucket. When done, you should have something like this:

![](/assets/images/s3-upload-2c02093f5772590d3bf33b0cd1f6caf0.png)

### Create your CloudFront distribution

Now you are ready to create the CloudFront distribution which will serve your JavaScript. Navigate to [CloudFront within AWS](https://console.aws.amazon.com/cloudfront/home).

1. Click **Create Distribution** and then **Get Started**
2. Use the table below to set all the required options

| Option                                   | Value                               |
| ---------------------------------------- | ----------------------------------- |
| Origin Domain Name                       | **\[Bucket Name].s3.amazonaws.com** |
| e.g. snowplow-static-js.s3.amazonaws.com |                                     |
| Other options                            | Leave as default                    |

Click **Create Distribution** and then you should see AWS beginning to create the distribution.

Note down your CloudFront distribution's **Domain Name** - e.g. `http://d1weib8el6blz6.cloudfront.net` You will need this later when you integrate Snowplow into your website.

### Testing your JavaScript file on CloudFront

Before testing, take a 10 minute coffee break (that's how long CloudFront takes to synchronize).

Done? Now just check that you can access your JavaScript file over both HTTP and HTTPS using a browser, `wget` or `curl`:

```text
http://{{SUBDOMAIN}}.cloudfront.net/{{VERSION}}/sp.js
https://{{SUBDOMAIN}}.cloudfront.net/{{VERSION}}/sp.js
```

If you have any problems, then double-check your CloudFront distribution's URL, and check the permissions on your `sp.js` file: it must have Read permissions for Everyone.

That's it - you now have a CloudFront distribution which will serve your Snowplow JavaScript to anybody anywhere in the world, fast. Now all that remains is to update your Snowplow tag to fetch your own version of `sp.js`.

---

# Self Hosting the JavaScript Tracker on GCP
> Host sp.js on Google Cloud Storage with CDN for reliable tracker distribution to your users.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/self-hosting-the-javascript-tracker-on-google-cloud/

## Pre-requisites

For the purposes of this guide, we are going to assume that you want to serve the standard `sp.js` from Google Cloud Storage. To accomplish this, you will need the following:

- An account with [Google Cloud](https://cloud.google.com/)
- Access to Google Cloud Storage (GCS) within your GCP account

## Download the JavaScript tracker file

Navigate to <https://github.com/snowplow/snowplow-javascript-tracker/releases> and download the latest version of the Snowplow JavaScript Tracker sp.js file

## gzip and rename the file

- rename `sp.js` to a random 8 character string to reduce the chance of AdBlockers preventing the script from loading e.g. `gh7rnghq.js`
- `gzip` the file to reduce the file size and reduce associated cloud storage and egress costs.

From a terminal / command prompt window, navigate to where you have downloaded the file and run:

`gzip -c sp.js > gh7rnghq.js`

**N.B.** on Windows you may need to [download the gzip binaries](http://gnuwin32.sourceforge.net/packages/gzip.htm)

We will continue referring to the file as `sp.js` throughout this guide, however where `sp.js` is mentioned we are referring to your renamed and gzipped file.

## Uploading to Google Cloud Storage

### Create a storage bucket

1. Navigate to [Google Cloud Console](https://console.cloud.google.com/) and ensure you are in the Google Cloud Project that you wish to host the Snowplow JavaScript Tracker in
2. Navigate to the [Storage section](https://console.cloud.google.com/storage/browser) of Google Cloud Console
3. Create a new bucket with the following settings

| Option                                                                               | Value                                                                                  |
| ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| **Name** (also see [Add DNS Record for bucket](#add-dns-record-for-bucket-optional)) | For example, `[company-name]-sp-js`                                                    |
| **Storage region**                                                                   | Select a suitable region, multi-region offers the widest availability and highest SLAs |
| **Storage class**                                                                    | Standard                                                                               |
| Access control options                                                               | Fine grained                                                                           |

Connecting your domain to your storage bucket (optional)

You may wish to connect your domain to your Cloud Storage bucket. In this case your bucket can be named in the format of a subdomain e.g.`spassets.acme.com`.

See the section below about adding a DNS record for an additional step you'll need to take.

### Upload your file

Click _Upload Files_ and upload your tracker file.

Make future upgrades easier

We recommend that you create a folder for each version of the JavaScript tracker to make future updates easier. If you wish to do this then first create a folder corresponding to the version and place your tracker file in that folder.

### Set metadata

Open the _Edit Metadata_ popup using the file menu.

1. Alter the `Content-Encoding` value to be `gzip`
2. Alter the `Cache-Control` to `max-age=31536000`
3. Save the meta data

### Set permissions

Open the [_Edit Access_ popup](https://cloud.google.com/storage/docs/access-control/making-data-public#objects) using the file menu.

Add a new item in the table, enter the following details and click Save.

| Option | Value    |
| ------ | -------- |
| Entity | Public   |
| Name   | allUsers |
| Access | Reader   |

Click _Copy URL_ next to _Public to internet_ in the file browser to get the file's URL e.g. <https://storage.googleapis.com/company-name-sp-js/gh7rnghq.js> 

### Add DNS Record for bucket (optional)

This will only work correctly if:

- you earlier created your bucket with a name corresponding to the subdomain (see "Create a storage bucket" section above) you wish to use
- you have verified ownership of this domain in Google Cloud: <https://cloud.google.com/storage/docs/domain-name-verification>.

To connect your domain to your Cloud Storage bucket, you will need to create a `CNAME` record as below:

| Option | Value                     |
| ------ | ------------------------- |
| Name   | \[your domain]            |
| Type   | CNAME                     |
| Data   | c.storage.googleapis.com. |

CNAME redirection only works on HTTP, to ensure this works on HTTPS you must follow [this troubleshooting guide](https://cloud.google.com/storage/docs/troubleshooting#https).

## Update your tracking tags

Update any existing tracking tags to point to your self-hosted file URL.

---

# Third party CDN hosting for the JavaScript tracker
> Use jsDelivr or unpkg CDN for quick testing and evaluation before moving to self-hosted production setup.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/third-party-cdn-hosting/

Whilst we’re offering CDN options, Snowplow does not recommend using third party providers in production for hosting JavaScript assets that you use on your websites. We encourage all users to host `sp.js` on their own servers, and this will always be the recommended best practice.

If you wish to follow our recommended best practices, please follow our [self hosting guide](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/).

## jsDelivr CDN

Snowplow JavaScript Tracker v2, v3 and v4 assets are available via the [jsDelivr](http://jsdelivr.com) CDN. Click the links below to explore the available assets:

[JavaScript Tracker v4 on jsDelivr](https://www.jsdelivr.com/package/npm/@snowplow/javascript-tracker?path=dist) [JavaScript Tracker v3 on jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/javascript-tracker@3.24.6/) [JavaScript Tracker v2 on jsDelivr](https://www.jsdelivr.com/package/gh/snowplow/sp-js-assets)

## cdnjs CDN

Snowplow JavaScript Tracker v2 is available via the [cdnjs](https://cdnjs.com/) CDN. v3 will be available soon. Click the link below to explore the available assets:

[Snowplow JavaScript Tracker v2 on cdnjs](https://cdnjs.com/libraries/snowplow)

## unpkg CDN

Snowplow JavaScript Tracker v4 is available via the [unpkg](https://unpkg.com/) CDN. Click the link below to explore the available assets:

[Snowplow JavaScript Tracker v4 on unpkg](https://unpkg.com/browse/@snowplow/javascript-tracker@latest/dist/)

---

# Set up the web trackers
> Install and initialize the JavaScript or Browser tracker with customizable configuration options.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/

**JavaScript (tag):**

There are two distributions of the JavaScript Tracker:

| Distribution | Size (minified + gzipped) | Default distribution | Best for                                                 | Plugins included? |
| ------------ | ------------------------- | -------------------- | -------------------------------------------------------- | ----------------- |
| `sp.js`      | \~25 KB                   | ✅                    | Quick start with the a range of common tracking features | ✅                 |
| `sp.lite.js` | \~18 KB                   | ❌                    | Smaller file size and core tracking only                 | ❌                 |

The `sp.js` distribution doesn't include all available plugins. See the [plugins](/docs/sources/web-trackers/plugins/) page for full details on which are included.

Use the following tag on your page to load `sp.js`:

```javascript
<script type="text/javascript" async=1>
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[]; p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments) };p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1; n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));
</script>
```

We recommend self hosting `sp.js` by following our [self hosting](/docs/sources/web-trackers/tracker-setup/hosting-the-javascript-tracker/) guides. All tracker versions are available on [GitHub](https://github.com/snowplow/snowplow-javascript-tracker/releases).

We also recommend renaming `sp.js` as this filename is commonly blocked by adblockers. Renaming to a random string will help ensure the JavaScript Tracker is loaded as expected.

As well as loading the JavaScript Tracker, this tag creates a global function called `snowplow` which you use to access the Tracker. The rest of the documentation will assume that the function is called `snowplow`, but you can replace the string with the function name of your choice.

If you're running multiple JavaScript Trackers on the same page, you can avoid conflicts by specifying different function names for each tracker.

> **Info:** If you're testing by opening your HTML file directly from your filesystem, e.g. `file:///home/joe/test.html`, protocol-relative URLs like `//cdn.example.com/sp.js` won't load. Use a local server or specify the full URL with `http://` or `https://`.

Once the tracker is loaded via the tag, you can move on to initializing the tracker.

**Browser (npm):**

Getting started with sending events using the Browser Tracker will be familiar for anyone who is used to installing npm packages into their web apps and is designed to work with frameworks such as React, Angular and Vue.

- Install the `@snowplow/browser-tracker` package using your preferred package manager

  - `npm install @snowplow/browser-tracker`
  - `yarn add @snowplow/browser-tracker`
  - `pnpm add @snowplow/browser-tracker`

- You can then import this library into your application

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
```

- Configure an instance of the tracker by calling `newTracker(...)` with your desired properties. This will create a module level instance of your tracker. You don't need to keep a reference to it.

```javascript
newTracker('sp1', '{{collector_url}}', {
  appId: 'my-app-id',
  plugins: [ ],
});
```

- Then you can use the track methods to send some events. You can send a Page View event to all initialised trackers with just:

```javascript
trackPageView();
```

***

---

# Initialize and configure the web trackers
> Configure tracker behavior with options for cookies, events, anonymous tracking, and collector endpoints.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/initialization-options/

Initialize the tracker by calling the `newTracker` function. It takes three arguments:

1. The tracker namespace
2. The Collector endpoint
3. An optional configuration object containing other settings

Here is a simple example of how to initialise a tracker, setting a few configuration options:

**JavaScript (tag):**

```javascript
// The tracker namespace is 'sp' in this example
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax', // Recommended
  contexts: {
    session: true
  }
});
```

**Browser (npm):**

```javascript
// The tracker namespace is 'sp' in this example
newTracker('sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  cookieSameSite: 'Lax', // Recommended
  contexts: {
    session: true
  }
});
```

***

The tracker will send events to the Collector URL you specify by replacing `{{collector_url_here}}`. The final argument is the configuration object. Here it's used to set the app ID and enable the [session](/docs/sources/web-trackers/tracking-events/session/) entity for each event. Each event the tracker sends will have an `app_id` field set to `my-app-id`.

If your code calls `newTracker` multiple times with the same namespace, only the first call is taken into account.

> **Tip:** Initialize one tracker per initial page load.

## Configuration options

The following table shows all the configuration parameters. These are **all optional**: you aren't required to provide any configuration object at all.

| Property                                                                                                                            | Description                                                                                          | Default (if applicable)               | Type                       |
| ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------- | -------------------------- |
| [`appId`](/docs/sources/web-trackers/tracking-events/#application-id)                                                               | Set the application ID.                                                                              |                                       | `string`                   |
| [`appVersion`](/docs/sources/web-trackers/tracking-events/#application-version)                                                     | Set the application version.                                                                         |                                       | `string`                   |
| [`platform`](/docs/sources/web-trackers/tracking-events/#application-platform)                                                      | Set the application platform.                                                                        | `web`                                 | `string` enum              |
| [`cookieDomain`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-domain)                           | Set the cookie domain.                                                                               | Current domain                        | `string`                   |
| [`discoverRootDomain`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-domain)                     | Automatic discovery and setting of the root domain, to facilitate tracking over multiple subdomains. | `true`                                | `boolean`                  |
| [`cookieName`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-name)                               | Set the cookie name prefix.                                                                          | `_sp_`                                | `string`                   |
| [`cookieSameSite`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-samesite-and-secure-attributes) | Set the cookie SameSite attribute.                                                                   | `Lax`                                 | `string` enum              |
| [`cookieSecure`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-samesite-and-secure-attributes)   | Set the cookie Secure attribute.                                                                     | `true`                                | `boolean`                  |
| [`encodeBase64`](/docs/sources/web-trackers/configuring-how-events-sent/#base64-encoding)                                           | Enable Base64 encoding for self-describing event and entity data.                                    | `true` for GET, `false` for POST      | `boolean`                  |
| [`respectDoNotTrack`](/docs/sources/web-trackers/anonymous-tracking/#respecting-do-not-track)                                       | Respect the browser Do Not Track setting.                                                            | `false`                               | `boolean`                  |
| [`eventMethod`](/docs/sources/web-trackers/configuring-how-events-sent/#network-protocol-and-method)                                | Send events by GET or POST.                                                                          | `post`                                | `string` enum              |
| [`bufferSize`](/docs/sources/web-trackers/configuring-how-events-sent/#number-of-events-per-request)                                | Number of events to buffer before sending.                                                           | 1                                     | `int`                      |
| [`maxPostBytes`](/docs/sources/web-trackers/configuring-how-events-sent/#maximum-payload-size)                                      | Maximum size in bytes for POST requests.                                                             | 40000                                 | `int`                      |
| [`maxGetBytes`](/docs/sources/web-trackers/configuring-how-events-sent/#maximum-payload-size)                                       | Maximum size for GET request URLs.                                                                   |                                       | `int`                      |
| [`postPath`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-post-path)                                              | Custom collector POST path.                                                                          | `/com.snowplowanalytics.snowplow/tp2` | `string`                   |
| [`crossDomainLinker`](/docs/sources/web-trackers/cross-domain-tracking/)                                                            | Function to determine which links to decorate for cross-domain tracking.                             |                                       | `function`                 |
| [`useExtendedCrossDomainLinker`](/docs/sources/web-trackers/cross-domain-tracking/)                                                 | Use extended format for cross-domain linker with additional user/session data.                       | `false`                               | `boolean` or `object`      |
| [`cookieLifetime`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#cookie-lifetime-and-duration)          | Set the cookie lifetime in seconds.                                                                  | 63072000 (2 years)                    | `int`                      |
| [`sessionCookieTimeout`](/docs/sources/web-trackers/tracking-events/session/)                                                       | Set the session timeout in seconds.                                                                  | 1800 (30 minutes)                     | `int`                      |
| [`stateStorageStrategy`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#storage-strategy)                | How to store tracker state.                                                                          | `cookieAndLocalStorage`               | `string` enum              |
| [`resetActivityTrackingOnPageView`](/docs/sources/web-trackers/tracking-events/activity-page-pings/#reset-page-pings-on-page-view)  | Reset page ping timers when a page view is tracked.                                                  | `true`                                | `boolean`                  |
| [`connectionTimeout`](/docs/sources/web-trackers/configuring-how-events-sent/#connection-timeout)                                   | Request timeout in milliseconds.                                                                     | 5000                                  | `int`                      |
| [`anonymousTracking`](/docs/sources/web-trackers/anonymous-tracking/)                                                               | Enable anonymous tracking (do not track user identifiers).                                           | `false`                               | `boolean` or `object`      |
| [`customHeaders`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-request-headers)                                   | Custom headers to add to POST requests.                                                              |                                       | `object`                   |
| [`credentials`](/docs/sources/web-trackers/configuring-how-events-sent/#disabling-sending-credentials-with-requests)                | Whether to send credentials (cookies) with requests.                                                 | `include`                             | `string` enum              |
| [`contexts`](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities)                                         | Configure context entities to add to all events.                                                     | `{ webPage: true }`                   | `object`                   |
| [`plugins`](/docs/sources/web-trackers/plugins/)                                                                                    | Plugins to extend tracker functionality.                                                             | `[]`                                  | `BrowserPlugin[]`          |
| [`retryStatusCodes`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-retry-http-codes)                               | HTTP status codes to retry requests on.                                                              |                                       | `int[]`                    |
| [`dontRetryStatusCodes`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-retry-http-codes)                           | HTTP status codes to never retry.                                                                    | `[400, 401, 403, 410, 422]`           | `int[]`                    |
| [`retryFailedRequests`](/docs/sources/web-trackers/configuring-how-events-sent/#retries)                                            | Retry failed requests (timeouts, network errors).                                                    | `true`                                | `boolean`                  |
| [`onSessionUpdateCallback`](/docs/sources/web-trackers/tracking-events/session/#on-session-update-callback)                         | Callback executed when the session identifier updates.                                               |                                       | `function`                 |
| [`onRequestSuccess`](/docs/sources/web-trackers/configuring-how-events-sent/#onrequestsuccess-callback)                             | Callback executed when a request succeeds (2xx status).                                              |                                       | `function`                 |
| [`onRequestFailure`](/docs/sources/web-trackers/configuring-how-events-sent/#onrequestfailure-callback)                             | Callback executed when a request fails (non-2xx status).                                             |                                       | `function`                 |
| [`preservePageViewIdForUrl`](/docs/sources/web-trackers/tracking-events/page-views/#change-id-behavior-for-spas)                    | Control when a new page view ID is generated based on URL changes.                                   | `false`                               | `boolean` or `string` enum |
| [`customFetch`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-event-store)                                         | Override the default fetch function with a custom implementation.                                    |                                       | `function`                 |
| [`eventStore`](/docs/sources/web-trackers/configuring-how-events-sent/#custom-event-store)                                          | Custom EventStore implementation for storing events before sending.                                  |                                       | `object`                   |
| [`keepalive`](/docs/sources/web-trackers/configuring-how-events-sent/#keepalive-option-for-collector-requests)                      | Allow requests to outlive the webpage. Enables requests to complete even if the page is closed.      | `false`                               | `boolean`                  |
| [`synchronousCookieWrite`](/docs/sources/web-trackers/configuring-how-events-sent/#synchronous-cookie-writes)                       | Write cookies synchronously (blocks main thread).                                                    | `false`                               | `boolean`                  |
| [`maxLocalStorageQueueSize`](/docs/sources/web-trackers/cookies-and-local-storage/configuring-cookies/#local-storage-queue-size)    | Maximum events to queue in local storage when they are failing to send.                              | 1000                                  | `int`                      |
| [`useLocalStorage`](/docs/sources/web-trackers/configuring-how-events-sent/)                                                        | Whether the event queue persists to localStorage.                                                    | `true`                                | `boolean`                  |
| [`useStm`](/docs/sources/web-trackers/configuring-how-events-sent/)                                                                 | Whether to add the sent timestamp to GET events.                                                     | `true`                                | `boolean`                  |

## Example configuration code

Here is a longer code example in which every tracker configuration parameter is set:

**JavaScript (tag):**

If you're using the minimal `sp.lite.js` distribution, setting the `performanceNavigationTiming`, `gaCookies`, or `webVitals` `context` options won't do anything as those plugins aren't available.

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  platform: 'web',
  cookieDomain: null,
  discoverRootDomain: true,
  cookieName: '_sp_',
  cookieSameSite: 'Lax', // Recommended
  cookieSecure: true,
  encodeBase64: true,
  respectDoNotTrack: false,
  eventMethod: 'post',
  bufferSize: 1,
  maxPostBytes: 40000,
  maxGetBytes: 1000, // available in v3.4+
  postPath: '/custom/path', // Collector must be configured
  crossDomainLinker: function (linkElement) {
    return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
  },
  useExtendedCrossDomainLinker: {
    userId: true,
  },
  cookieLifetime: 63072000,
  sessionCookieTimeout: 1800,
  stateStorageStrategy: 'cookieAndLocalStorage',
  maxLocalStorageQueueSize: 1000,
  resetActivityTrackingOnPageView: true,
  connectionTimeout: 5000,
  anonymousTracking: false,
  // anonymousTracking: { withSessionTracking: true },
  // anonymousTracking: { withSessionTracking: true, withServerAnonymisation: true },
  customHeaders: {}, // Use with caution. Available from v3.2.0+
  credentials: 'include', // Available from v4+
  contexts: {
    webPage: true, // Default
    session: false, // Adds client session entity to events, off by default. Available in v3.5+.
    browser: false, // Adds browser entity to events, off by default. Available in v3.9+.
    performanceNavigationTiming: false, // Adds performance navigation timing entity. Available in v4.0.2+
    gaCookies: false, // Adds Google Analytics cookies entity. Available in v3.0+
    webVitals: false // Adds web vitals entity. Available in v4.0+
  },
  retryStatusCodes: [],
  dontRetryStatusCodes: [],
  retryFailedRequests: true,
  onSessionUpdateCallback: function(clientSession) { }, // Allows the addition of a callback, whenever a new session is generated. Available in v3.11+.
  onRequestSuccess: function(data) => { }, // Available in v3.18.1+
  onRequestFailure: function(data) => { }, // Available in v3.18.1+
  preservePageViewIdForUrl: false,
  keepalive: false, // Introduced in v4
  customFetch: undefined, // Introduced in v4
  eventStore: undefined, // Introduced in v4
  synchronousCookieWrite: false,
  useLocalStorage: true,
  useStm: true,
});
```

The `contexts` block also accepts `clientHints` and `geolocation` keys. The [client hints](/docs/sources/web-trackers/tracking-events/client-hints/) and [geolocation](/docs/sources/web-trackers/tracking-events/timezone-geolocation/) plugins used to be included in `sp.js`, but were removed in version 4. Setting these configuration options won't do anything in version 4+ unless you've also installed the plugins separately.

**Browser (npm):**

```javascript
newTracker('sp', '{{collector_url_here}}', {
  appId: 'my-app-id',
  appVersion: '0.1.0',
  platform: 'web',
  cookieDomain: null,
  discoverRootDomain: true,
  cookieName: '_sp_',
  cookieSameSite: 'Lax', // Recommended
  cookieSecure: true,
  encodeBase64: true,
  respectDoNotTrack: false,
  eventMethod: 'post',
  bufferSize: 1,
  maxPostBytes: 40000,
  maxGetBytes: 1000, // available in v3.4+
  postPath: '/custom/path', // Collector must be configured
  crossDomainLinker: function (linkElement) {
    return (linkElement.href === 'http://acme.de' || linkElement.id === 'crossDomainLink');
  },
  useExtendedCrossDomainLinker: {
    userId: true,
  },
  cookieLifetime: 63072000,
  sessionCookieTimeout: 1800,
  stateStorageStrategy: 'cookieAndLocalStorage',
  maxLocalStorageQueueSize: 1000,
  resetActivityTrackingOnPageView: true,
  connectionTimeout: 5000,
  anonymousTracking: false,
  // anonymousTracking: { withSessionTracking: true },
  // anonymousTracking: { withSessionTracking: true, withServerAnonymisation: true },
  customHeaders: {}, // Use with caution. Available from v3.2.0+
  credentials: 'include', // Available from v4+
  contexts: {
    webPage: true, // Default
    session: false, // Adds client session context entity to events, off by default. Available in v3.5+.
    browser: false // Adds browser context entity to events, off by default. Available in v3.9+.
  },
  plugins: [],
  retryStatusCodes: [],
  dontRetryStatusCodes: [],
  retryFailedRequests: true,
  onSessionUpdateCallback: function(clientSession) { }, // Allows the addition of a callback, whenever a new session is generated. Available in v3.11+.
  onRequestSuccess: function(data) => { }, // Available in v3.18.1+
  onRequestFailure: function(data) => { }, // Available in v3.18.1+
  preservePageViewIdForUrl: false,
  keepalive: false, // Introduced in v4
  customFetch: undefined, // Introduced in v4
  eventStore: undefined, // Introduced in v4
  synchronousCookieWrite: false,
  useLocalStorage: true,
  useStm: true,
});
```

***

You can further extend the tracker functionality by installing plugins. Read more about them [here](/docs/sources/web-trackers/plugins/).

---

# Manage multiple trackers on web
> Run multiple tracker instances to send events to different collectors with targeted method execution.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/managing-multiple-trackers/

You can have more than one tracker instance running on the same page at once. This may be useful if you want to log events to different collectors. By default, any Snowplow method you call will be executed by every tracker you have created so far.

You can override this behavior and specify which trackers will execute a Snowplow method.

**JavaScript (tag):**

To do this, change the method name by adding a colon followed by a list of tracker names separated by semicolons as such:

```javascript
/* Sending a pageview on a single tracker */
snowplow('trackPageView:{{TRACKER_NAME}}');
/* Sending a pageview on multiple trackers */
snowplow('trackPageView:{{TRACKER_NAME}};{{ANOTHER_TRACKER_NAME}}');

/**
 * Where {{TRACKER_NAME}} and {{ANOTHER_TRACKER_NAME}} are names of
 * trackers initialized on the page using the `newTracker` API.
*/
```

**Browser (npm):**

To do this, there is a final parameter on each function call that accepts an array containing the tracker identifiers you wish this function to be executed against as such:

```javascript
/* Sending a pageview on a single tracker */
trackPageView({}, [ '{{TRACKER_NAME}}' ]);
/* Sending a pageview on multiple trackers */
trackPageView({}, [ '{{TRACKER_NAME}}', '{{ANOTHER_TRACKER_NAME}}' ]);

/**
 * Where {{TRACKER_NAME}} and {{ANOTHER_TRACKER_NAME}} are names of
 * trackers initialized on the page using the `newTracker` API.
*/
```

***

Example usage:

**JavaScript (tag):**

```javascript
snowplow("newTracker", "sp1", "{{FIRST_COLLECTOR_URL}}", {
  appId: "my-app",
  platform: "mob"
});

snowplow("newTracker", "sp2", "{{SECOND_COLLECTOR_URL}}", {
  appId: "my-app",
  platform: "mob"
});

/* Both trackers will use this custom title */
snowplow('setCustomUrl', 'http://mysite.com/checkout-page');

/* Both trackers will fire a structured event */
snowplow('trackStructEvent', {
  category: 'Mixes',
  action: 'Play',
  label: 'MRC/fabric-0503-mix',
  property: '',
  value: 0.0,
});

/* Only the first tracker will be affected by the plugin addition */
snowplow(
  "addPlugin:sp1",
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-web-vitals@3/dist/index.umd.min.js",
  ["snowplowWebVitals", "WebVitalsPlugin"]
);

/* Only the first tracker will fire this structured event */
snowplow('trackStructEvent:sp1', {
  category: 'Mixes',
  action: 'Play',
  label: 'MRC/fabric-0503-mix',
  property: '',
  value: 0.0,
});

/* Only the second tracker will fire this self-describing event */
snowplow('trackSelfDescribingEvent:sp2', {
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/1-0-0',
    data: {
        product_id: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        available_since$dt: new Date(2013,3,7)
    }
  }
});

/* Both trackers will fire a page view event */
snowplow('trackPageView:sp1;sp2');
```

**Browser (npm):**

```javascript
import {
  newTracker,
  setCustomUrl,
  trackStructEvent,
  trackSelfDescribingEvent,
  trackPageView
} from '@snowplow/browser-tracker';
import { WebVitalsPlugin } from "@snowplow/browser-plugin-web-vitals";

newTracker('sp1', '{{FIRST_COLLECTOR_URL}}', {
  appId: 'my-app',
  platform: 'mob'
});

newTracker('sp2', '{{SECOND_COLLECTOR_URL}}', {
  appId: 'my-app',
  platform: 'web'
});

/* Both trackers will use this custom title */
setCustomUrl('http://mysite.com/checkout-page');

/* Both trackers will fire a structured event */
trackStructEvent({
  category: 'Mixes',
  action: 'Play',
  label: 'MRC/fabric-0503-mix',
  property: '',
  value: 0.0,
});

/* Only the first tracker will be affected by the plugin addition */
addPlugin({ plugin: WebVitalsPlugin() }, ['sp1']);

/* Only the first tracker will fire this structured event */
trackStructEvent({
  category: 'Mixes',
  action: 'Play',
  label: 'MRC/fabric-0503-mix',
  property: '',
  value: 0.0,
},
[ 'sp1' ]);

/* Only the second tracker will fire this self-describing event */
trackSelfDescribingEvent({
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/1-0-0',
    data: {
        product_id: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        available_since$dt: new Date(2013,3,7)
    }
  }
},
[ 'sp2' ]);

/* Both trackers will fire a page view event */
trackPageView({}, [ 'sp1', 'sp2' ]);
```

***

---

# Deploy the JavaScript tracker using the getanalytics.io plugin
> Deploy the web tracker through the analytics npm package to send events to multiple providers.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracker-setup/snowplow-plugin-for-analytics-npm-package/

The Snowplow JavaScript tracker can now be deployed directly into your web and node applications using the `analytics` and `@analytics/snowplow` NPM packages.

> **Note:** Snowplow recommends using the official Snowplow Browser Tracker or Node.js Tracker instead of this package as you will receive updates faster and install a smaller dependency into your application, unless you wish to send your events to multiple providers which is where the @analytics packages are particularly useful.

### Quick Start

```bash
npm install analytics
npm install @analytics/snowplow
```

Initialise the plugin:

```javascript
import Analytics from 'analytics'
import snowplowPlugin from '@analytics/snowplow'

const analytics = Analytics({
  app: 'awesome-app',
  plugins: [
    // Minimal recommended configuration
    snowplowPlugin({
      name: 'snowplow',
      collectorUrl: 'collector.mysite.com',
      trackerSettings: {
        appId: 'myApp',
        contexts: {
          webPage: true
        }
      }
    })
  ]
})
```

Then track a page view event:

```javascript
analytics.page()
```

### Full documentation

[Snowplow Plugin documentation](https://getanalytics.io/plugins/snowplow/) (getanalytics.io)

[Analytics package documentation](https://getanalytics.io/) (getanalytics.io)

---

# Track activity with page pings on web
> Monitor user engagement over time with page ping events that track scrolling, mouse movement, and tab focus on web pages.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/activity-page-pings/

As well as tracking page views, we can monitor whether users continue to engage with pages over time, and record how they digest content on each page over time.

That is accomplished using [page ping events](/docs/events/ootb-data/page-activity-tracking/#page-engagement). If activity tracking is enabled, the web page is monitored to see if a user is engaging with it e.g. is the tab in focus, does the mouse move over the page, does the user scroll, is `updatePageActivity` called, etc. If any of these things occur in a set period of time, a page ping event fires, and records the maximum scroll left / right and up / down in the last ping period. If there is no activity in the page, e.g. because the user is on a different browser tab, no page ping fires.

Page ping events are **automatically tracked** once configured.

## Track page pings

### Enable activity tracking

Page pings are enabled by:

**JavaScript (tag):**

```javascript
snowplow('enableActivityTracking', {
  minimumVisitLength: number,
  heartbeatDelay: number
});
```

**Browser (npm):**

```javascript
import {
  enableActivityTracking
} from '@snowplow/browser-tracker';

enableActivityTracking({
  minimumVisitLength: number,
  heartbeatDelay: number
});
```

***

where `minimumVisitLength` is the time period from page load before the first page ping occurs, in seconds. `heartbeat` is the number of seconds between each page ping, once they have started.

Activity tracking will be disabled if either `minimumVisitLength` or `heartbeatDelay` is not integer. This is to prevent relentless callbacks.

You can elect to enable activity tracking on specific pages. It is executed as part of the main Snowplow tracking tag.

The following example would generate the first ping event after 30 seconds, and subsequent pings every 10 seconds as long as the user continued to browse the page actively.

> **Info:** The `enableActivityTracking` method **must** be called before the `trackPageView` method.

**JavaScript (tag):**

```javascript
snowplow('enableActivityTracking', {
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

snowplow('trackPageView');
```

**Browser (npm):**

```javascript
import {
  enableActivityTracking,
  trackPageView
} from '@snowplow/browser-tracker';

enableActivityTracking({
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

trackPageView();
```

***

### Disable activity tracking

> **Note:** Available since version 3.14 of the tracker.

To disable activity tracking, you can use the `disableActivityTracking` method.

**JavaScript (tag):**

```javascript
snowplow('disableActivityTracking');
```

**Browser (npm):**

```javascript
import { disableActivityTracking } from '@snowplow/browser-tracker';

disableActivityTracking();
```

***

Disabling activity tracking will stop page activity intervals and will not send additional activity tracking events.

### Page activity

You can also mark the user as active with:

**JavaScript (tag):**

```javascript
snowplow('updatePageActivity');
```

**Browser (npm):**

```javascript
import { updatePageActivity } from '@snowplow/browser-tracker';

updatePageActivity();
```

***

On the next interval after this call, a ping will be generated even if the user had no other activity.

This is particularly useful when a user is passively engaging with your content, e.g. watching a video.

## Activity tracking callback

You can now perform edge analytics in the browser to reduce the number of events sent to your Collector whilst still tracking user activity. The Snowplow JavaScript Tracker enables this by allowing a callback to be specified in place of a page ping being sent.

### Enable callback

**JavaScript (tag):**

```javascript
snowplow('enableActivityTrackingCallback', {
  minimumVisitLength: number,
  heartbeatDelay: number,
  callback: (data: ActivityCallbackData) => void
});
```

**Browser (npm):**

```javascript
import {
  enableActivityTrackingCallback
} from '@snowplow/browser-tracker';

enableActivityTrackingCallback({
  minimumVisitLength: number,
  heartbeatDelay: number,
  callback: (data: ActivityCallbackData) => void
});
```

***

where `minimumVisitLength` is the time period from page load before the first page ping occurs, in seconds. `heartbeat` is the number of seconds between each page ping, once they have started. The `callback` should be a function which will receive an event object containing the page ping activity information, including `pageViewId`, and any Page View entities.

```javascript
type ActivityCallbackData = {
    /**
     * All context for the activity tracking
     * Often generated by the page view events context callback
     */
    context: Array<SelfDescribingJson>;
    /** The current page view id */
    pageViewId: string;
    /** The minimum X scroll position for the current page view */
    minXOffset: number;
    /** The maximum X scroll position for the current page view */
    minYOffset: number;
    /** The minimum Y scroll position for the current page view */
    maxXOffset: number;
    /** The maximum Y scroll position for the current page view */
    maxYOffset: number;
};
```

A full example of how this might be used to aggregate page ping information and then send an event on page unload is below:

**JavaScript (tag):**

```javascript
snowplow('newTracker', 'sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
    eventMethod: 'beacon'
});
var aggregatedEvent = {
    pageViewId: null,
    minXOffset: 0,
    maxXOffset: 0,
    minYOffset: 0,
    maxYOffset: 0,
    numEvents: 0
};
snowplow('enableActivityTrackingCallback', {
  minimumVisitLength: 10,
  heartbeatDelay: 10,
  callback: function (event) {
    aggregatedEvent = {
        pageViewId: event.pageViewId,
        minXOffset: aggregatedEvent.minXOffset < event.minXOffset ? aggregatedEvent.minXOffset : event.minXOffset,
        maxXOffset: aggregatedEvent.maxXOffset > event.maxXOffset ? aggregatedEvent.maxXOffset : event.maxXOffset,
        minYOffset: aggregatedEvent.minYOffset < event.minYOffset ? aggregatedEvent.minYOffset : event.minYOffset,
        maxYOffset: aggregatedEvent.maxYOffset > event.maxYOffset ? aggregatedEvent.maxYOffset : event.maxYOffset,
        numEvents: aggregatedEvent.numEvents + 1
    };
});
document.addEventListener('visibilitychange', function() {
  if (document.visibilityState == 'hidden') {
    window.snowplow('trackSelfDescribingEvent', {
      event: {
        schema: 'iglu:com.acme_company/page_unload/jsonschema/1-0-0',
        data: {
            minXOffset: Math.max(0, Math.round(aggregatedEvent.minXOffset)),
            maxXOffset: Math.max(0, Math.round(aggregatedEvent.maxXOffset)),
            minYOffset: Math.max(0, Math.round(aggregatedEvent.minYOffset)),
            maxYOffset: Math.max(0, Math.round(aggregatedEvent.maxYOffset)),
            activeSeconds: aggregatedEvent.numEvents * 10
        }
      }
    });
  }
});
window.snowplow('trackPageView');
```

**Browser (npm):**

```javascript
import {
  newTracker,
  enableActivityTrackingCallback,
  trackPageView,
  trackSelfDescribingEvent
} from '@snowplow/browser-tracker';

newTracker('sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
    eventMethod: 'beacon'
});
var aggregatedEvent = {
    pageViewId: null,
    minXOffset: 0,
    maxXOffset: 0,
    minYOffset: 0,
    maxYOffset: 0,
    numEvents: 0
};
enableActivityTrackingCallback({
  minimumVisitLength: 10,
  heartbeatDelay: 10,
  callback: function (event) {
    aggregatedEvent = {
        pageViewId: event.pageViewId,
        minXOffset: aggregatedEvent.minXOffset < event.minXOffset ? aggregatedEvent.minXOffset : event.minXOffset,
        maxXOffset: aggregatedEvent.maxXOffset > event.maxXOffset ? aggregatedEvent.maxXOffset : event.maxXOffset,
        minYOffset: aggregatedEvent.minYOffset < event.minYOffset ? aggregatedEvent.minYOffset : event.minYOffset,
        maxYOffset: aggregatedEvent.maxYOffset > event.maxYOffset ? aggregatedEvent.maxYOffset : event.maxYOffset,
        numEvents: aggregatedEvent.numEvents + 1
    };
});
document.addEventListener('visibilitychange', function() {
  if (document.visibilityState == 'hidden') {
    trackSelfDescribingEvent({
      event: {
        schema: 'iglu:com.acme_company/page_unload/jsonschema/1-0-0',
        data: {
            minXOffset: Math.max(0, Math.round(aggregatedEvent.minXOffset)),
            maxXOffset: Math.max(0, Math.round(aggregatedEvent.maxXOffset)),
            minYOffset: Math.max(0, Math.round(aggregatedEvent.minYOffset)),
            maxYOffset: Math.max(0, Math.round(aggregatedEvent.maxYOffset)),
            activeSeconds: aggregatedEvent.numEvents * 10
        }
      }
    });
  }
});
trackPageView();
```

***

> **Note:** For this technique of sending on visibility change to work reliably, we recommend initialising the Snowplow tracker with `eventMethod: 'beacon'` and/or `stateStorageStrategy: 'cookieAndLocalStorage'` (if navigating to a page that also contains the JS Tracker). Using the visibility change technique may not work as expected for Single Page Applications (SPA), you would need to send the aggregated event to the Snowplow collector on navigation within your application.

We are using `visibilitychange` events as `beforeunload` isn't a reliable option for mobile devices when using `beacon`. You can read more about this on [MDN](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon#description). An idea on the different levels of compatibility of the different Page Visiblity API across browsers and mobile can here found [here](https://www.igvita.com/2015/11/20/dont-lose-user-and-app-state-use-page-visibility/).

### Disable callback

> **Note:** Available since version 3.14 of the tracker.

To disable the activity tracking callback, you can use the `disableActivityTrackingCallback` method.

**JavaScript (tag):**

```javascript
snowplow('disableActivityTrackingCallback');
```

**Browser (npm):**

```javascript
import { disableActivityTrackingCallback } from '@snowplow/browser-tracker';

disableActivityTrackingCallback();
```

***

This will stop any further activity tracking callback from being executed.

## Reset page pings on page view

By default, tracking a [page view](/docs/sources/web-trackers/tracking-events/page-views/) using `trackPageView()`resets activity tracking timers and scroll offsets. This is particularly useful for single page applications (SPAs) where page navigation doesn't cause a full page reload.

This behavior is the default since version 2.13.0. To prevent it, use the `resetActivityTrackingOnPageView` option during [tracker initialization](/docs/sources/web-trackers/tracker-setup/initialization-options/).

## Add entities dynamically

When tracking [page views](/docs/sources/web-trackers/tracking-events/page-views/), you can attach custom entities dynamically to each page view **and** subsequent page pings using the `contextCallback` parameter.

---

# Track advertisements on web
> Track ad impressions, clicks, and conversions with support for various cost models across ad networks and campaigns.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/ads/

Snowplow tracking code can be included in ad tags in order to track impressions and ad clicks. This is used by e.g. ad networks to identify which sites and web pages users visit across a network, so that they can be segmented, for example.

Each ad tracking method has a `costModel` field and a `cost` field. If you provide the `cost` field, you must also provide one of `'cpa'`, `'cpc'`, and `'cpm'` for the `costModel` field.

It may be the case that multiple ads from the same source end up on a single page. If this happens, it is important that the different Snowplow code snippets associated with those ads not interfere with one another. The best way to prevent this is to randomly name each tracker instance you create so that the probability of a name collision is negligible. See [Managing multiple trackers](/docs/sources/web-trackers/tracker-setup/managing-multiple-trackers/) for more on having more than one tracker instance on a single page.

Snowplow ad events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                     |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)               |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-ad-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-ad-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-ad-tracking@latest/dist/index.umd.min.js",
  ["snowplowAdTracking", "AdTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-ad-tracking`
- `yarn add @snowplow/browser-plugin-ad-tracking`
- `pnpm add @snowplow/browser-plugin-ad-tracking`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { AdTrackingPlugin, trackAdClick } from '@snowplow/browser-plugin-ad-tracking';

newTracker('sp1', '{{collector_url}}', {
  appId: 'my-app-id',
  plugins: [ AdTrackingPlugin() ],
});
```

***

## Events

The ad tracking plugin includes three event types: ad impression, ad click, and ad conversion.

### Impression

Ad impression tracking is accomplished using the `trackAdImpression` method. Here are the arguments it accepts:

| **Name**       | **Required?** | **Description**                                                      | **Type**    |
| -------------- | ------------- | -------------------------------------------------------------------- | ----------- |
| `impressionId` | No            | Identifier for the particular impression instance                    | string      |
| `costModel`    | No            | The cost model for the campaign: 'cpc', 'cpm', or 'cpa'              | string enum |
| `cost`         | No            | Ad cost                                                              | number      |
| `targetUrl`    | No            | The destination URL                                                  | string      |
| `bannerId`     | No            | Adserver identifier for the ad banner (creative) being displayed     | string      |
| `zoneId`       | No            | Adserver identifier for the zone where the ad banner is located      | string      |
| `advertiserId` | No            | Adserver identifier for the advertiser which the campaign belongs to | string      |
| `campaignId`   | No            | Adserver identifier for the ad campaign which the banner belongs to  | string      |

> **Note:** All properties are optional but you must specify at least 1 for this to be a valid call to `trackAdImpression`.

An example:

**JavaScript (tag):**

```javascript
snowplow('trackAdImpression', {
  impressionId: '67965967893',
  costModel: 'cpm', // 'cpa', 'cpc', or 'cpm'
  cost: 5.5,
  targetUrl: 'http://www.example.com',
  bannerId: '23',
  zoneId: '7',
  advertiserId: '201',
  campaignId: '12'
});
```

**Browser (npm):**

Example usage:

```javascript
import { trackAdImpression } from '@snowplow/browser-plugin-ad-tracking';

trackAdImpression({
  impressionId: '67965967893',
  costModel: 'cpm', // 'cpa', 'cpc', or 'cpm'
  cost: 5.5,
  targetUrl: 'http://www.example.com',
  bannerId: '23',
  zoneId: '7',
  advertiserId: '201',
  campaignId: '12'
});
```

***

Ad impression events are implemented as Snowplow self describing events. [Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ad_impression/jsonschema/1-0-0) is the JSON schema for an ad impression event.

### Click

Ad click tracking is accomplished using the `trackAdClick` method. Here are the arguments it accepts:

| **Name**       | **Required?** | **Description**                                                      | **Type**    |
| -------------- | ------------- | -------------------------------------------------------------------- | ----------- |
| `targetUrl`    | Yes           | The destination URL                                                  | string      |
| `clickId`      | No            | Identifier for the particular click instance                         | string      |
| `costModel`    | No            | The cost model for the campaign: 'cpc', 'cpm', or 'cpa'              | string enum |
| `cost`         | No            | Ad cost                                                              | number      |
| `bannerId`     | No            | Adserver identifier for the ad banner (creative) being displayed     | string      |
| `zoneId`       | No            | Adserver identifier for the zone where the ad banner is located      | string      |
| `advertiserId` | No            | Adserver identifier for the advertiser which the campaign belongs to | string      |
| `campaignId`   | No            | Adserver identifier for the ad campaign which the banner belongs to  | string      |

An example:

**JavaScript (tag):**

```javascript
snowplow('trackAdClick',{
  targetUrl: 'http://www.example.com',
  clickId: '12243253',
  costModel: 'cpm',
  cost: 2.5,
  bannerId: '23',
  zoneId: '7',
  impressionId: '67965967893', // the same as in trackAdImpression
  advertiserId: '201',
  campaignId: '12'
});
```

**Browser (npm):**

```javascript
import { trackAdClick } from '@snowplow/browser-plugin-ad-tracking';

trackAdClick({
  targetUrl: 'http://www.example.com',
  clickId: '12243253',
  costModel: 'cpm',
  cost: 2.5,
  bannerId: '23',
  zoneId: '7',
  impressionId: '67965967893', // the same as in trackAdImpression
  advertiserId: '201',
  campaignId: '12'
});
```

***

Ad click events are implemented as Snowplow self describing events.[Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ad_click/jsonschema/1-0-0) is the JSON schema for an ad click event.

### Conversion

Use the `trackAdConversion` method to track ad conversions. Here are the arguments it accepts:

| **Name**       | **Required?** | **Description**                                                      | **Type**    |
| -------------- | ------------- | -------------------------------------------------------------------- | ----------- |
| `conversionId` | No            | Identifier for the particular conversion instance                    | string      |
| `costModel`    | No            | The cost model for the campaign: 'cpc', 'cpm', or 'cpa'              | string enum |
| `cost`         | No            | Ad cost                                                              | number      |
| `category`     | No            | Conversion category                                                  | string      |
| `action`       | No            | The type of user interaction, e.g. 'purchase'                        | string      |
| `property`     | No            | Describes the object of the conversion                               | string      |
| `initialValue` | No            | How much the conversion is initially worth                           | number      |
| `advertiserId` | No            | Adserver identifier for the advertiser which the campaign belongs to | string      |
| `campaignId`   | No            | Adserver identifier for the ad campaign which the banner belongs to  | string      |

> **Note:** All properties are optional but you must specify at least one for this to be a valid call to `trackAdConversion`.

An example:

**JavaScript (tag):**

```javascript
snowplow('trackAdConversion', {
  conversionId: '743560297',
  costModel: 'cpa',
  cost: 10,
  category: 'ecommerce',
  action: 'purchase',
  property: '',
  initialValue: 99,
  advertiserId: '201',
  campaignId: '12'
});
```

**Browser (npm):**

```javascript
import { trackAdConversion } from '@snowplow/browser-plugin-ad-tracking';

trackAdConversion({
  conversionId: '743560297',
  costModel: 'cpa',
  cost: 10,
  category: 'ecommerce',
  action: 'purchase',
  property: '',
  initialValue: 99,
  advertiserId: '201',
  campaignId: '12'
});
```

***

Ad conversion events are implemented as Snowplow self describing events. [Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/ad_conversion/jsonschema/1-0-0) is the schema for an ad conversion event.

## Avoiding name collisions

It may be the case that multiple ads from the same source end up on a single page. If this happens, it is important that the different Snowplow code snippets associated with those ads not interfere with one another. The best way to prevent this is to randomly name each tracker instance you create so that the probability of a name collision is negligible. See [Managing multiple trackers](/docs/sources/web-trackers/tracker-setup/managing-multiple-trackers/) for more on having more than one tracker instance on a single page.

Below is an example of how to achieve this when using Snowplow ad impression tracking.

**JavaScript (tag):**

```javascript
<!-- Snowplow starts plowing -->
<script type="text/javascript">

// Wrap script in a closure.
// This prevents rnd from becoming a global variable.
// So if multiple copies of the script are loaded on the same page,
// each instance of rnd will be inside its own namespace and will
// not overwrite any of the others.
// See http://benalman.com/news/2010/11/immediately-invoked-function-expression/
(function(){
  // Randomly generate tracker namespace to prevent clashes
  var rnd = Math.random().toString(36).substring(2);

  // Load Snowplow
  ;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[];
  p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments)
  };p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1;
  n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));

  // Create a new tracker namespaced to rnd
  snowplow('newTracker', rnd, '{{COLLECTOR_URL}}', {
    appId: 'myApp',
  });

  // Replace the values below with macros from your adserver
  snowplow('trackAdImpression:' + rnd, {
    impressionId: '67965967893',
    costModel: 'cpm',          // 'cpa', 'cpc', or 'cpm'
    cost: 5.5,
    targetUrl: 'http://www.example.com',
    bannerId: '23',
    zoneId: '7',
    advertiserId: '201',
    campaignId: '12'
  });
}());
</script>
<!-- Snowplow stops plowing -->
```

**Browser (npm):**

```javascript
import { newTracker } from "@snowplow/browser-tracker";
import { trackAdImpression } from '@snowplow/browser-plugin-ad-tracking';

var rnd = Math.random().toString(36).substring(2);

newTracker(rnd, {{COLLECTOR_URL}}, {
  appId: "myApp",
  platform: "web"
});

trackAdImpression({
  impressionId: '67965967893',
  costModel: 'cpm',          // 'cpa', 'cpc', or 'cpm'
  cost: 5.5,
  targetUrl: 'http://www.example.com',
  bannerId: '23',
  zoneId: '7',
  advertiserId: '201',
  campaignId: '12',
}, [rnd]);
```

***

Even if several copies of the above script appear on a page, the trackers created will all (probably) have different names and so will not interfere with one another. The same technique should be used when tracking ad clicks.

---

# Track browser information with the web trackers
> Track browser information including viewport, device properties, and client hints with the browser entity.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/browsers/

The Snowplow web tracker supports tracking in all modern browsers, including newer browser features such as [client hints](/docs/sources/web-trackers/tracking-events/client-hints/).

## Browser entity

Add an entity to every tracked event that records [information about the user's browser](/docs/events/ootb-data/device-and-browser/#browser-entity). Configure it at [tracker initialization](/docs/sources/web-trackers/tracker-setup/initialization-options/).

The browser entity is **automatically tracked** once configured.

> **Note:** The browser entity is available since version 3.9 of the tracker.

## Atomic event properties

The web trackers automatically capture [browser and page information](/docs/events/ootb-data/device-and-browser/) and add it to every event. You don't need to configure or manually include these properties.

### Set once per tracker initialization

These values are captured when the tracker initializes and remain constant for the session:

| Property                                 | Description                              |
| ---------------------------------------- | ---------------------------------------- |
| `br_cookies`                             | Whether cookies are enabled (`1` or `0`) |
| `doc_charset`                            | Document character set (e.g., `UTF-8`)   |
| `br_lang`                                | Browser language                         |
| `dvce_screenwidth` / `dvce_screenheight` | Screen resolution                        |
| `br_colordepth`                          | Color depth of the display               |
| `os_timezone`                            | User's timezone                          |

### Set on every event

These values are freshly captured for each event:

| Property                         | Description                                    |
| -------------------------------- | ---------------------------------------------- |
| `br_viewwidth` / `br_viewheight` | Viewport dimensions at the time of the event   |
| `doc_width` / `doc_height`       | Document dimensions at the time of the event   |
| `page_url`                       | Current page URL                               |
| `page_referrer`                  | Referrer URL                                   |
| `domain_userid`                  | Domain user ID (first-party cookie identifier) |
| `domain_sessionidx`              | Session index (visit count)                    |
| `domain_sessionid`               | Session ID                                     |
| `user_id`                        | Business user ID, if set via `setUserId()`     |

> **Note:** Some properties may be omitted when [anonymous tracking](/docs/sources/web-trackers/anonymous-tracking/) is enabled.

---

# Track button clicks on web
> Automatically track clicks on button elements and input buttons with configurable filtering and custom context entities.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/button-click/

This plugin enables the automatic tracking of clicks on buttons, covering both `<button>` and `<input type="button">` elements.

> **Note:** The plugin is available from Version 3.18 of the tracker.

Button click events are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                               |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                         |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-button-click-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-button-click-tracking@latest/dist/index.umd.min.js) (latest)               |

**Browser (npm):**

- `npm install @snowplow/browser-plugin-button-click-tracking`
- `yarn add @snowplow/browser-plugin-button-click-tracking`
- `pnpm add @snowplow/browser-plugin-button-click-tracking`

***

## Enable and disable tracking

To start tracking all button clicks, call the `enableButtonClickTracking` function:

**JavaScript (tag):**

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-button-click-tracking@latest/dist/index.umd.min.js',
    ['snowplowButtonClickTracking', 'ButtonClickTrackingPlugin']
);

window.snowplow('enableButtonClickTracking');
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking, ButtonClickTrackingPlugin } from '@snowplow/browser-plugin-button-click-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ ButtonClickTrackingPlugin() ],
});

enableButtonClickTracking();
```

***

Then, to stop tracking button clicks, call the `disableButtonClickTracking` function:

**JavaScript (tag):**

```javascript
window.snowplow('disableButtonClickTracking');
```

**Browser (npm):**

```javascript
import { disableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';
disableButtonClickTracking();
```

***

## Configuration

There are three ways to configure which buttons are tracked. If no configuration is specified, as shown above, all buttons on the page will be tracked.

### Allowlist

You can specify a list of CSS classes to match against the buttons you want to track. Only buttons that contain the class will be tracked.

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  filter: {
    allowlist: ['my-button'],
  }
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';
enableButtonClickTracking({
  filter: {
    allowlist: ['my-button'],
  }
});
```

***

### Denylist

Inversely, you can specify a list of CSS classes to match against the buttons you want to ignore. Only buttons that do not contain the class will be tracked.

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  filter: {
    denylist: ['my-button'],
  }
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';
enableButtonClickTracking({
  filter: {
    denylist: ['my-button'],
  }
});
```

***

### Filter function

Finally, you can specify a function that will be called for each button on the page. If the function returns `true`, the button will be tracked. If it returns `false`, the button will not be tracked.

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  filter: (element) => {
    return element.id === 'my-button';
  },
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';

enableButtonClickTracking({
  filter: (element) => {
    return element.id === 'my-button';
  },
});
```

***

### Default label in events

The tracker automatically adds the `label` information in events based on the button's content or data attributes.

Since the `label` property in the event schema is required, starting from version 4.5 of the tracker, the tracker adds a `(empty)` string as the default label in case it can't be inferred from the button itself. The default value is configurable using the `defaultLabel` option:

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  defaultLabel: "custom-default-label",
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';

enableButtonClickTracking({
  defaultLabel: "custom-default-label",
});
```

***

## Adding context entities to tracked events

You can also attach context entities to the tracked button click events as either an array of self-describing JSON objects or a callback function that returns the entities dynamically.

### Statically defined context entities

To add a static list of context entities to the events, use the following configuration

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  context: [
    {
      schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
      data: { pageType: 'test' }
    }
  ],
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';

enableButtonClickTracking({
  context: [
    {
      schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
      data: { pageType: 'test' }
    }
  ],
});
```

***

### Dynamic context entities

You can define a callback function that takes the click event and button element as parameters and returns a context entity for that specific event.

**JavaScript (tag):**

```javascript
window.snowplow('enableButtonClickTracking', {
  context: function (event, element) {
    return {
      schema: 'iglu:com.example_company/click/jsonschema/1-2-1',
      data: { content: element.textContent }
    };
  },
});
```

**Browser (npm):**

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';

enableButtonClickTracking({
  context: function (event, element) {
    return {
      schema: 'iglu:com.example_company/click/jsonschema/1-2-1',
      data: { content: element.textContent }
    };
  },
});
```

***

## Tracked data

The plugin will track the following data (if present on the element):

| Field     | Description                                         | Type       | Required? |
| --------- | --------------------------------------------------- | ---------- | --------- |
| `label`   | The text on the button, or a user-provided override | `string`   | Yes       |
| `id`      | The ID of the button                                | `string`   | No        |
| `classes` | The classes of the button                           | `string[]` | No        |
| `name`    | The name of the button                              | `string`   | No        |

[Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0) is the JSON schema for a button click event.

### Overriding the label with `data-sp-button-label`

By default, the plugin will use the text on the button as the label. However, you can override this by adding a `data-sp-button-label` attribute to the button:

```html
<button data-sp-button-label="My custom label">Click me</button>
```

This will result in the following event:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0",
    "data": {
        // Note the label is "My custom label", not "Click me"
        "label": "My custom label",
    }
}
```

This can also be useful in the case of icon buttons, where there is no text on the button.

## Full example

**JavaScript (tag):**

Suppose we have the following button on our page:

```html
<button id="home-btn" class="nav-btn blue-btn outlined" data-sp-button-label="Home" name="home">
    <i className="fa fa-home">
</button>
```

We can configure the plugin to only track this button class:

```javascript
window.snowplow('enableButtonClickTracking', {
  filter: {
    allowlist: ['nav-btn'],
  }
});
```

On click, this will result in the following event:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0",
    "data": {
        "label": "Home",
        "id": "home-btn",
        "classes": ["nav-btn", "blue-btn", "outlined"],
        "name": "home"
    }
}
```

**Browser (npm):**

Suppose we have the following button on our page:

```html
<button id="home-btn" class="nav-btn blue-btn outlined" data-sp-button-label="Home" name="home">
    <i className="fa fa-home">
</button>
```

We can configure the plugin to only track this button class:

```javascript
import { enableButtonClickTracking } from '@snowplow/browser-plugin-button-click-tracking';

enableButtonClickTracking({
  filter: {
    allowlist: ['nav-btn'],
  }
});
```

On click, this will result in the following event:

```json
{
    "schema": "iglu:com.snowplowanalytics.snowplow/button_click/jsonschema/1-0-0",
    "data": {
        "label": "Home",
        "id": "home-btn",
        "classes": ["nav-btn", "blue-btn", "outlined"],
        "name": "home"
    }
}
```

***

---

# Track campaigns and UTMs on web
> Identify traffic sources from paid and organic campaigns using UTM parameters and referrer analysis for marketing attribution.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/campaigns-utms/

Campaign tracking is used to identify the source of traffic coming to a website.

At the highest level, we can distinguish **paid** traffic (that derives from ad spend) with **non-paid** traffic: visitors who come to the website by entering the URL directly, clicking on a link from a referrer site or clicking on an organic link returned in a search results, for example.

In order to identify **paid** traffic, Snowplow users need to set five query parameters on the links used in ads. Snowplow checks for the presence of these query parameters on the web pages that users load: if it finds them, it knows that that user came from a paid source, and stores the values of those parameters so that it is possible to identify the paid source of traffic exactly.

If the query parameters are not present, Snowplow reasons that the user is from a **non-paid** source of traffic. It then checks the page referrer (the url of the web page the user was on before visiting our website), and uses that to deduce the source of traffic:

1. If the URL is identified as a search engine, the traffic medium is set to "organic" and Snowplow tries to derive the search engine name from the referrer URL domain and the keywords from the query string.
2. If the URL is a non-search 3rd party website, the medium is set to "referrer". Snowplow derives the source from the referrer URL domain.

Campaign information is **automatically tracked**.

### Identifying paid sources

Your different ad campaigns (PPC campaigns, display ads, email marketing messages, Facebook campaigns etc.) will include one or more links to your website e.g.:

```html
<a href="http://mysite.com/myproduct.html">Visit website</a>
```

We want to be able to identify people who've clicked on ads e.g. in a marketing email as having come to the site having clicked on a link in that particular marketing email. To do that, we modify the link in the marketing email with query parameters, like so:

```html
<a href="http://mysite.com/myproduct.html?utm_source=newsletter-october&utm_medium=email&utm_campaign=cn0201">Visit website</a>
```

For the prospective customer clicking on the link, adding the query parameters does not change the user experience. (The user is still directed to the webpage at `http://mysite.com/myproduct.html`.) But Snowplow then has access to the fields given in the query string, and uses them to identify this user as originating from the October Newsletter, an email marketing campaign with campaign id = cn0201.

### Anatomy of the query parameters

Snowplow uses the same query parameters used by Google Analytics. Because of this, Snowplow users who are also using GA do not need to do any additional work to make their campaigns trackable in Snowplow as well as GA. Those parameters are:

| **Parameter**  | **Name**         | **Description**                                                                                                                                       |
| -------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `utm_source`   | Campaign source  | Identify the advertiser driving traffic to your site e.g. Google, Facebook, autumn-newsletter etc.                                                    |
| `utm_medium`   | Campaign medium  | The advertising / marketing medium e.g. cpc, banner, email newsletter, in-app ad, cpa                                                                 |
| `utm_campaign` | Campaign id      | A unique campaign id. This can be a descriptive name or a number / string that is then looked up against a campaign table as part of the analysis     |
| `utm_term`     | Campaign term(s) | Used for search marketing in particular, this field is used to identify the search terms that triggered the ad being displayed in the search results. |
| `utm_content`  | Campaign content | Used either to differentiate similar content or two links in the same ad. (So that it is possible to identify which is generating more traffic.)      |

The parameters are described in the [Google Analytics help page](https://support.google.com/analytics/answer/1033863). Google also provides a [urlbuilder](https://support.google.com/analytics/answer/1033867?hl=en) which can be used to construct the URL incl. query parameters to use in your campaigns.

---

# Track User-Agent Client Hints on web
> Capture browser information through Client Hints as an alternative to user-agent strings with basic and high-entropy options.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/client-hints/

User-Agent [Client Hints](https://www.chromium.org/updates/ua-ch) are being rolled out across a number of browsers and are an alternative to the tracking the user-agent, which is particularly useful in those browsers which are freezing the user-agent string. See [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-CH#Browser_compatibility) for browser support.

This is useful data to capture as browsers are moving away from high entropy user-agent strings. Client Hints offer useful information to understand browser usage without the potential to infringe on a users privacy as is often the case with the user-agent string.

This entity can be configured in two ways:

1. `clientHints: true`: captures the "basic" client hints `isMobile` and `brands`.
2. `clientHints: { includeHighEntropy: true }`: captures the "basic" client hints as well as hints that are deemed "High Entropy" and could be used to fingerprint users. Browsers may choose to prompt the user before making this data available.

The high entropy properties are `architecture`, `model`, `platform`, `platformVersion`, and `uaFullVersion`.

For the full schema details see the [device and browser tracking](/docs/events/ootb-data/device-and-browser/#client-hints-entity) overview page.

The Client Hints entity is **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                      |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-client-hints@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-client-hints@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
// Basic
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-client-hints@latest/dist/index.umd.min.js",
  ["snowplowClientHints", "ClientHintsPlugin"]
);

// High Entropy
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-client-hints@latest/dist/index.umd.min.js",
  ["snowplowClientHints", "ClientHintsPlugin"],
  { includeHighEntropy: true }
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-client-hints`
- `yarn add @snowplow/browser-plugin-client-hints`
- `pnpm add @snowplow/browser-plugin-client-hints`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { ClientHintsPlugin } from '@snowplow/browser-plugin-client-hints';

// Basic
newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ ClientHintsPlugin() ],
});

// High Entropy
newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ ClientHintsPlugin(true) ],
});
```

***

---

# Track consent and GDPR on web
> Track user consent preferences and GDPR compliance with enhanced consent events for acceptance, selection, denial, expiration, and withdrawal.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/consent-gdpr/

Track user consent preferences selection events using the Enhanced Consent plugin.

Enhanced consent events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                          |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                    |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-enhanced-consent@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-enhanced-consent@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-enhanced-consent@latest/dist/index.umd.min.js',
    ['snowplowEnhancedConsentTracking', 'EnhancedConsentPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-enhanced-consent`
- `yarn add @snowplow/browser-plugin-enhanced-consent`
- `pnpm add @snowplow/browser-plugin-enhanced-consent`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { EnhancedConsentPlugin } from '@snowplow/browser-plugin-enhanced-consent';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ EnhancedConsentPlugin() ],
});
```

***

> **Note:** The plugin is available since version 3.8 of the tracker.

## Events

| API                     | To track                                                |
| ----------------------- | ------------------------------------------------------- |
| `trackConsentAllow`     | Acceptance of user consent                              |
| `trackConsentSelected`  | A specific selection of consented scopes                |
| `trackConsentPending`   | The unconfirmed selection about user consent            |
| `trackConsentImplicit`  | The implicit consent on user consent preferences        |
| `trackConsentDeny`      | A denial of user consent                                |
| `trackConsentExpired`   | The expiration of a consent selection                   |
| `trackConsentWithdrawn` | The withdrawal of user consent                          |
| `trackCmpVisible`       | The render time of a consent management platform banner |

With the exception of the CMP visible event, these methods use the same [`consent_preferences`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/consent_preferences/jsonschema/1-0-0) event schema.

### Consent allow

To track the complete acceptance of a user consent prompt, you can use the `trackConsentAllow` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentAllow:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentAllow } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentAllow({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent selected

To track the specific selection of scopes of a user consent preferences, you can use the `trackConsentSelected` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentSelected:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentSelected } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentSelected({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent pending

Some consent management platform installations, allow the user to take website actions or/and navigating to other pages without accepting the consent prompt. To track the unconfirmed selection of user consent, you can use the `trackConsentPending` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentPending:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentPending } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentPending({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent implicit

Some consent management platforms have a configuration which allows the setting of consent implicitly after a set of user interactions like clicks, scroll etc. To track the implicit selection of a user consent, you can use the `trackConsentImplicit` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentImplicit:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentImplicit } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentImplicit({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent deny

To track the complete denial of a user consent, you can use the `trackConsentDeny` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentDeny:{trackerName}", {
    consentScopes: ["necessary"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentDeny } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentDeny({
    consentScopes: ["necessary"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent expired

To track the expiration of a user consent, you can use the `trackConsentExpired` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentExpired:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentExpired } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentExpired({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### Consent withdrawn

To track the withdrawal of a user consent, you can use the `trackConsentWithdrawn` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackConsentWithdrawn:{trackerName}", {
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

**Browser (npm):**

```js
import { trackConsentWithdrawn } from "@snowplow/browser-plugin-enhanced-consent";

trackConsentWithdrawn({
    consentScopes: ["necessary", "marketing", "personalization"],
    basisForProcessing: "consent",
    consentUrl: "https://www.example.com/",
    consentVersion: "1.0",
    domainsApplied: ["https://www.example.com/"],
    gdprApplies: true
});
```

***

### CMP visible

Consent management platform banners are an important part of a website’s first impression and performance. Snowplow provides a way to track what we call `elapsedTime`, which is the timestamp of the consent management platform banner becoming visible on the user’s screen.

**JavaScript (tag):**

```js
window.snowplow("trackCmpVisible:{trackerName}", {
    /* Using the performance.now API to retrieve the elapsed time from the page navigation. */
    elapsedTime: performance.now(),
});
```

**Browser (npm):**

```js
import { trackCmpVisible } from "@snowplow/browser-plugin-enhanced-consent";

trackCmpVisible({
    /* Using the performance.now API to retrieve the elapsed time from the page navigation. */
    elapsedTime: performance.now(),
});
```

***

The CMP visible event uses [this](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/cmp_visible/jsonschema/1-0-0) schema.

---

# Legacy enhanced ecommerce plugin for web
> Legacy plugin based on Google Analytics Enhanced Ecommerce that has been superseded by the newer Snowplow ecommerce plugin.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/ecommerce/enhanced/

> **Warning:** This plugin has been deprecated and superseded by the [Snowplow ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/). We highly recommend using this newer plugin, which is more fully featured and allows you to use the [Snowplow Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) dbt model.

This plugin is based on Google Analytics' Enhanced Ecommerce package. For more information on the Enhanced Ecommerce functions please see the Google Analytics [documentation](https://developers.google.com/analytics/devguides/collection/analyticsjs/enhanced-ecommerce).

Enhanced ecommerce events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                      |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-enhanced-ecommerce@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-enhanced-ecommerce@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-enhanced-ecommerce@latest/dist/index.umd.min.js",
  ["snowplowEnhancedEcommerce", "EnhancedEcommercePlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-enhanced-ecommerce`
- `yarn add @snowplow/browser-plugin-enhanced-ecommerce`
- `pnpm add @snowplow/browser-plugin-enhanced-ecommerce`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { EnhancedEcommercePlugin, trackEnhancedEcommerceAction } from '@snowplow/browser-plugin-enhanced-ecommerce';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ EnhancedEcommercePlugin() ],
});
```

***

## Event

The enhanced ecommerce plugin is based around the `EnhancedEcommerceAction` event, to which can be added the Action, Impression, Product and Promo context entities. The context entities must be added first, before the event is tracked.

Use the `trackEnhancedEcommerceAction` method to track a GA Enhanced Ecommerce Action. When this function is called all of the added Ecommerce context entities are attached to this action and flushed from the tracker.

| **Name** | **Required?** | **Type** |
| -------- | ------------- | -------- |
| `action` | Yes           | string   |

The allowed actions:

- `click`
- `detail`
- `add`
- `remove`
- `checkout`
- `checkout_option`
- `purchase`
- `refund`
- `promo_click`
- `view`

Adding an action using Google Analytics:

```javascript
ga('ec:setAction', 'refund', {
  'id': 'T12345'
});
```

Adding an action using Snowplow:

**JavaScript (tag):**

```javascript
snowplow('addEnhancedEcommerceActionContext', {
  id: 'T12345'
});
snowplow('trackEnhancedEcommerceAction', {
  action: 'refund'
});
```

**Browser (npm):**

```javascript
addEnhancedEcommerceActionContext({ id: 'T12345' });
trackEnhancedEcommerceAction({ action: 'refund' });
```

***

## Context entities

The enhanced ecommerce context entities are specific to this plugin, and cannot be added to any other event types.

### Action

Use the `addEnhancedEcommerceActionContext` method to add a GA Enhanced Ecommerce Action Context to the Tracker:

| **Name**      | **Required?** | **Type**          |
| ------------- | ------------- | ----------------- |
| `id`          | Yes           | string            |
| `affiliation` | No            | string            |
| `revenue`     | No            | number OR string  |
| `tax`         | No            | number OR string  |
| `shipping`    | No            | number OR string  |
| `coupon`      | No            | string            |
| `list`        | No            | string            |
| `step`        | No            | integer OR string |
| `option`      | No            | string            |
| `currency`    | No            | string            |

Adding an action using Google Analytics:

```javascript
ga('ec:setAction', 'purchase', {
  'id': 'T12345',
  'affiliation': 'Google Store - Online',
  'revenue': '37.39',
  'tax': '2.85',
  'shipping': '5.34',
  'coupon': 'SUMMER2013'
});
```

> **Note:** The action type is passed with the action context in the Google Analytics example. We have separated this by asking you to call the `trackEnhancedEcommerceAction` function to actually send the context and the action.

Adding an action using Snowplow:

**JavaScript (tag):**

````javascript
snowplow('addEnhancedEcommerceActionContext', {
  id: 'T12345',
  affiliation: 'Google Store - Online',
  revenue: '37.39', // Can also pass as number
  tax: '2.85', // Can also pass as number
  shipping: '5.34', // Can also pass as number
  coupon: 'WINTER2016'
});
``
  </TabItem>
  <TabItem value="browser" label="Browser (npm)">

```javascript
addEnhancedEcommerceActionContext({
  id: 'T12345',
  affiliation: 'Google Store - Online',
  revenue: '37.39', // Can also pass as number
  tax: '2.85', // Can also pass as number
  shipping: '5.34', // Can also pass as number
  coupon: 'WINTER2016'
});
````

***

### Impression

Use the `addEnhancedEcommerceImpressionContext` method to add a GA Enhanced Ecommerce Impression Context to the Tracker:

| **Name**   | **Required?** | **Type**          |
| ---------- | ------------- | ----------------- |
| `id`       | Yes           | string            |
| `name`     | No            | string            |
| `list`     | No            | string            |
| `brand`    | No            | string            |
| `category` | No            | string            |
| `variant`  | No            | string            |
| `position` | No            | integer OR string |
| `price`    | No            | number OR string  |
| `currency` | No            | string            |

Adding an impression using Google Analytics:

```javascript
ga('ec:addImpression', {
  'id': 'P12345',
  'name': 'Android Warhol T-Shirt',
  'list': 'Search Results',
  'brand': 'Google',
  'category': 'Apparel/T-Shirts',
  'variant': 'Black',
  'position': 1
});
```

Adding an impression using Snowplow:

**JavaScript (tag):**

```javascript
snowplow('addEnhancedEcommerceImpressionContext', {
  id: 'P12345',
  name: 'Android Warhol T-Shirt',
  list: 'Search Results',
  brand: 'Google',
  category: 'Apparel/T-Shirts',
  variant: 'Black',
  position: 1
});
```

**Browser (npm):**

```javascript
addEnhancedEcommerceImpressionContext({
  id: 'P12345',
  name: 'Android Warhol T-Shirt',
  list: 'Search Results',
  brand: 'Google',
  category: 'Apparel/T-Shirts',
  variant: 'Black',
  position: 1
});
```

***

### Product

Use the `addEnhancedEcommerceProductContext` method to add a GA Enhanced Ecommerce Product Field Context:

| **Name**   | **Required?** | **Type**          |
| ---------- | ------------- | ----------------- |
| `id`       | Yes           | string            |
| `name`     | No            | string            |
| `list`     | No            | string            |
| `brand`    | No            | string            |
| `category` | No            | string            |
| `variant`  | No            | string            |
| `price`    | No            | number OR string  |
| `quantity` | No            | integer OR string |
| `coupon`   | No            | string            |
| `position` | No            | integer OR string |
| `currency` | No            | string            |

Adding a product using Google Analytics:

```javascript
ga('ec:addProduct', {
  'id': 'P12345',
  'name': 'Android Warhol T-Shirt',
  'brand': 'Google',
  'category': 'Apparel/T-Shirts',
  'variant': 'Black',
  'position': 1
});
```

Adding a product using Snowplow:

**JavaScript (tag):**

```javascript
snowplow('addEnhancedEcommerceProductContext', {
  id: 'P12345',
  name: 'Android Warhol T-Shirt',
  list: 'Search Results',
  brand: 'Google',
  category: 'Apparel/T-Shirts',
  variant: 'Black',
  quantity: 1
});
```

**Browser (npm):**

```javascript
addEnhancedEcommerceProductContext({
  id: 'P12345',
  name: 'Android Warhol T-Shirt',
  list: 'Search Results',
  brand: 'Google',
  category: 'Apparel/T-Shirts',
  variant: 'Black',
  quantity: 1
});
```

***

### Promo

Use the `addEnhancedEcommercePromoContext` method to add a GA Enhanced Ecommerce Promotion Field Context:

| **Name**   | **Required?** | **Type** |
| ---------- | ------------- | -------- |
| `id`       | Yes           | string   |
| `name`     | No            | string   |
| `creative` | No            | string   |
| `position` | No            | string   |
| `currency` | No            | string   |

Adding a promotion using Google Analytics:

```javascript
ga('ec:addPromo', {
  'id': 'PROMO_1234',
  'name': 'Summer Sale',
  'creative': 'summer_banner2',
  'position': 'banner_slot1'
});
```

Adding a promotion using Snowplow:

**JavaScript (tag):**

```javascript
snowplow('addEnhancedEcommercePromoContext', {
  id: 'PROMO_1234', // The Promotion ID
  name: 'Summer Sale', // The name
  creative: 'summer_banner2', // The name of the creative
  position: 'banner_slot1' // The position
});
```

**Browser (npm):**

```javascript
addEnhancedEcommercePromoContext({
  id: 'PROMO_1234', // The Promotion ID
  name: 'Summer Sale', // The name
  creative: 'summer_banner2', // The name of the creative
  position: 'banner_slot1' // The position
});
```

***

---

# Track ecommerce events on web
> Track comprehensive ecommerce interactions including product views, cart actions, checkout steps, transactions, and refunds with standardized event schemas.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/ecommerce/

This plugin helps you track ecommerce activity. See the full details for all ecommerce schemas in the [ecommerce tracking overview](/docs/events/ootb-data/ecommerce-events/) page.

It provides several `trackX` methods, which each create a Snowplow ecommerce action event with the appropriate entities attached. The event itself has only one property, an enum describing the ecommerce action taken e.g. `add_to_cart`.

There are also two entities that can be globally configured: ecommerce page and ecommerce user.

This plugin is supported by the [Snowplow Ecommerce](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-ecommerce-data-model/) dbt model.

> **Note:** The plugin is available since version 3.8 of the tracker.

Snowplow ecommerce events and entities must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                      |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-snowplow-ecommerce@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-snowplow-ecommerce@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-snowplow-ecommerce@latest/dist/index.umd.min.js',
    ['snowplowEcommerceAccelerator', 'SnowplowEcommercePlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-snowplow-ecommerce`
- `yarn add @snowplow/browser-plugin-snowplow-ecommerce`
- `pnpm add @snowplow/browser-plugin-snowplow-ecommerce`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { SnowplowEcommercePlugin } from '@snowplow/browser-plugin-snowplow-ecommerce';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SnowplowEcommercePlugin() ],
});
```

***

## Events

| API                     | Used for:                                                                                                                                      |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `trackProductView`      | Tracking a visit to a product page. Known also as product detail view.                                                                         |
| `trackAddToCart`        | Track an addition to cart.                                                                                                                     |
| `trackRemoveFromCart`   | Track a removal from cart.                                                                                                                     |
| `trackProductListView`  | Track an impression of a product list. The list could be a search results page, recommended products, upsells etc.                             |
| `trackProductListClick` | Track the click/selection of a product from a product list.                                                                                    |
| `trackPromotionView`    | Track an impression for an internal promotion banner or slider or any other type of content that showcases internal products/categories.       |
| `trackPromotionClick`   | Track the click/selection of an internal promotion.                                                                                            |
| `trackCheckoutStep`     | Track a checkout step completion in the checkout process together with common step attributes for user choices throughout the checkout funnel. |
| `trackTransaction`      | Track a transaction/purchase completion.                                                                                                       |
| `trackRefund`           | Track a transaction partial or complete refund.                                                                                                |
| `trackTransactionError` | Track an error happening during a transaction process.                                                                                         |

### Product view

To track a product view, use the `trackProductView` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackProductView:{trackerName}", {
    id: "12345",
    name: "Baseball T",
    brand: "Snowplow",
    category: "apparel",
    price: 200,
    currency: "USD",
});
```

**Browser (npm):**

```js
import { trackProductView } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackProductView({
    id: "12345",
    name: "Baseball T",
    brand: "Snowplow",
    category: "apparel",
    price: 200,
    currency: "USD",
});
```

***

### Add to cart

To track products being added to the cart, use the `trackAddToCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackAddToCart:{trackerName}", {
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      currency: "USD",
    },
  ],
  total_value: 200,
  currency: "USD",
});
```

**Browser (npm):**

```js
import { trackAddToCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackAddToCart({
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      currency: "USD",
    },
  ],
  total_value: 200,
  currency: "USD",
});
```

***

- Where `products` is an array with the products added to cart.
- Where `total_value` is the value of the cart after the addition.
- Where `currency` is the currency of the cart.

### Remove from cart

To track products being removed from the cart, use the `trackRemoveFromCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackRemoveFromCart:{trackerName}", {
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      currency: "USD",
    },
  ],
  total_value: 0,
  currency: "USD",
});
```

**Browser (npm):**

```js
import { trackRemoveFromCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackRemoveFromCart({
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      currency: "USD",
    },
  ],
  total_value: 0,
  currency: "USD",
});
```

***

- Where `products` is an array with the products removed from the cart.
- Where `total_value` is the value of the cart after the removal of the products.
- Where `currency` is the currency of the cart.

### Product list view

To track a product list view, use the `trackProductListView` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackProductListView:{trackerName}", {
  products: [
    {
      id: "P123",
      name: "Fashion red",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 100,
      inventory_status: "in stock",
      currency: "USD",
      position: 1,
    },
    {
      id: "P124",
      name: "Fashion green",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 119,
      inventory_status: "in stock",
      currency: "USD",
      position: 2,
    },
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      position: 3,
    },
  ],
  name: "Recommended Products",
});
```

**Browser (npm):**

```js
import { trackProductListView } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackProductListView({
  products: [
    {
      id: "P123",
      name: "Fashion red",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 100,
      inventory_status: "in stock",
      currency: "USD",
      position: 1,
    },
    {
      id: "P124",
      name: "Fashion green",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 119,
      inventory_status: "in stock",
      currency: "USD",
      position: 2,
    },
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      position: 3,
    },
  ],
  name: "Recommended Products",
});
```

***

- Where `products` is an array of products being viewed from the list.
- Where `name` is the name of the list being viewed. For the list names, you can use any kind of friendly name or a codified language to express the labeling of the list. E.g. `Shoes - Men - Sneakers`, `Search results: "unisex shoes"`, or `Product page upsells`.

### Product list click

**JavaScript (tag):**

```js
window.snowplow("trackProductListClick:{trackerName}", {
  product: {
    id: "P124",
    name: "Fashion green",
    brand: "Snowplow",
    category: "Mens/Apparel",
    price: 119,
    inventory_status: "in stock",
    currency: "USD",
    position: 2,
  },
  name: "Recommended Products",
});
```

**Browser (npm):**

```js
import { trackProductListClick } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackProductListClick({
  product: {
    id: "P124",
    name: "Fashion green",
    brand: "Snowplow",
    category: "Mens/Apparel",
    price: 119,
    inventory_status: "in stock",
    currency: "USD",
    position: 2,
  },
  name: "Recommended Products",
});
```

***

- Where `product` is the product being clicked or selected from the list.
- Where `name` is the name of the list the product is in. For the list names, you can use any kind of friendly name or a codified language to express the labeling of the list. E.g. `Shoes - Men - Sneakers`, `Search results: "unisex shoes"`, or `Product page upsells`.

### Promotion view

**JavaScript (tag):**

```js
/* Carousel slide 1 viewed */
window.snowplow("trackPromotionView:{trackerName}", {
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 1,
    product_ids: ['P1234'],
});

/* On carousel slide 2 view */
window.snowplow("trackPromotionView:{trackerName}", {
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 2,
    product_ids: ['P1235'],
});
```

**Browser (npm):**

```js
import { trackPromotionView } from '@snowplow/browser-plugin-snowplow-ecommerce';

/* Carousel slide 1 viewed */
trackPromotionView({
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 1,
    product_ids: ['P1234'],
});

/* On carousel slide 2 view */
trackPromotionView({
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 2,
    product_ids: ['P1235'],
});
```

***

### Promotion click

**JavaScript (tag):**

```js
window.snowplow("trackPromotionClick:{trackerName}", {
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 1,
    product_ids: ['P1234'],
});
```

**Browser (npm):**

```js
import { trackPromotionClick } from "@snowplow/browser-plugin-snowplow-ecommerce";

trackPromotionClick({
    id: 'IP1234',
    name: 'promo_winter',
    type: 'carousel',
    position: 1,
    product_ids: ['P1234'],
});
```

***

### Checkout step

To track a checkout step, use the `trackCheckoutStep` method with the following attributes:

**JavaScript (tag):**

```js
/* Step 1 - Account type selection */
window.snowplow("trackCheckoutStep:{trackerName}", {
  step: 1,
  account_type: "guest checkout",
});

/* Step 2 - Billing options selection */
window.snowplow("trackCheckoutStep:{trackerName}", {
  step: 2,
  payment_method: "credit card",
  proof_of_payment: "invoice",
});
```

**Browser (npm):**

```js
import { trackCheckoutStep } from '@snowplow/browser-plugin-snowplow-ecommerce';

/* Step 1 - Account type selection */
trackCheckoutStep({
  step: 1,
  account_type: "guest checkout",
});

/* Step 2 - Billing options selection */
trackCheckoutStep({
  step: 2,
  payment_method: "credit card",
  proof_of_payment: "invoice",
});
```

***

### Transaction

To track a completed transaction, use the `trackTransaction` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackTransaction:{trackerName}", {
  transaction_id: "T12345",
  revenue: 230,
  currency: "USD",
  payment_method: "credit_card",
  total_quantity: 1,
  tax: 20,
  shipping: 10,
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      quantity: 1,
    },
  ],
});
```

**Browser (npm):**

```js
import { trackTransaction } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackTransaction({
  transaction_id: "T12345",
  revenue: 230,
  currency: "USD",
  payment_method: "credit_card",
  total_quantity: 1,
  tax: 20,
  shipping: 10,
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      quantity: 1,
    },
  ],
});
```

***

- Where `products` is an array with the products taking part in the transaction.

### Refund

> **Note:** Available from version 3.10.

To track a complete or partial refund you can use the `trackRefund` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackRefund:{trackerName}", {
  transaction_id: "T12345",
  currency: "USD",
  refund_amount: 200,
  refund_reason: "return",
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      quantity: 1,
    },
  ],
});
```

**Browser (npm):**

```js
import { trackRefund } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackRefund({
  transaction_id: "T12345",
  currency: "USD",
  refund_amount: 200,
  refund_reason: "return",
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      inventory_status: "in stock",
      currency: "USD",
      quantity: 1,
    },
  ],
});
```

***

- Where `products` is an array with the products taking part in the refund.

### Transaction error

> **Note:** Available from version 3.13.

To track an error happening during a transaction process, use the `trackTransactionError` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackTransactionError:{trackerName}", {
  resolution: "rejection",
  error_code: "E123",
  error_shortcode: "CARD_DECLINE",
  error_description: "Card has been declined by the issuing bank.",
  error_type: "hard",
  transaction: {
    revenue: 45,
    currency: "EUR",
    transaction_id: "T12345",
    payment_method: "card",
    total_quantity: 1
  }
});
```

**Browser (npm):**

```js
import { trackTransactionError } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackTransactionError({
  resolution: "rejection",
  error_code: "E123",
  error_shortcode: "CARD_DECLINE",
  error_description: "Card has been declined by the issuing bank.",
  error_type: "hard",
  transaction: {
    revenue: 45,
    currency: "EUR",
    transaction_id: "T12345",
    payment_method: "card",
    total_quantity: 1
  }
});
```

***

- Where `transaction` is the transaction entity being processed.

## Page and user entities

Once set, the [ecommerce page or user entity](/docs/events/ootb-data/ecommerce-events/#global-ecommerce-entities) will be attached to **all** subsequent Snowplow events.

There's no way to unset these entities.

**JavaScript (tag):**

```js
window.snowplow("setPageType:{trackerName}", { type, language, locale });

window.snowplow("setEcommerceUser:{trackerName}", { id, is_guest, email });
```

**Browser (npm):**

```js
import { setPageType } from '@snowplow/browser-plugin-snowplow-ecommerce';

setPageType({ type, language, locale });

setEcommerceUser({ id, is_guest, email });
```

***

## GA4/UA Ecommerce transitional API

> **Note:** Available from version 3.10.

If you already use Google Analytics 4 ecommerce or Universal Analytics Enhanced Ecommerce to collect information about the shopping behavior of your users, we've prepared a way to quickly implement Snowplow Ecommerce without making many changes on your current setup.

This transitional API depends on the standardized [dataLayer](https://developers.google.com/tag-platform/tag-manager/web/datalayer) structure for both Google Analytics ecommerce implementations. This would make it easier for the transition to happen either through Google Tag Manager, which has more control over the dataLayer, or custom code that uses the standard ecommerce structures.

### Universal Analytics Enhanced Ecommerce

The standard Universal Analytics Enhanced Ecommerce implementation is based on the official [guide reference](https://developers.google.com/analytics/devguides/collection/ua/gtm/enhanced-ecommerce).

**Important:** The `dataLayer.currencyCode` attribute must be available for all product interactions. Otherwise, almost all methods accept an `Options` object which can include the currency code as follows:

```ts
method({{dataLayer.ecommerce reference}} , { currency: "currency code" });
```

#### trackEnhancedEcommerceProductListView

To track an Enhanced Ecommerce product list view, you can use the `trackEnhancedEcommerceProductListView` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceProductListView:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceProductListView } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceProductListView({{dataLayer.ecommerce reference}});
```

***

#### trackEnhancedEcommerceProductListClick

To track an Enhanced Ecommerce product list click, you can use the `trackEnhancedEcommerceProductListClick` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceProductListClick:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceProductListClick } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceProductListClick({{dataLayer.ecommerce reference}});
```

***

#### trackEnhancedEcommerceProductDetail

To track an Enhanced Ecommerce product detail view, you can use the `trackEnhancedEcommerceProductDetail` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceProductDetail:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceProductDetail } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceProductDetail({{dataLayer.ecommerce reference}});
```

***

#### trackEnhancedEcommercePromoView

To track an Enhanced Ecommerce internal promotion view, you can use the `trackEnhancedEcommercePromoView` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommercePromoView:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackEnhancedEcommercePromoView } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommercePromoView({{dataLayer.ecommerce reference}});
```

***

#### trackEnhancedEcommercePromoClick

To track an Enhanced Ecommerce internal promotion click, you can use the `trackEnhancedEcommercePromoClick` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommercePromoClick:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackEnhancedEcommercePromoClick } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommercePromoClick({{dataLayer.ecommerce reference}});
```

***

#### trackEnhancedEcommerceAddToCart

To track an Enhanced Ecommerce add to cart event, you can use the `trackEnhancedEcommerceAddToCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceAddToCart:{trackerName}", {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceAddToCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceAddToCart( {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

***

- Where `finalCartValue` is the value of the cart after the addition.

#### trackEnhancedEcommerceRemoveFromCart

To track an Enhanced Ecommerce remove from cart event, you can use the `trackEnhancedEcommerceRemoveFromCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceRemoveFromCart:{trackerName}", {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceRemoveFromCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceRemoveFromCart( {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

***

- Where `finalCartValue` is the value of the cart after the removal.

#### trackEnhancedEcommerceCheckoutStep

To track an Enhanced Ecommerce remove from cart event, you can use the `trackEnhancedEcommerceCheckoutStep` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommerceCheckoutStep:{trackerName}", {{dataLayer.ecommerce reference}}, {
    checkoutOption: { delivery_method: "express_delivery" },
});
```

**Browser (npm):**

```js
import { trackEnhancedEcommerceCheckoutStep } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommerceCheckoutStep( {{dataLayer.ecommerce reference}}, {
    checkoutOption: { delivery_method: "express_delivery" },
});
```

***

- Where `checkoutOption` is a key value pair object of available [Snowplow checkout options](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow.ecommerce/checkout_step), except `step` which is retrieved from the dataLayer directly.

#### trackEnhancedEcommercePurchase

To track an Enhanced Ecommerce remove from cart event, you can use the `trackEnhancedEcommercePurchase` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackEnhancedEcommercePurchase:{trackerName}", {{dataLayer.ecommerce reference}}, {
    paymentMethod: "bank_transfer",
});
```

**Browser (npm):**

```js
import { trackEnhancedEcommercePurchase } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackEnhancedEcommercePurchase( {{dataLayer.ecommerce reference}}, {
    paymentMethod: "bank_transfer",
});
```

***

- Where `paymentMethod` is the payment method selected in this transaction. This attributes corresponds to the `payment_method` of the [transaction schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.ecommerce/transaction/jsonschema/1-0-0#L30). Defaults to `unknown`.

### Google Analytics 4 Ecommerce

The Google Analytics 4 ecommerce implementation is based on the official [guide reference](https://developers.google.com/analytics/devguides/collection/ga4/ecommerce?client_type=gtm).

**Important:** The `dataLayer.ecommerce.currency` attribute must be available for all product interactions. Otherwise, almost all methods accept an `Options` object which can include the currency code as follows:

```ts
method( {{dataLayer.ecommerce reference}} , { currency: "currency code" });
```

#### trackGA4ViewItemList

To track an GA4 Ecommerce item list view, you can use the `trackGA4ViewItemList` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4ViewItemList:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackGA4ViewItemList } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4ViewItemList({{dataLayer.ecommerce reference}});
```

***

#### trackGA4SelectItem

To track an GA4 Ecommerce item selection from a list, you can use the `trackGA4SelectItem` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4SelectItem:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackGA4SelectItem } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4SelectItem({{dataLayer.ecommerce reference}});
```

***

#### trackGA4ViewItem

To track an GA4 Ecommerce item view, you can use the `trackGA4ViewItem` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4ViewItem:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackGA4ViewItem } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4ViewItem({{dataLayer.ecommerce reference}});
```

***

#### trackGA4ViewPromotion

To track an GA4 Ecommerce internal promotion view, you can use the `trackGA4ViewPromotion` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4ViewPromotion:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackGA4ViewPromotion } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4ViewPromotion({{dataLayer.ecommerce reference}});
```

***

#### trackGA4SelectPromotion

To track an GA4 Ecommerce internal promotion selection, you can use the `trackGA4SelectPromotion` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4SelectPromotion:{trackerName}", {{dataLayer.ecommerce reference}});
```

**Browser (npm):**

```js
import { trackGA4SelectPromotion } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4SelectPromotion( {{dataLayer.ecommerce reference}});
```

***

#### trackGA4AddToCart

To track an GA4 Ecommerce add to cart event, you can use the `trackGA4AddToCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4AddToCart:{trackerName}", {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

**Browser (npm):**

```js
import { trackGA4AddToCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4AddToCart( {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

***

- Where `finalCartValue` is the value of the cart after the addition.

#### trackGA4RemoveFromCart

To track an GA4 Ecommerce remove from cart event, you can use the `trackGA4RemoveFromCart` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4RemoveFromCart:{trackerName}", {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

**Browser (npm):**

```js
import { trackGA4RemoveFromCart } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4RemoveFromCart( {{dataLayer.ecommerce reference}}, {
    finalCartValue: 20,
});
```

***

- Where `finalCartValue` is the value of the cart after the removal.

#### trackGA4BeginCheckout

To track an GA4 Ecommerce checkout beginning, you can use the `trackGA4BeginCheckout` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4BeginCheckout:{trackerName}", {
    step: 1,
});
```

**Browser (npm):**

```js
import { trackGA4BeginCheckout } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4BeginCheckout({
    step: 1,
});
```

***

- Where `step` is a number representing the step of the checkout funnel. Defaults to 1, mimicking the `begin_checkout` GA4 event.

#### trackGA4AddShippingInfo

To track an GA4 Ecommerce checkout shipping info step completion, you can use the `trackGA4AddShippingInfo` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4AddShippingInfo:{trackerName}", {
    step: 1,
});
```

**Browser (npm):**

```js
import { trackGA4AddShippingInfo } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4AddShippingInfo({
    step: 1,
});
```

***

- Where `step` is a number representing the step of the checkout funnel.

#### trackGA4AddPaymentOptions

To track an GA4 Ecommerce checkout payment option step completion, you can use the `trackGA4AddPaymentOptions` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4AddPaymentOptions:{trackerName}", {
    step: 1,
});
```

**Browser (npm):**

```js
import { trackGA4AddPaymentOptions } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4AddPaymentOptions({
    step: 1,
});
```

***

- Where `step` is a number representing the step of the checkout funnel.

#### trackGA4Transaction

To track an GA4 Ecommerce checkout payment option step completion, you can use the `trackGA4Transaction` method with the following attributes:

**JavaScript (tag):**

```js
window.snowplow("trackGA4Transaction:{trackerName}", {
    paymentMethod: "bank_transfer",
});
```

**Browser (npm):**

```js
import { trackGA4Transaction } from '@snowplow/browser-plugin-snowplow-ecommerce';

trackGA4Transaction({
    paymentMethod: "bank_transfer",
});
```

***

- Where `paymentMethod` is the payment method selected in this transaction. This attributes corresponds to the `payment_method` of the [transaction schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow.ecommerce/transaction/jsonschema/1-0-0#L30). Defaults to `unknown`.

---

# Track page element visibility and lifecycle on web
> Declaratively track page element visibility and lifecycle events as they are created, destroyed, scrolled into view, or scrolled out of view with configurable rules.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/element-tracking/

Element visibility tracking enables declarative tracking of page elements existing on web pages and scrolling into view.

This is useful for impression tracking, including:

- Funnel steps e.g. form on page > form in view > [form tracking events](/docs/sources/web-trackers/tracking-events/form-tracking/)
- List impression tracking e.g. product impressions
- Component performance e.g. recommendations performance, newsletter sign-up forms, modal popups
- Product usage e.g. elements that appear on-hover, labeling or grouping events related to specific features
- Advertisement impression tracking

Once you call `startElementTracking`, the plugin watches the DOM and automatically fires events whenever:

- Elements appear on the page: tracks `create_element`
- Elements scroll into view: tracks `expose_element`
- Elements scroll out of view: tracks `obscure_element`
- Elements are removed from the page: tracks `destroy_element`

You can define rules for which elements to track, and can also trigger events when elements change to match or no longer match a rule. An entity containing details about the element is attached to each event, and you can also configure other entities.

Element lifecycle events are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                          |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [GitHub Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                    |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-element-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-element-tracking@latest/dist/index.umd.min.js) (latest)               |

> **Note:** The links to the CDNs point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third-party CDN in production.

**Browser (npm):**

- `npm install @snowplow/browser-plugin-element-tracking`
- `yarn add @snowplow/browser-plugin-element-tracking`
- `pnpm add @snowplow/browser-plugin-element-tracking`

***

## Start element tracking

Begin tracking elements by providing configuration to the plugin's `startElementTracking` method:

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-element-tracking@latest/dist/index.umd.min.js",
  ["snowplowElementTracking", "SnowplowElementTrackingPlugin"]
);

snowplow('startElementTracking', { elements: [/* configuration */] });
```

**Browser (npm):**

First, add the plugin when initializing the tracker.

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SnowplowElementTrackingPlugin() ],
});

startElementTracking({ elements: [/* configuration */] });
```

***

The `elements` configuration can take a single rule, or an array of rules. You can call `startElementTracking` multiple times to add more rules as needed.

## Events and entities

The plugin can generate four events:

- `create_element`: when a matching element is added to the page
- `expose_element`: when a matching element scrolls into view
- `obscure_element`: when a matching element scrolls out of view
- `destroy_element`: when a matching element is removed from the page

Each of these events has only one property, `element_name`. Check out the [page element tracking overview](/docs/events/ootb-data/page-elements/#page-element-visibility-and-lifecycle) page to see the schema details.

Every element event includes an `element` entity with details about the element that triggered the event. The attributes tracked depend on your `detail` configuration.

By default, only the `expose_element` event is tracked. Configure which event types to track using booleans, or provide objects for more fine-grained control. Check out the configuration options on this page for details.

**JavaScript (tag):**

```javascript
// This minimal example tracks expose events for all `.product-card` elements
snowplow('startElementTracking', {
  elements: { selector: '.product-card' }
});

// It's equivalent to this more explicit configuration
snowplow('startElementTracking', {
  elements: {
    selector: '.product-card',
    create: false,    // won't fire when element added to DOM
    expose: true,     // WILL fire when element scrolls into view
    obscure: false,   // won't fire when element scrolls out of view
    destroy: false    // won't fire when element removed from DOM
  }
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

// This minimal example tracks expose events for all `.product-card` elements
startElementTracking({
  elements: { selector: '.product-card' }
});

// It's equivalent to this more explicit configuration
startElementTracking({
  elements: {
    selector: '.product-card',
    create: false,    // won't fire when element added to DOM
    expose: true,     // WILL fire when element scrolls into view
    obscure: false,   // won't fire when element scrolls out of view
    destroy: false    // won't fire when element removed from DOM
  }
});
```

***

### Example event

This example shows how to track an `expose_element` event as users scroll through a web page. All event types are configured similarly.

The example uses the `details` data selector option to specify what data to capture.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: {
    selector: "section", // Matches all <section> elements
    expose: { when: "element" }, // Fires when element becomes visible, once per element
    details: { child_text: { title: "h2" } } // Captures the main section header
  }
});
```

**Browser (npm):**

```javascript
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: {
    selector: "section", // Matches all <section> elements
    expose: { when: "element" }, // Fires when element becomes visible, once per element
    details: { child_text: { title: "h2" } } // Captures the main section header
  }
});
```

***

In this example, the page has several sections. As a user scrolls down the page and each section becomes visible, an `expose_element` event is generated for each one. All events will have `"element_name": "section"`.

Example `element` entity for the first section's `expose_element` event. The section title is "Why Data Teams Choose Snowplow":

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
  "data": {
    "element_name": "section",
    "width": 1920,
    "height": 1111.7333984375,
    "position_x": 0,
    "position_y": 716.4500122070312,
    "doc_position_x": 0,
    "doc_position_y": 716.4500122070312,
    "element_index": 2,
    "element_matches": 10,
    "originating_page_view": "06dbb0a2-9acf-4ae4-9562-1469b6d12c5d",
    "attributes": [
      {
        "source": "child_text",
        "attribute": "title",
        "value": "Why Data Teams Choose Snowplow"
      }
    ]
  }
}
```

Example `element` entity for the second section's `expose_element` event. The section title is "How Does Snowplow Work?":

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
  "data": {
    "element_name": "section",
    "width": 1920,
    "height": 2880,
    "position_x": 0,
    "position_y": 896.683349609375,
    "doc_position_x": 0,
    "doc_position_y": 1828.183349609375,
    "element_index": 3,
    "element_matches": 10,
    "originating_page_view": "06dbb0a2-9acf-4ae4-9562-1469b6d12c5d",
    "attributes": [
      {
        "source": "child_text",
        "attribute": "title",
        "value": "How Does Snowplow Work?"
      }
    ]
  }
}
```

## Stop element tracking

To turn off tracking, use `endElementTracking`. You can remove all configured rules, or selectively remove specific rules.

**JavaScript (tag):**

```javascript
// Remove all configured rules and listeners
snowplow('endElementTracking');

// Removes based on `name` matching
// Multiple rules may share a name
snowplow('endElementTracking', {
  elements: ['name1', 'name2']
});

// Removes rules based on `id` matching
// At most one rule can have the same `id`
snowplow('endElementTracking', {
  elementIds: ['id1']
});

// More complicated matching
// Rules where the `filter` function returns true will be removed
snowplow('endElementTracking', {
  filter: (rule) => /recommendations/i.test(rule.name)
});

// Passing an empty object removes no rules
snowplow('endElementTracking', {});
```

**Browser (npm):**

```javascript
import { SnowplowElementTrackingPlugin, endElementTracking } from '@snowplow/browser-plugin-element-tracking';

// Remove all configured rules and listeners
endElementTracking();

// Removes based on `name` matching
// Multiple rules may share a name
endElementTracking({
  elements: ['name1', 'name2']
});

// Removes rules based on `id` matching
// At most one rule can have the same `id`
endElementTracking({
  elementIds: ['id1']
});

// More complicated matching
// Rules where the `filter` function returns true will be removed
endElementTracking({
  filter: (rule) => /recommendations/i.test(rule.name)
});

// Passing an empty object removes no rules
endElementTracking({});
```

***

If you specify more than one of the `elementIds`, `elements`, and `filter` options, they get evaluated in that order.

## Configure entities

You can configure additional element tracking or custom entities by modifying the `startElementTracking` call.

Additional entities can be attached depending on configuration:

- `element_statistics`: visibility and scroll depth statistics for the element
- `element_content`: information about nested elements within the matched element
- `component_parents`: the component hierarchy that the element belongs to
- Custom entities

Check out the [page element tracking overview](/docs/events/ootb-data/page-elements/#page-element-visibility-and-lifecycle) page to see the schema details.

The configuration is per-rule, so different rules can have different settings.

### Element statistics

Use the `includeStats` option to attach the `element_statistics` entity to specified events, including those not generated by this plugin.

This example will add the `element_statistics` entity to `expose_element` and `page_ping` events:

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', { elements: {
  selector: 'main.article',
  name: 'article_content',
  includeStats: ['expose_element', 'page_ping']
} });
```

**Browser (npm):**

```javascript
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({ elements: {
  selector: 'main.article',
  name: 'article_content',
  includeStats: ['expose_element', 'page_ping']
} });
```

***

Adding element statistics to page pings can be useful to understand how a user moves through the content. It'll show scroll depth increasing over time, backtracking behavior, and total engagement duration.

For [baked-in events](/docs/fundamentals/events/#baked-in-events), use the following names:

- Page view: `page_view`
- Page ping: `page_ping`
- Structured: `event`

Be cautious with the `selector`. If it matches a lot of elements, this can enlarge event payload sizes.

### Element content

Add the `element_content` entity by setting `contents`. It captures data about specified nested elements within the matched parent element.

In this example, the plugin will track an `expose_element` event when a `.product-grid` element scrolls into view. This event will have an `element` entity for the grid itself, and multiple `element_content` entities for each `.product-card` within the grid.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: {
    selector: '.product-grid',
    name: 'product_list',
    expose: { when: 'element' },
    contents: [
      {
        selector: '.product-card',
        name: 'product_item',
        details: [
          { dataset: ['productId', 'price'] },
          { child_text: { name: 'h3', brand: '.brand-name' } }
        ]
      }
    ]
  }
});
```

**Browser (npm):**

```javascript
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: {
    selector: '.product-grid',
    name: 'product_list',
    expose: { when: 'element' },
    contents: [
      {
        selector: '.product-card',
        name: 'product_item',
        details: [
          { dataset: ['productId', 'price'] },
          { child_text: { name: 'h3', brand: '.brand-name' } }
        ]
      }
    ]
  }
});
```

***

The `details` configuration sets which element `attributes` to capture.

**Example entities for this configuration**

The `expose_element` event will have `"element_name": "product_list"`.

One `element` entity:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
  "data": {
    "element_name": "product_list",
    "element_index": 1,
    "element_matches": 1,
    "width": 1200,
    "height": 400,
    "attributes": []
  }
}
```

Multiple `element_content` entities:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0",
  "data": {
    "element_name": "product_item",
    "parent_name": "product_list", // element_name of parent element
    "parent_position": 1,          // element_index of parent element
    "position": 1,
    "attributes": [
      { "source": "dataset", "attribute": "productId", "value": "SKU-001" },
      { "source": "dataset", "attribute": "price", "value": "29.99" },
      { "source": "child_text", "attribute": "name", "value": "Wireless Mouse" },
      { "source": "child_text", "attribute": "brand", "value": "Logitech" }
    ]
  }
}
```

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0",
  "data": {
    "element_name": "product_item",
    "parent_name": "product_list", // element_name of parent element
    "parent_position": 1,          // element_index of parent element
    "position": 2,
    "attributes": [
      { "source": "dataset", "attribute": "productId", "value": "SKU-002" },
      { "source": "dataset", "attribute": "price", "value": "49.99" },
      { "source": "child_text", "attribute": "name", "value": "Mechanical Keyboard" },
      { "source": "child_text", "attribute": "brand", "value": "Keychron" }
    ]
  }
}
```

### Component parents

You can mark elements as components to track hierarchy, using `component` rules. Events for child elements include a `component_parents` entity listing their ancestor components.

This is useful when you have the same appearing in multiple places on your site. Without component tracking, all those events look identical.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: [
    // Define components (containers)
    {
      selector: 'header', // Mark the header as a component
      name: 'site_header',
      component: true,
      expose: false // Don't track expose events for the component itself
    },
    {
      selector: 'footer', // Mark the footer as a component
      name: 'site_footer',
      component: true,
      expose: false // Don't track expose events for the component itself
    },
    // Track elements - events will include component_parents
    {
      selector: '.newsletter-form',
      name: 'newsletter_signup',
      create: true, // Fire create_element events
      expose: { when: 'element' } // Fire expose_element events
    }
  ]
});
```

**Browser (npm):**

```javascript
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: [
    // Define components (containers)
    {
      selector: 'header', // Mark the header as a component
      name: 'site_header',
      component: true,
      expose: false // Don't track expose events for the component itself
    },
    {
      selector: 'footer', // Mark the footer as a component
      name: 'site_footer',
      component: true,
      expose: false // Don't track expose events for the component itself
    },
    // Track elements - events will include component_parents
    {
      selector: '.newsletter-form',
      name: 'newsletter_signup',
      create: true, // Fire create_element events
      expose: { when: 'element' } // Fire expose_element events
    }
  ]
});
```

***

For this example, imagine a page has two `.newsletter-form` elements: one in the page sidebar, and one in the footer.

The `component_parents` entity for the sidebar form, which isn't within either of the defined component containers, could look like this:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/component_parents/jsonschema/1-0-0",
  "data": {
    "element_name": "newsletter_signup",
    "component_list": []
  }
}
```

The `component_parents` entity for the footer form, which is within the `footer` component, could look like this:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/component_parents/jsonschema/1-0-0",
  "data": {
    "element_name": "newsletter_signup",
    "component_list": ["site_footer"]
  }
}
```

#### Generate entities for other events

The plugin also exposes a `getComponentListGenerator` utility function for attaching component hierarchy information to custom events, or to events generated by other plugins like the [form](/docs/sources/web-trackers/tracking-events/form-tracking/) or [link](/docs/sources/web-trackers/tracking-events/link-click/) tracking plugins.

This function returns two entity generator functions that determine component hierarchy for a given element:

- `componentGenerator`: returns a single `component_parents` entity
- `componentGeneratorWithDetail`: returns a `component_parents` entity plus an `element` entity

**JavaScript (tag):**

The JavaScript tracker uses a callback pattern to access the generators asynchronously:

```javascript
// This snippet assumes you've already defined component rules in startElementTracking

snowplow('getComponentListGenerator', function (componentGenerator, componentGeneratorWithDetail) {
   // attach the component_parents entity to events from these plugins
   snowplow('enableLinkClickTracking', { context: [componentGenerator] });
   snowplow('enableFormTracking', { context: [componentGenerator] });
});
```

**Browser (npm):**

The Browser tracker returns the generators directly as an array:

```javascript
// This snippet assumes you've already defined component rules in startElementTracking

import { getComponentListGenerator } from '@snowplow/browser-plugin-element-tracking';
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

const [componentGenerator, componentGeneratorWithDetail] = getComponentListGenerator();

// attach the component_parents entity to events from these plugins
enableLinkClickTracking({ context: [componentGenerator] });
enableFormTracking({ context: [componentGenerator] });
```

***

> **Note:** `componentGeneratorWithDetail` returns multiple entities and isn't directly compatible with the `context` arrays used by the link and form tracking plugins.

### Custom entities

There are two ways you can add custom entities to element tracking events:

- Plugin `context` option, applies to all rules
- Per-rule `context` option, applies only to events from that specific rule

**JavaScript (tag):**

```javascript
// Configure at plugin level to apply to all rules
snowplow('startElementTracking', {
  elements: [/* rules */],
  context: [/* entities attached to ALL events */]
});

// Configure at rule level to apply to a specific rule
snowplow('startElementTracking', {
  elements: {
    selector: '.promo-banner',
    name: 'promotion',
    context: [/* entities attached only to this rule's events */]
  }
});
```

**Browser (npm):**

```javascript
// Configure at plugin level to apply to all rules
startElementTracking({
  elements: [/* rules */],
  context: [/* entities attached to ALL events */]
});

// Configure at rule level to apply to a specific rule
startElementTracking({
  elements: {
    selector: '.promo-banner',
    name: 'promotion',
    context: [/* entities attached only to this rule's events */]
  }
});
```

***

You can configure static or dynamic entities:

- Use static entities when the same data should be attached to every event, e.g. A/B test variant

```javascript
context: [
    {
      schema: 'iglu:com.example/campaign/jsonschema/1-0-0',
      data: { campaign_id: 'summer_2025', variant: 'A' }
    }
  ]
```

- Use callbacks to generate dynamic entities when the data depends on the specific element that triggered the event

```javascript
context: [
  (element, rule) => ({
    schema: 'iglu:com.example/promotion/jsonschema/1-0-0',
    data: {
      promo_id: element.dataset.promoId,
      position: element.dataset.position,
      rule_name: rule.name
    }
  })
]
```

## Configure the plugin

As well as configuring the `element_statistics`, `element_content`, and `component_parents` entities, you can customize how element visibility tracking works using the options below.

The core options are explained in this table:

| Property   | Type     | Description                                                                                                                                                                                                                                                                                                                                  | Status        |
| ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `selector` | `string` | A CSS selector string that matches one or more elements on the page that should trigger events from this rule.                                                                                                                                                                                                                               | **Required**  |
| `name`     | `string` | A label to name this rule. Allows you to keep a stable name for events generated by this rule, even if the `selector` changes, so the data produced remains consistent. You can share a single `name` between many rules to have different configurations for different selectors. If not supplied, the `selector` value becomes the `name`. | _Recommended_ |
| `id`       | `string` | A specific identifier for this rule. Useful if you share a `name` between many rules and need to specifically remove individual rules within that group.                                                                                                                                                                                     |               |

You'll see `selector` and `name` in the examples on this page.

### Event frequency with `when`

The `when` option controls how often events fire. The default is `always`.

This example shows the options:

**JavaScript (tag):**

```javascript
// "always" - fires every time (e.g., every scroll in/out of view)
// Boolean shorthand - expose: true
snowplow('startElementTracking', {
  elements: {
    selector: '.ad-banner',
    name: 'ad_impression',
    expose: { when: 'always' }  // fires each time banner scrolls into view
  }
});

// "element" - fires once per matched element
snowplow('startElementTracking', {
  elements: {
    selector: '.product-card',
    name: 'product_impression',
    expose: { when: 'element' }  // fires once per card, even if user scrolls back
  }
});

// "pageview" - fires once per element, resets on new page view (useful for SPAs)
snowplow('startElementTracking', {
  elements: {
    selector: '.hero-section',
    name: 'hero_viewed',
    expose: { when: 'pageview' }  // resets when tracker fires next page_view event
  }
});

// "once" - fires exactly once for the entire rule, regardless of how many elements match
snowplow('startElementTracking', {
  elements: {
    selector: '.newsletter-form',
    name: 'newsletter_form_exists',
    expose: { when: 'once' }  // fires once even if multiple forms exist
  }
});

// "never": never track this event for this rule
// This is useful for defining components
// Boolean shorthand - expose: false
snowplow('startElementTracking', {
  elements: {
    selector: 'section',
    expose: { when: 'never' }  // never fires
  }
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

// "always" - fires every time (e.g., every scroll in/out of view)
// Boolean shorthand - expose: true
startElementTracking({
  elements: {
    selector: '.ad-banner',
    name: 'ad_impression',
    expose: { when: 'always' }  // fires each time banner scrolls into view
  }
});

// "element" - fires once per matched element
startElementTracking({
  elements: {
    selector: '.product-card',
    name: 'product_impression',
    expose: { when: 'element' }  // fires once per card, even if user scrolls back
  }
});

// "pageview" - fires once per element, resets on new page view (useful for SPAs)
startElementTracking({
  elements: {
    selector: '.hero-section',
    name: 'hero_viewed',
    expose: { when: 'pageview' }  // resets when tracker fires next page_view event
  }
});

// "once" - fires exactly once for the entire rule, regardless of how many elements match
startElementTracking({
  elements: {
    selector: '.newsletter-form',
    name: 'newsletter_form_exists',
    expose: { when: 'once' }  // fires once even if multiple forms exist
  }
});

// "never": never track this event for this rule
// This is useful for defining components
// Boolean shorthand - expose: false
startElementTracking({
  elements: {
    selector: 'section',
    expose: { when: 'never' }  // never fires
  }
});
```

***

If you're using `when: pageview`, ensure that the tracker is firing page view events appropriately for your needs, especially if it's a single page application (SPA).

The plugin assumes that you'll call `startElementTracking()` before `trackPageView()`. The first page view doesn't reset the element visibility state, because the plugin sets `ignoreNextPageView: true` by default internally.

If your site tracks page views before calling `startElementTracking()`, you can disable this behavior by passing `ignoreNextPageView: false` in the plugin options when adding it to the tracker.

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-element-tracking@latest/dist/index.umd.min.js",
  ["snowplowElementTracking", "SnowplowElementTrackingPlugin"],
  [{ ignoreNextPageView: false }]
);

snowplow('startElementTracking', { elements: [/* configuration */] });
```

**Browser (npm):**

First, add the plugin when initializing the tracker.

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { SnowplowElementTrackingPlugin, startElementTracking } from '@snowplow/browser-plugin-element-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SnowplowElementTrackingPlugin({ ignoreNextPageView: false }) ],
});

startElementTracking({ elements: [/* configuration */] });
```

***

### Visibility thresholds for `expose`

Control what counts as "visible" for `expose_element` events:

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: {
    selector: '.video-player',
    name: 'video_impression',
    expose: {
      when: 'element',
      minPercentage: 0.5,      // at least 50% of element must be visible
      minTimeMillis: 2000,     // must be visible for 2 seconds cumulative
      minSize: 10000,          // element must be at least 10,000px² (e.g., 100x100)
      boundaryPixels: 50       // adds 50px padding when calculating visibility
    }
  }
});

// boundaryPixels accepts different formats:
expose: {
  when: 'element',
  boundaryPixels: 20           // 20px all sides
  // or: [10, 20]              // 10px vertical, 20px horizontal
  // or: [10, 20, 30, 40]      // top, right, bottom, left
}
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: {
    selector: '.video-player',
    name: 'video_impression',
    expose: {
      when: 'element',
      minPercentage: 0.5,      // at least 50% of element must be visible
      minTimeMillis: 2000,     // must be visible for 2 seconds cumulative
      minSize: 10000,          // element must be at least 10,000px² (e.g., 100x100)
      boundaryPixels: 50       // adds 50px padding when calculating visibility
    }
  }
});

// boundaryPixels accepts different formats:
expose: {
  when: 'element',
  boundaryPixels: 20           // 20px all sides
  // or: [10, 20]              // 10px vertical, 20px horizontal
  // or: [10, 20, 30, 40]      // top, right, bottom, left
}
```

***

### Data selectors using `details`

The plugin uses data selectors when deciding if an element should trigger an event using `condition`, or when building the `element` entity's `attributes` property.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: {
    selector: '.product-card',
    name: 'product',
    expose: { when: 'element' },
    details: [
      // HTML attributes (from getAttribute)
      { attributes: ['id', 'data-category'] },

      // Element properties (may differ from HTML attributes)
      { properties: ['className', 'tagName'] },

      // Dataset values (data-* attributes, camelCase)
      // <div data-product-id="123" data-price="29.99">
      { dataset: ['productId', 'price'] },

      // Text content from child elements
      { child_text: {
          name: 'h3',           // text from first <h3>
          brand: '.brand-name'  // text from first .brand-name
      }},

      // Regex extraction from element's textContent
      { content: {
          sku: /SKU-(\d+)/     // captures first group
      }},

      // Include the selector that matched
      { selector: true },

      // Validate collected attributes - discards results if no match
      // Useful for filtering in `condition`
      { match: {
          category: 'electronics',                        // exact value match
          price: (val) => parseFloat(val) > 0             // function match
      }},

      // Custom callback function
      (element) => ({
        isOnSale: element.classList.contains('on-sale') ? 'true' : 'false',
        position: element.dataset.position
      })
    ]
  }
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: {
    selector: '.product-card',
    name: 'product',
    expose: { when: 'element' },
    details: [
      // HTML attributes (from getAttribute)
      { attributes: ['id', 'data-category'] },

      // Element properties (may differ from HTML attributes)
      { properties: ['className', 'tagName'] },

      // Dataset values (data-* attributes, camelCase)
      // <div data-product-id="123" data-price="29.99">
      { dataset: ['productId', 'price'] },

      // Text content from child elements
      { child_text: {
          name: 'h3',           // text from first <h3>
          brand: '.brand-name'  // text from first .brand-name
      }},

      // Regex extraction from element's textContent
      { content: {
          sku: /SKU-(\d+)/     // captures first group
      }},

      // Include the selector that matched
      { selector: true },

      // Validate collected attributes - discards results if no match
      // Useful for filtering in `condition`
      { match: {
          category: 'electronics',                        // exact value match
          price: (val) => parseFloat(val) > 0             // function match
      }},

      // Custom callback function
      (element) => ({
        isOnSale: element.classList.contains('on-sale') ? 'true' : 'false',
        position: element.dataset.position
      })
    ]
  }
});
```

***

### Conditional event firing with `condition`

Only fire events when elements match certain criteria. Use data selectors to define the conditions:

**JavaScript (tag):**

```javascript
// Example: only track visible notifications
snowplow('startElementTracking', {
  elements: {
    selector: '.notification',
    name: 'notification_shown',
    create: {
      when: 'element',
      condition: [
        // Only fire if notification has data-visible="true"
        { dataset: ['visible'] },
        { match: { visible: 'true' } }
      ]
    }
  }
});

// Example: only track products that are in stock
snowplow('startElementTracking', {
  elements: {
    selector: '.product-card',
    name: 'in_stock_product',
    expose: {
      when: 'element',
      condition: [
        { dataset: ['stockStatus'] },
        { match: { stockStatus: (val) => val !== 'out-of-stock' } }
      ]
    }
  }
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

// Example: only track visible notifications
startElementTracking({
  elements: {
    selector: '.notification',
    name: 'notification_shown',
    create: {
      when: 'element',
      condition: [
        // Only fire if notification has data-visible="true"
        { dataset: ['visible'] },
        { match: { visible: 'true' } }
      ]
    }
  }
});

// Example: only track products that are in stock
startElementTracking({
  elements: {
    selector: '.product-card',
    name: 'in_stock_product',
    expose: {
      when: 'element',
      condition: [
        { dataset: ['stockStatus'] },
        { match: { stockStatus: (val) => val !== 'out-of-stock' } }
      ]
    }
  }
});
```

***

### Shadow DOM tracking

If the elements you want to track exist within [shadow DOM](https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_shadow_DOM) trees, the plugin might not identify them automatically.

Use these settings to notify the plugin that it should descend into shadow hosts to identify elements to match the rule against.

By default, the plugin matches specified elements both outside and inside `shadowSelector` shadow hosts. Set `shadowOnly` to `true` to only match elements within those shadow hosts.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: {
    selector: 'button.submit',
    name: 'submit_button',
    shadowSelector: 'my-custom-form',  // CSS selector for elements that are shadow hosts containing the targeted elements
    shadowOnly: true,  // only match elements inside shadow DOM, not elsewhere
    expose: { when: 'element' }
  }
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: {
    selector: 'button.submit',
    name: 'submit_button',
    shadowSelector: 'my-custom-form',  // CSS selector for elements that are shadow hosts containing the targeted elements
    shadowOnly: true,  // only match elements inside shadow DOM, not elsewhere
    expose: { when: 'element' }
  }
});
```

***

### Send to specific trackers

If you have multiple trackers loaded on the same page, you can specify which trackers should receive events using the `tracker` option. Provide a list of tracker namespaces.

If omitted, events go to all trackers the plugin has been activated for.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: { selector: '.promo-banner' }
}, ['tracker1', 'tracker2']);
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: { selector: '.promo-banner' }
}, ['tracker1', 'tracker2']);
```

***

## Further examples

These examples are based on [a snapshot](https://web.archive.org/web/20250422013533/https://snowplow.io/) of the [Snowplow website](https://snowplow.io/).

### Content depth

The blog posts have longer-form content. Snowplow's page ping events track scroll depth by pixels, but those measurements become inconsistent between devices and page. To see how much content gets consumed, you can generate stats based on the paragraphs in the content. You can also get periodic stats based on the entire article in page pings.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: [
    {
      selector: ".blogs_blog-post-body_content",
      name: "blog content",
      expose: false,
      includeStats: ["page_ping"]
    },
    {
      selector: ".blogs_blog-post-body_content p",
      name: "blog paragraphs"
    }
  ]
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: [
    {
      selector: ".blogs_blog-post-body_content",
      name: "blog content",
      expose: false,
      includeStats: ["page_ping"]
    },
    {
      selector: ".blogs_blog-post-body_content p",
      name: "blog paragraphs"
    }
  ]
});
```

***

Because the expose event contains the `element_index` and `element_matches`, you can easily query the largest `element_index` by page view ID. The result tells you consumption statistics for individual views of each article. You can then summarize that metric to the content or category level, or converted to a percentage by comparing with `element_matches`.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
      "data": {
        "element_name": "blog paragraphs",
        "width": 800,
        "height": 48,
        "position_x": 320,
        "position_y": 533.25,
        "doc_position_x": 320,
        "doc_position_y": 1373,
        "element_index": 6,
        "element_matches": 24,
        "originating_page_view": "f390bec5-f63c-48af-b3ad-a03f0511af7f",
        "attributes": []
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
      "data": {
        "id": "f390bec5-f63c-48af-b3ad-a03f0511af7f"
      }
    }
  ]
}
```

The periodic page ping events also give you a summary of the total progress in the `max_y_depth_ratio`/`max_y_depth` values. With `y_depth_ratio` you can also see when users backtrack up the page.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/element_statistics/jsonschema/1-0-0",
  "data": {
    "element_name": "blog content",
    "element_index": 1,
    "element_matches": 1,
    "current_state": "unknown",
    "min_size": "800x3928",
    "current_size": "800x3928",
    "max_size": "800x3928",
    "y_depth_ratio": 0.20302953156822812,
    "max_y_depth_ratio": 0.4931262729124236,
    "max_y_depth": "1937/3928",
    "element_age_ms": 298379,
    "times_in_view": 0,
    "total_time_visible_ms": 0
  }
}
```

### Simple funnels

A newsletter sign-up form exists at the bottom of the page. Performance measurement becomes difficult because many visitors don't even see it. To test this you first need to know:

- When the form exists on a page
- When the form is actually seen
- When people actually interact with the form
- When the form is finally submitted

The form tracking plugin can only do the last parts, but the element tracker gives you the earlier steps. If you end up adding more forms in the future, you'll want to know which is which, so you can mark the footer as a component so you can split it out later.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: [
    {
      selector: ".hbspt-form",
      name: "newsletter signup",
      create: true,
    },
    {
      selector: "footer",
      component: true,
      expose: false
    }
  ]
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: [
    {
      selector: ".hbspt-form",
      name: "newsletter signup",
      create: true,
    },
    {
      selector: "footer",
      component: true,
      expose: false
    }
  ]
});
```

***

If you try this on a blog page, you actually get two `create_element` events. Blog posts have a second newsletter sign-up in a sidebar next to the content. Because only the second form is a member of the `footer` component, you can easily see which one you are trying to measure when you query the data later.

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
      "data": {
        "element_name": "newsletter signup",
        "width": 336,
        "height": 161,
        "position_x": 1232,
        "position_y": 238.88333129882812,
        "doc_position_x": 1232,
        "doc_position_y": 3677.883331298828,
        "element_index": 1,
        "element_matches": 2,
        "originating_page_view": "02e30714-a84a-42f8-8b07-df106d669db0",
        "attributes": []
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
      "data": {
        "id": "02e30714-a84a-42f8-8b07-df106d669db0"
      }
    }
  ]
}
```

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
      "data": {
        "element_name": "newsletter signup",
        "width": 560,
        "height": 137,
        "position_x": 320,
        "position_y": 1953.5,
        "doc_position_x": 320,
        "doc_position_y": 5392.5,
        "element_index": 2,
        "element_matches": 2,
        "originating_page_view": "02e30714-a84a-42f8-8b07-df106d669db0",
        "attributes": []
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/component_parents/jsonschema/1-0-0",
      "data": {
        "element_name": "newsletter signup",
        "component_list": [
          "footer"
        ]
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
      "data": {
        "element_name": "footer",
        "width": 1920,
        "height": 1071.5,
        "position_x": 0,
        "position_y": 1212,
        "doc_position_x": 0,
        "doc_position_y": 4651,
        "originating_page_view": "",
        "attributes": []
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
      "data": {
        "id": "02e30714-a84a-42f8-8b07-df106d669db0"
      }
    }
  ]
}
```

### Recommendations performance

The homepage contains a section for the "Latest Blogs from Snowplow." This could represent recommendations or some other form of personalization. If it did, one might want to optimize it. Link tracking could tell you when a recommendation worked and a visitor clicked it, but how would identify the recommendation not encouraging clicks? If you track when the widget becomes visible and include the items that got recommended, you could correlate that with the clicks to measure performance. For fairer measurement of visibility, you can configure that visibility only counts if at least 50% is in view, and it has to be on screen for at least 1.5 seconds. You'll also collect the post title and author information.

**JavaScript (tag):**

```javascript
snowplow('startElementTracking', {
  elements: [
    {
      selector: ".blog_list-header_list-wrapper",
      name: "recommended_posts",
      create: true,
      expose: { when: "element", minTimeMillis: 1500, minPercentage: 0.5 },
      contents: [
        {
          selector: ".collection-item",
          name: "recommended_item",
          details: { child_text: { title: "h3", author: ".blog_list-header_author-text > p" } }
        }
      ]
    }
  ]
});
```

**Browser (npm):**

```javascript
import { startElementTracking } from '@snowplow/browser-plugin-element-tracking';

startElementTracking({
  elements: [
    {
      selector: ".blog_list-header_list-wrapper",
      name: "recommended_posts",
      create: true,
      expose: { when: "element", minTimeMillis: 1500, minPercentage: 0.5 },
      contents: [
        {
          selector: ".collection-item",
          name: "recommended_item",
          details: { child_text: { title: "h3", author: ".blog_list-header_author-text > p" } }
        }
      ]
    }
  ]
});
```

***

Scrolling down to see the items and you see the items that get served to the visitor:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element/jsonschema/1-0-0",
      "data": {
        "element_name": "recommended_posts",
        "width": 1280,
        "height": 680.7666625976562,
        "position_x": 320,
        "position_y": 437.70001220703125,
        "doc_position_x": 320,
        "doc_position_y": 6261.066711425781,
        "element_index": 1,
        "element_matches": 1,
        "originating_page_view": "034db1d6-1d60-42ca-8fe1-9aafc0442a22",
        "attributes": []
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0",
      "data": {
        "element_name": "recommended_item",
        "parent_name": "recommended_posts",
        "parent_position": 1,
        "position": 1,
        "attributes": [
          {
            "source": "child_text",
            "attribute": "title",
            "value": "Data Pipeline Architecture Patterns for AI: Choosing the Right Approach"
          },
          {
            "source": "child_text",
            "attribute": "author",
            "value": "Matus Tomlein"
          }
        ]
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0",
      "data": {
        "element_name": "recommended_item",
        "parent_name": "recommended_posts",
        "parent_position": 1,
        "position": 2,
        "attributes": [
          {
            "source": "child_text",
            "attribute": "title",
            "value": "Data Pipeline Architecture For AI: Why Traditional Approaches Fall Short"
          },
          {
            "source": "child_text",
            "attribute": "author",
            "value": "Matus Tomlein"
          }
        ]
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/element_content/jsonschema/1-0-0",
      "data": {
        "element_name": "recommended_item",
        "parent_name": "recommended_posts",
        "parent_position": 1,
        "position": 3,
        "attributes": [
          {
            "source": "child_text",
            "attribute": "title",
            "value": "Agentic AI Applications: How They Will Turn the Web Upside Down"
          },
          {
            "source": "child_text",
            "attribute": "author",
            "value": "Yali\tSassoon"
          }
        ]
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0",
      "data": {
        "id": "034db1d6-1d60-42ca-8fe1-9aafc0442a22"
      }
    }
  ]
}
```

---

# Track errors on web
> Track handled and unhandled JavaScript exceptions with manual error tracking and automatic error tracking capabilities.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/errors/

The Errors tracker plugin provides two ways of tracking exceptions: manual tracking of handled exceptions using `trackError` and automatic tracking of unhandled exceptions using `enableErrorTracking`.

Error events can be **manually tracked** and/or **automatically tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                        |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                  |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-error-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-error-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-error-tracking@latest/dist/index.umd.min.js",
  ["snowplowErrorTracking", "ErrorTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-error-tracking`
- `yarn add @snowplow/browser-plugin-error-tracking`
- `pnpm add @snowplow/browser-plugin-error-tracking`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { ErrorTrackingPlugin, enableErrorTracking } from '@snowplow/browser-plugin-error-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ ErrorTrackingPlugin() ],
});

enableErrorTracking();
```

***

## Manual error tracking

Use the `trackError` method to track handled exceptions (application errors) in your JS code. This is its signature:

**JavaScript (tag):**

```javascript
snowplow('trackError', {
  /** The error message */
  message: string;
  /** The filename where the error occurred */
  filename?: string;
  /** The line number which the error occurred on */
  lineno?: number;
  /** The column number which the error occurred on */
  colno?: number;
  /** The error object */
  error?: Error;
});
```

**Browser (npm):**

```javascript
trackError({
  /** The error message */
  message: string;
  /** The filename where the error occurred */
  filename?: string;
  /** The line number which the error occurred on */
  lineno?: number;
  /** The column number which the error occurred on */
  colno?: number;
  /** The error object */
  error?: Error;
});
```

***

| **Name**   | **Required?** | **Description**                     | **Type**   |
| ---------- | ------------- | ----------------------------------- | ---------- |
| `message`  | Yes           | Error message                       | string     |
| `filename` | No            | Filename or URL                     | string     |
| `lineno`   | No            | Line number of problem code chunk   | number     |
| `colno`    | No            | Column number of problem code chunk | number     |
| `error`    | No            | JS `ErrorEvent`                     | ErrorEvent |

Of these arguments, only `message` is required. Signature of this method defined to match `window.onerror` callback in modern browsers.

**JavaScript (tag):**

```javascript
try {
  var user = getUser()
} catch(e) {
  snowplow('trackError', {
    message: 'Cannot get user object',
    filename: 'shop.js',
    error: e
  });
}
```

**Browser (npm):**

```javascript
try {
  var user = getUser()
} catch(e) {
  trackError({
    message: 'Cannot get user object',
    filename: 'shop.js',
    error: e
  });
}
```

***

Using `trackError` it's assumed that developer knows where errors could happen, which is not often the case. Therefor it's recommended to use `enableErrorTracking` as it allows you to discover errors that weren't expected.

## Automatic error tracking

Use the `enableErrorTracking` method to track unhandled exceptions (application errors) in your JS code. This is its signature:

**JavaScript (tag):**

```javascript
snowplow('enableErrorTracking', {
  /** A callback which allows on certain errors to be tracked */
  filter?: (error: ErrorEvent) => boolean;
  /** A callback to dynamically add extra context based on the error */
  contextAdder?: (error: ErrorEvent) => Array<SelfDescribingJson>;
  /** Context to be added to every error */
  context?: Array<SelfDescribingJson>;
}
```

**Browser (npm):**

```javascript
enableErrorTracking({
  /** A callback which allows on certain errors to be tracked */
  filter?: (error: ErrorEvent) => boolean;
  /** A callback to dynamically add extra context based on the error */
  contextAdder?: (error: ErrorEvent) => Array<SelfDescribingJson>;
  /** Context to be added to every error */
  context?: Array<SelfDescribingJson>;
});
```

***

| **Name**       | **Required?** | **Description**                 | **Type**                                    |
| -------------- | ------------- | ------------------------------- | ------------------------------------------- |
| `filter`       | No            | Predicate to filter exceptions  | `(ErrorEvent) => Boolean`                   |
| `contextAdder` | No            | Function to get dynamic context | `(ErrorEvent) => Array<SelfDescribingJson>` |
| context        | No            | Additional custom context       | `Array<SelfDescribingJson>`                 |

Unlike `trackError` you need enable error tracking only once:

**JavaScript (tag):**

```javascript
snowplow('enableErrorTracking')
```

**Browser (npm):**

```javascript
enableErrorTracking();
```

***

Application error events are implemented as Snowplow self-describing events. [Here](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.snowplow/application_error/jsonschema/1-0-1) is the schema for an `application_error` event.

---

# Integrate with your event specifications on web
> Use the event specifications plugin to automatically attach event specification entities from your tracking plan to matching browser events.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/event-specifications/

This plugin allows you to integrate with [Media Web](/docs/event-studio/tracking-plans/templates/#media-web) event specifications. The plugin will add an event specification entity to the matching [Snowplow media](/docs/events/ootb-data/media-events/) events.

Retrieve the configuration directly from your [tracking plan](https://docs.snowplow.io/docs/fundamentals/tracking-plans/) in [Snowplow Console](https://console.snowplowanalytics.com).

> **Note:** The plugin is available since version 3.23 of the tracker. It's only available for tracking plans created using the [Media Web template](/docs/event-studio/tracking-plans/templates/#media-web).

The event specification entity is **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                              |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                        |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-event-specifications@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-event-specifications@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-event-specifications@latest/dist/index.umd.min.js',
    ['eventSpecifications', 'EventSpecificationsPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-event-specifications`
- `yarn add @snowplow/browser-plugin-event-specifications`
- `pnpm add @snowplow/browser-plugin-event-specifications`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { EventSpecificationsPlugin } from '@snowplow/browser-plugin-event-specifications';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ EventSpecificationsPlugin(/* plugin configuration */) ],
});
```

***

## Configuration

You can retrieve the configuration for your event specifications directly from your tracking plan after clicking on the `Implement tracking` button.

![implement tracking button](/assets/images/implement_tracking-237d544d543211d13699e36aac03fc1c.png)

Configure the plugin by mapping each tracked event to the event specification ID from your tracking plan. For example:

**JavaScript (tag):**

```javascript
// Initialize tracker
window.snowplow('newTracker', 'sp1', '{{collector_url}}', {
  appId: 'my-app-id'
});

// Add the Media plugin
window.snowplow(
  'addPlugin',
  'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js',
  ['snowplowMedia', 'SnowplowMediaPlugin']
);

// Add the Event Specifications plugin with configuration
window.snowplow(
  'addPlugin',
  'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-event-specifications@latest/dist/index.umd.min.js',
  ['eventSpecifications', 'EventSpecificationsPlugin'],
  [
    {
      SnowplowMediaPlugin: {
        // Map event names to your event specification IDs from Console
        "ad_break_end_event": "abb057a9-eb05-41b8-8d13-0a020f5f9960",
        "ad_break_start_event": "896fc117-aad9-4ef1-ab52-13fcf9156a08",
        "ad_click_event": "df27c1dd-b7e6-4d4a-8ba1-613d859594c4",
        "ad_complete_event": "a382f36b-39ed-46b4-9e6f-ac9bd1d65360",
        "ad_pause_event": "6bd62180-37ab-4a9c-9aa4-580aa39d7888",
        "ad_quartile_event": "7d946906-80eb-4ca0-bf7d-4a0f04ae3598",
        "ad_resume_event": "d5ae264a-3983-478b-b9d5-bcf46c66cab1",
        "ad_skip_event": "5f79e53b-9318-4644-b4e8-8bf7804c244b",
        "ad_start_event": "442a8e75-4884-434e-8e1d-d80bc35c4157",
        "buffer_end_event": "8951ce07-b497-45b0-81c3-49962d36fa6a",
        "buffer_start_event": "c53f4ea7-e7c3-44d9-97a8-1edf5a61d898",
        "error_event": "594bd013-8e6b-4ec4-9828-e5e609b4297c",
        "fullscreen_change_event": "3780c3e5-17ed-4e39-b22c-c0568c486bf3",
        "ping_event": "a4870ad5-e028-42ae-bfca-603a3d6837f1",
        "pause_event": "bf90af15-840d-4a76-a7f0-ccc8865a9c5c",
        "percent_progress_event": "6684eea3-82e6-4c2e-98db-ab0be61fdf0d",
        "picture_in_picture_change_event": "2e8be82e-11fb-4aa3-a5a2-7f49efc29abb",
        "end_event": "aaac78f1-8ee4-42a6-8e3c-46f660c32709",
        "play_event": "1094455c-4e99-4e1f-8445-f4fb12b4eccc",
        "quality_change_event": "bdf91319-c5f0-476f-922f-9215b76186af",
        "ready_event": "c1f9f850-dc68-47d8-9fd1-0db10328858c",
        "seek_end_event": "d748d09f-a361-4620-8651-f883b1502a23",
        "seek_start_event": "5fb03fae-3f0f-4538-908b-a55a6f7e69cb",
        "volume_change_event": "972836c4-73b8-45c3-abcf-22e3bd7eae6c"
      }
    }
  ]
);
```

**Browser (npm):**

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { SnowplowMediaPlugin, enableMediaTracking } from '@snowplow/browser-plugin-media';
import { EventSpecificationsPlugin } from '@snowplow/browser-plugin-event-specifications';

// Initialize tracker with both plugins
newTracker('sp1', '{{collector_url}}', {
  appId: 'my-app-id',
  plugins: [
    SnowplowMediaPlugin(),
    EventSpecificationsPlugin({
      SnowplowMediaPlugin: {
        // Map event names to your event specification IDs from Console
        "ad_break_end_event": "abb057a9-eb05-41b8-8d13-0a020f5f9960",
        "ad_break_start_event": "896fc117-aad9-4ef1-ab52-13fcf9156a08",
        "ad_click_event": "df27c1dd-b7e6-4d4a-8ba1-613d859594c4",
        "ad_complete_event": "a382f36b-39ed-46b4-9e6f-ac9bd1d65360",
        "ad_pause_event": "6bd62180-37ab-4a9c-9aa4-580aa39d7888",
        "ad_quartile_event": "7d946906-80eb-4ca0-bf7d-4a0f04ae3598",
        "ad_resume_event": "d5ae264a-3983-478b-b9d5-bcf46c66cab1",
        "ad_skip_event": "5f79e53b-9318-4644-b4e8-8bf7804c244b",
        "ad_start_event": "442a8e75-4884-434e-8e1d-d80bc35c4157",
        "buffer_end_event": "8951ce07-b497-45b0-81c3-49962d36fa6a",
        "buffer_start_event": "c53f4ea7-e7c3-44d9-97a8-1edf5a61d898",
        "error_event": "594bd013-8e6b-4ec4-9828-e5e609b4297c",
        "fullscreen_change_event": "3780c3e5-17ed-4e39-b22c-c0568c486bf3",
        "ping_event": "a4870ad5-e028-42ae-bfca-603a3d6837f1",
        "pause_event": "bf90af15-840d-4a76-a7f0-ccc8865a9c5c",
        "percent_progress_event": "6684eea3-82e6-4c2e-98db-ab0be61fdf0d",
        "picture_in_picture_change_event": "2e8be82e-11fb-4aa3-a5a2-7f49efc29abb",
        "end_event": "aaac78f1-8ee4-42a6-8e3c-46f660c32709",
        "play_event": "1094455c-4e99-4e1f-8445-f4fb12b4eccc",
        "quality_change_event": "bdf91319-c5f0-476f-922f-9215b76186af",
        "ready_event": "c1f9f850-dc68-47d8-9fd1-0db10328858c",
        "seek_end_event": "d748d09f-a361-4620-8651-f883b1502a23",
        "seek_start_event": "5fb03fae-3f0f-4538-908b-a55a6f7e69cb",
        "volume_change_event": "972836c4-73b8-45c3-abcf-22e3bd7eae6c"
      }
    })
  ]
});
```

***

## Event specification entity

When an event is tracked that matches one of the configured event names, the plugin will automatically add an event specification entity to it.

### `event_specification`

**Type:** Entity

Entity schema for referencing an event specification

**Schema:** `iglu:com.snowplowanalytics.snowplow/event_specification/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "abb057a9-eb05-41b8-8d13-0a020f5f9960"
}
```

**Properties:**

| Property      | Description                                                                  |
| ------------- | ---------------------------------------------------------------------------- |
| `id` _string_ | _Required._ Identifier for the event specification that the event adheres to |

---

# Track Kantar Focal Meter events on web
> Integrate with Kantar Focal Meter router meters to measure content audience by sending domain user IDs to Focal Meter endpoints.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/focalmeter/

This plugin provides integration with [Focal Meter by Kantar](https://www.virtualmeter.co.uk/focalmeter). Focal Meter is a box that connects directly to the broadband router and collects viewing information for the devices on your network.

This integration enables measuring the audience of content through the Focal Meter router meter. The plugin has the ability to send the [domain user ID](/docs/fundamentals/canonical-event/#user-fields) to a [Kantar Focal Meter](https://www.virtualmeter.co.uk/focalmeter) endpoint. A request is made when the first event with a new user ID is tracked.

The plugin inspects the domain user ID property in tracked events. Whenever it changes from the previously recorded value, it makes an HTTP GET request to the `kantarEndpoint` URL with the ID as a query parameter.

Optionally, the tracker may store the last published domain user ID value in local storage in order to prevent it from making the same request on the next page load. If local storage is not used, the request is made on each page load.

> **Note:** The plugin is available since version 3.16 of the tracker.

The Focal Meter integration is **automatic** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                    |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)              |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-focalmeter@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-focalmeter@latest/dist/index.umd.min.js) (latest)               |

**Browser (npm):**

- `npm install @snowplow/browser-plugin-focalmeter`
- `yarn add @snowplow/browser-plugin-focalmeter`
- `pnpm add @snowplow/browser-plugin-focalmeter`

***

## Enable integration

**JavaScript (tag):**

To integrate with the Kantar FocalMeter, use the snippet below after [setting up your tracker](/docs/sources/web-trackers/quick-start-guide/):

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-focalmeter@latest/dist/index.umd.min.js',
    ['snowplowFocalMeter', 'FocalMeterPlugin']
);

window.snowplow('enableFocalMeterIntegration', {
    kantarEndpoint: '{{kantar_url}}',
    useLocalStorage: false // optional, defaults to false
});
```

**Browser (npm):**

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { FocalMeterPlugin, enableFocalMeterIntegration } from '@snowplow/browser-plugin-focalmeter';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ FocalMeterPlugin() ],
});

enableFocalMeterIntegration({
  kantarEndpoint: '{{kantar_url}}',
  useLocalStorage: false // optional, defaults to false
});
```

***

The `enableFocalMeterIntegration` function has the following arguments:

| Parameter         | Type                         | Default | Description                                                                                                                                                 | Required |
| ----------------- | ---------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `kantarEndpoint`  | `string`                     | -       | URL of the Kantar endpoint to send the requests to (including protocol)                                                                                     | Yes      |
| `processUserId`   | `(userId: string) => string` | -       | Callback to process user ID before sending it in a request. This may be used to apply hashing to the value.                                                 | No       |
| `useLocalStorage` | `boolean`                    | `false` | Whether to store information about the last submitted user ID in local storage to prevent sending it again on next load (defaults not to use local storage) | No       |

If you choose to storage the last submitted user ID in local storage, the plugin will use the key `sp-fclmtr-{trackerId}`. The `trackerId` is your tracker namespace.

### Processing the user ID

By default, the plugin sends the domain user ID as a GET parameter in requests to Kantar without modifying it. In case you want to apply some transformation on the value, such as hashing, you can provide the `processUserId` callback in the `enableFocalMeterIntegration` call:

**JavaScript (tag):**

```javascript
window.snowplow('enableFocalMeterIntegration', {
    kantarEndpoint: "https://kantar.example.com",
    processUserId: (userId) => md5(userId).toString(), // apply the custom hashing here
});
```

**Browser (npm):**

```javascript
import md5 from 'crypto-js/md5';
enableFocalMeterIntegration({
    kantarEndpoint: "https://kantar.example.com",
    processUserId: (userId) => md5(userId).toString(), // apply the custom hashing here
});
```

***

### Configure multiple trackers

If you have multiple trackers loaded on the same page, you can enable the Focal Meter integration for each of them by specifying the tracker namespace as the third parameter to the `enableFocalMeterIntegration` function:

**JavaScript (tag):**

```javascript
window.snowplow(
    'enableFocalMeterIntegration',
    { kantarEndpoint: 'https://kantar.example.com' },
    ['sp1', 'sp2']  // Only these tracker namespaces will send to Kantar
);
```

**Browser (npm):**

```javascript
enableFocalMeterIntegration(
    { kantarEndpoint: 'https://kantar.example.com' },
    ['sp1', 'sp2']  // Only these tracker namespaces will send to Kantar
);
```

***

## Request format

The tracker will send requests with this format:

```text
GET https://your-kantar-endpoint.com?vendor=snowplow&cs_fpid=d5c4f9a2-3b7e-4d1f-8c6a-9e2b5f0a3c8d&c12=not_set
```

Where:

- `vendor` is always `snowplow`
- `cs_fpid` is the domain user ID, or the processed version if a `processUserId` callback is provided
- `c12` is always `not_set`

---

# Track form interactions on web
> Automatically track form changes, submissions, and focus events with configurable allowlists, denylists, and transform functions for field values.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/form-tracking/

Snowplow form tracking creates three event types: `change_form`, `submit_form` and `focus_form`. Using the `enableFormTracking` method adds event listeners to the document listening for events from form elements and their interactive fields (that is, all `input`, `textarea`, and `select` elements).

> **Note:** Events on password fields will not be tracked.

Form events are **automatically tracked** once configured.

## Installation

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                       |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                 |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-form-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-form-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

**Browser (npm):**

- `npm install @snowplow/browser-plugin-form-tracking`
- `yarn add @snowplow/browser-plugin-form-tracking`
- `pnpm add @snowplow/browser-plugin-form-tracking`

***

## Toggle form tracking

Start tracking form events by enabling the plugin:

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-form-tracking@latest/dist/index.umd.min.js",
  ["snowplowFormTracking", "FormTrackingPlugin"]
);

snowplow('enableFormTracking');
```

**Browser (npm):**

Initialize your tracker with the plugin.

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { FormTrackingPlugin, enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ FormTrackingPlugin() ],
});

enableFormTracking();
```

***

To stop form tracking, call `disableFormTracking`:

**JavaScript (tag):**

```javascript
snowplow('disableFormTracking');
```

**Browser (npm):**

```javascript
import { FormTrackingPlugin, disableFormTracking } from '@snowplow/browser-plugin-form-tracking';

disableFormTracking();
```

***

## Events

By default, all three event types are tracked. However, it is possible to subscribe only to specific event types using the `options.events` option when enabling form tracking:

**JavaScript (tag):**

```javascript
// subscribing to specific event types
snowplow('enableFormTracking', {
    options: {
        events: ['submit_form', 'focus_form', 'change_form']
    },
});
```

**Browser (npm):**

```javascript
// subscribing to specific event types
enableFormTracking({
    options: {
        events: ['submit_form', 'focus_form', 'change_form']
    },
});
```

***

Check out the [form tracking overview](/docs/events/ootb-data/page-elements/#form-interactions) page to see the schema details.

### Change form

When a user changes the value of a `textarea`, `input`, or `select` element inside a form, a [`change_form`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/change_form/jsonschema/1-0-0) event will be fired. It will capture the name, type, and new value of the element, and the id of the parent form.

### Submit form

When a user submits a form, a [`submit_form`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/submit_form/jsonschema/1-0-0) event will be fired. It will capture the id and classes of the form and the name, type, and value of all `textarea`, `input`, and `select` elements inside the form.

Note that this will only work if the original form submission event is actually fired. If you prevent it from firing, for example by using a jQuery event handler which returns `false` to handle clicks on the form's submission button, the Snowplow `submit_form` event will not be fired.

### Focus form

When a user focuses on a form element, a [`focus_form`](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/focus_form/jsonschema/1-0-0) event will be fired. It will capture the id and classes of the form and the name, type, and value of the `textarea`, `input`, or `select` element inside the form that received focus.

## Configuration

It may be that you do not want to track every field in a form, or every form on a page. You can customize form tracking by passing a configuration argument to the `enableFormTracking` method. This argument should be an object with two elements named "forms" and "fields". The "forms" element determines which forms will be tracked; the "fields" element determines which fields inside the tracked forms will be tracked. As with link click tracking, there are three ways to configure each field: a denylist, an allowlist, or a filter function. You do not have to use the same method for both fields.

**Denylists**

This is an array of strings used to prevent certain elements from being tracked. Any form with a CSS class in the array will be ignored. Any field whose name property is in the array will be ignored. All other elements will be tracked.

**Allowlists**

This is an array of strings used to turn on tracking. Any form with a CSS class in the array will be tracked. Any field in a tracked form whose name property is in the array will be tracked. All other elements will be ignored.

**Filter functions**

This is a function used to determine which elements are tracked. The element is passed as the argument to the function and is tracked if and only if the value returned by the function is truthy.

**Event phase**

From v4 onwards, this plugin uses [capture-phase](https://developer.mozilla.org/en-US/docs/Web/API/Event/eventPhase#value) event listeners to detect form events. The capture phase is the earliest phase of event handlers, so events might be tracked before other code executes (e.g. form validation). If your filter or transform functions are relying on other event handlers to have executed to function correctly, they may not behave as expected when using capture-phase event handlers.

From v4.6.8 onwards, the plugin supports a `useCapture` option, which you can set to `false` (default is `true`) to revert to the v3 behavior of using bubble-phase event handlers. This allows other event handlers time to execute before the event is detected and your filter/transform functions are executed.

When using the bubble phase, other event handlers may [cancel the event's propagation](https://developer.mozilla.org/en-US/docs/Web/API/Event/stopPropagation) and the plugin will not receive the event and nothing will be tracked. This may be desirable if you want to wait for the form to validate before tracking a "form\_submit" event, for example. Native HTML form validation automatically prevents the "submit" event firing until the form is valid, so only validation code that doesn't integrate with native APIs should require explicitly using the bubble phase.

The `focus` event for form fields [does not bubble](https://developer.mozilla.org/en-US/docs/Web/API/Element/focus_event), so this setting is ignored for `form_focus` tracking, which will always use capture-phase event listeners; only "change" and "submit" handlers will use the bubble phase when setting `useCapture: false`.

### Transform functions

This is a function used to transform data in each form field. The value and element are passed as arguments to the function and the tracked value is replaced by the value returned.

The transform function receives three arguments:

1. The value of the element.
2. Either the HTML element (for `change_form` and `focus_form` events) or an instance of `ElementData` (for `submit_form` events).
3. The HTML element (in all form tracking events).

The function signature is:

```typescript
type transformFn = (
  elementValue: string | null,
  elementInfo: ElementData | TrackedHTMLElement,
  elt: TrackedHTMLElement
) => string | null;
```

This means that you can specify a transform function that applies the exact same logic to all `submit_form`, `change_form` and `focus_form` events independent of the element's attributes the logic may depend on. For example:

**JavaScript (tag):**

```javascript
function redactPII(eltValue, _, elt) {
  if (elt.id === 'pid') {
    return 'redacted';
  }
  return eltValue;
}

snowplow('enableFormTracking', {
  options: {
    fields: { transform: redactPII },
  },
});
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

function redactPII(eltValue, _, elt) {
  if (elt.id === 'pid') {
    return 'redacted';
  }
  return eltValue;
}

enableFormTracking({
  options: {
    fields: { transform: redactPII },
  },
});
```

***

### Examples

To track every form element and every field except those fields named "password":

**JavaScript (tag):**

```javascript
var opts = {
  forms: {
    denylist: []
  },
  fields: {
    denylist: ['password']
  }
};

snowplow('enableFormTracking', { options: opts });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

var options = {
  forms: {
    denylist: []
  },
  fields: {
    denylist: ['password']
  }
};

enableFormTracking({ options });
```

***

To track only the forms with CSS class "tracked", and only those fields whose ID is not "private":

**JavaScript (tag):**

```javascript
var opts = {
  forms: {
    allowlist: ["tracked"]
  },
  fields: {
    filter: function (elt) {
      return elt.id !== "private";
    }
  }
};

snowplow('enableFormTracking', { options: opts });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

var opts = {
  forms: {
    allowlist: ["tracked"]
  },
  fields: {
    filter: function (elt) {
      return elt.id !== "private";
    }
  }
};

enableFormTracking({ options: opts });
```

***

To transform the form fields with an MD5 hashing function:

**JavaScript (tag):**

```javascript
function hashMD5(value, _, elt) {
  // can use elt to make transformation decisions
  return MD5(value);
}

var opts = {
  forms: {
    allowlist: ["tracked"]
  },
  fields: {
    filter: function (elt) {
      return elt.id !== "private";
    },
    transform: hashMD5
  }
};

snowplow('enableFormTracking', { options: opts });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

function hashMD5(value, _, elt) {
  // can use elt to make transformation decisions
  return MD5(value);
}

var options = {
  forms: {
    allowlist: ["tracked"]
  },
  fields: {
    filter: function (elt) {
      return elt.id !== "private";
    },
    transform: hashMD5
  }
};

enableFormTracking({ options });
```

***

To use the bubble-phase event listeners:

**JavaScript (tag):**

```javascript
snowplow('enableFormTracking', { options: { useCapture: false } });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

enableFormTracking({ options: { useCapture: false } });
```

***

## Tracking forms embedded inside iframes

The options for tracking forms inside of iframes are limited – browsers block access to contents of iframes that are from different domains than the parent page. We are not able to provide a solution to track events using trackers initialized on the parent page in such cases.

It is possible to track events from forms embedded in iframes loaded from the same domain as the parent page or iframes created using JavaScript on the parent page (e.g. HubSpot forms).

In case you are able to access form elements inside an iframe, you can pass them in the `options.forms` argument when calling `enableFormTracking` on the parent page. This will enable form tracking for the specific form elements. The feature may also be used for forms not embedded in iframes, but it's most useful in this particular case.

The following example shows how to identify the form elements inside an iframe and pass them to the `enableFormTracking` function:

**JavaScript (tag):**

```javascript
let iframe = document.getElementById('form_iframe'); // find the element for the iframe
let forms = iframe.contentWindow.document.getElementsByTagName('form'); // find form elements inside the iframe
snowplow('enableFormTracking', {
    options: {
        forms: forms // pass the embedded forms when enabling form tracking
    },
});
```

**Browser (npm):**

```javascript
let iframe = document.getElementById('form_iframe'); // find the element for the iframe
let forms = iframe.contentWindow.document.getElementsByTagName('form'); // find form elements inside the iframe
enableFormTracking({
    options: {
        forms: forms // pass the embedded forms when enabling form tracking
    },
});
```

***

Alternatively, you can specify the iframe's `document` as a `target` directly; this will enable form tracking for all forms within the iframe's document:

**JavaScript (tag):**

```javascript
let iframe = document.getElementById('form_iframe'); // find the element for the iframe
let formDoc = iframe.contentWindow.document; // find iframe document that contains forms
snowplow('enableFormTracking', {
    options: {
        targets: [document, formDoc] // pass the embedded document when enabling form tracking
    },
});
```

**Browser (npm):**

```javascript
let iframe = document.getElementById('form_iframe'); // find the element for the iframe
let formDoc = iframe.contentWindow.document; // find iframe document that contains forms
enableFormTracking({
    options: {
        targets: [document, formDoc] // pass the embedded document when enabling form tracking
    },
});
```

***

`targets` can also be used to only track subsets of a document by passing a parent element directly.

## Tracking forms from inside shadow trees

Forms created within [shadow trees](https://developer.mozilla.org/en-US/docs/Glossary/Shadow_tree) (e.g. within custom [Web Components](https://developer.mozilla.org/en-US/docs/Web/API/Web_components)) can only be tracked once the user first focuses a field.

The plugin relies on composed events to detect the form interactions at the document level. Only `focus` events are considered composed, `change` and `submit` events are not composed and so are not automatically detected by the plugin.

When the user focuses a field in a form that is detected as being inside a shadow tree, the event listeners are added directly to the `form` element within the shadow tree in addition to the document-level event listeners in order to track future `change` and `submit` events correctly. If the form has no interactive field elements to first trigger the `focus` event, any `change` or `submit` events that fire will not be tracked.

If the shadow root is attached in "closed" mode, no events will be tracked for elements in that shadow tree, only "open" mode is supported.

## Custom context entities

Context entities can be sent with all form tracking events by supplying them in an array in the `context` argument.

**JavaScript (tag):**

```javascript
snowplow('enableFormTracking', { options: {}, context: [] });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

enableFormTracking({ options: {}, context: [] });
```

***

These context entities can be dynamic, i.e. they can be traditional self-describing JSON objects, or callbacks that generate valid self-describing JSON objects.

For form change events, context generators are passed `(elt, type, value)`, and form submission events are passed `(elt, innerElements)`.

A dynamic context could therefore look something like this for form change events:

**JavaScript (tag):**

```javascript
let dynamicContext = function (elt, type, value) {
  // perform operations here to construct the context entity
  return context;
};

snowplow('enableFormTracking', { options: {}, context: [dynamicContext] });
```

**Browser (npm):**

```javascript
import { enableFormTracking } from '@snowplow/browser-plugin-form-tracking';

var dynamicContext = function (elt, type, value) {
  // perform operations here to construct the context entity
  return context;
};

enableFormTracking({ options: {}, context: [dynamicContext] });
```

***

---

# Track Google Analytics cookies with the web trackers
> Automatically capture Google Analytics cookie values including GA4 and Universal Analytics cookies as context entities on every event.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/ga-cookies/

If this plugin is used, the tracker will look for Google Analytics cookies (GA4/Universal Analytics and "classic" GA; specifically the `_ga` cookie and older `__utma`, `__utmb`, `__utmc`, `__utmv`, `__utmz`) and combine their values into event context entities that get sent with every event.

GA cookies information is **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                    |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)              |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-ga-cookies@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-ga-cookies@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

**Browser (npm):**

- `npm install @snowplow/browser-plugin-ga-cookies`
- `yarn add @snowplow/browser-plugin-ga-cookies`
- `pnpm add @snowplow/browser-plugin-ga-cookies`

***

## Initialization

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-ga-cookies@latest/dist/index.umd.min.js",
  ["snowplowGaCookies", "GaCookiesPlugin"],
  [pluginOptions] // note: for sp.js, pluginOptions can also be specified when calling `newTracker`; e.g. `contexts: { gaCookies: { ua: true, ga4: false } }`
);
```

**Browser (npm):**

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { GaCookiesPlugin } from '@snowplow/browser-plugin-ga-cookies';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ GaCookiesPlugin(pluginOptions) ],
});
```

***

The `pluginOptions` parameter allows to configure the plugin. Its type is:

```javascript
interface GACookiesPluginOptions {
  ua?: boolean;
  ga4?: boolean;
  ga4MeasurementId?: string | string[];
  cookiePrefix?: string | string[];
}
```

| Name             | Default | Description                                                                                                                                                                                                                                                                                                                                                     |
| ---------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ua               | `false` | Send Universal Analytics specific cookie values.                                                                                                                                                                                                                                                                                                                |
| ga4              | `true`  | Send Google Analytics 4 specific cookie values.                                                                                                                                                                                                                                                                                                                 |
| ga4MeasurementId | `""`    | Measurement id(s) to search the Google Analytics 4 session cookie. Can be a single measurement id as a string or an array of measurement id strings. The cookie has the form of `<cookie_prefix>_ga_<container-id>` where `<container-id>` is the data stream container id and `<cookie_prefix>` is the optional `cookie_prefix` option of the gtag.js tracker. |
| cookiePrefix     | `[]`    | Cookie prefix set on the Google Analytics 4 cookies using the `cookie_prefix` option of the gtag.js tracker.                                                                                                                                                                                                                                                    |

## Context entities

Adding this plugin will automatically capture the following entities:

1. For GA4 cookies: `iglu:com.google.ga4/cookies/jsonschema/1-0-0` (default)

   ```json
   {
       "_ga": "G-1234",
       "cookie_prefix": "prefix",
       "session_cookies": [
           {
               "measurement_id": "G-1234",
               "session_cookie": "567"
           }
       ]
   }
   ```

2. For Universal Analytics cookies: `iglu:com.google.analytics/cookies/jsonschema/1-0-0` (if enabled)

   ```json
   {
       "_ga": "GA1.2.3.4"
   }
   ```

---

# Track data out-of-the-box with the web trackers
> Track page views, structured events, and self-describing events with automatic context entities and custom timestamps using the web trackers.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/

To track an event, the API is slightly different depending if you're using the JavaScript or Browser version of our web tracker.

The main built-in events are [page views](/docs/sources/web-trackers/tracking-events/page-views/) and [page pings](/docs/sources/web-trackers/tracking-events/activity-page-pings/). Here's how to track them:

**JavaScript (tag):**

```javascript
<script type="text/javascript">
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[];
p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments)
};p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1;
n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));

snowplow('newTracker', 'sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
});

snowplow('enableActivityTracking', {
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

snowplow('trackPageView');
</script>
```

**Browser (npm):**

```javascript
import {
  newTracker,
  trackPageView,
  enableActivityTracking
} from '@snowplow/browser-tracker';

newTracker('sp', '{{collector_url_here}}', {
    appId: 'my-app-id',
});

enableActivityTracking({
  minimumVisitLength: 30,
  heartbeatDelay: 10
});

trackPageView();
```

***

As well as page views and activity tracking, you can track [custom events](/docs/sources/web-trackers/custom-tracking-using-schemas/), or use [plugins](/docs/sources/web-trackers/plugins/) to track a wide range of other events and entities.

## Add contextual data with entities

The tracker can be set up to automatically add [entities](/docs/fundamentals/entities/) to every event sent.

Most entity autotracking is specifically configured using plugins, which are imported, enabled, and configured individually. However, you can configure some entities directly when instrumenting the tracker, using the [configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/).

| Entity                                                                                               | Usage                            | Added by default | JavaScript (tag) tracker | Browser (npm) tracker |
| ---------------------------------------------------------------------------------------------------- | -------------------------------- | ---------------- | ------------------------ | --------------------- |
| [`webPage`](/docs/sources/web-trackers/tracking-events/page-views/#page-view-id-and-web_page-entity) | UUID for the page view           | ✅                | `contexts` config        | `contexts` config     |
| [`session`](/docs/sources/web-trackers/tracking-events/session/)                                     | Data about the current session   | ❌                | `contexts` config        | `contexts` config     |
| [`browser`](/docs/sources/web-trackers/tracking-events/browsers/)                                    | Properties of the user's browser | ❌                | `contexts` config        | `contexts` config     |
| [`performanceTiming`](/docs/sources/web-trackers/tracking-events/timings/)                           | Performance timing metrics       | ❌                | `contexts` config        | Plugin                |
| [`gaCookies`](/docs/sources/web-trackers/tracking-events/ga-cookies/)                                | Extract GA cookie values         | ❌                | `contexts` config        | Plugin                |
| [`geolocation`](/docs/sources/web-trackers/tracking-events/timezone-geolocation/)                    | User's geolocation               | ❌                | `contexts` config        | Plugin                |

If you're using the `sp.lite.js` JavaScript tracker distribution, only the `webPage`, `session`, and `browser` entities are available out of the box, as the others require plugins that aren't included in that distribution.

You can also attach your own [custom entities](/docs/sources/web-trackers/custom-tracking-using-schemas/) to events.

For example, here is a page view with an additional custom entity:

**JavaScript (tag):**

```javascript
snowplow('trackPageView', {
  context: [{
    schema: "iglu:com.example_company/page/jsonschema/1-2-1",
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  }]
});
```

**Browser (npm):**

```javascript
trackPageView({
  context: [{
    schema: 'iglu:com.example_company/page/jsonschema/1-2-1',
    data: {
      pageType: 'test',
      lastUpdated: new Date(2021,04,01)
    }
  }]
});
```

***

> **Note:** Tracker methods available through plugins do not necessarily support adding custom entities. For those please refer to the corresponding plugin documentation for details.

## Set event properties

Certain event properties, including `domain_userid` or `application_id`, can be set as [atomic properties](/docs/fundamentals/canonical-event/) in the raw event.

### Application ID

Set the application ID using the `appId` field of the [tracker configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This will be attached to every event the tracker fires. You can set different application IDs on different parts of your site. You can then distinguish events that occur on different applications by grouping results based on `application_id`.

### Application version

> **Info:** The option to track the application version was introduced in version 4.1 of the JavaScript tracker.

Set the application ID using the `appVersion` field of the [tracker configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This will be attached to every event the tracker fires using the [application entity](/docs/events/ootb-data/app-information/#entity-definitions).

The version of can be a semver-like structure (e.g 1.1.0) or a Git commit SHA hash.

### Application platform

Set the application platform using the `platform` field of the [tracker configuration object](/docs/sources/web-trackers/tracker-setup/initialization-options/). This will be attached to every event the tracker fires. Its default value is `web`. For a list of supported platforms, please see the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields).

### Business user ID

The JavaScript Tracker automatically sets a `domain_userid` based on a first party cookie. Read more about cookies [here](/docs/sources/web-trackers/cookies-and-local-storage/).

There are many situations, however, when you will want to identify a specific user using an ID generated by one of your business systems. To do this, you use one of the methods described in this section: `setUserId`, `setUserIdFromLocation`, `setUserIdFromReferrer`, and `setUserIdFromCookie`.

Typically, companies do this at points in the customer journey where users identify themselves e.g. if they log in.

> **Note:** This will only set the user ID on further events fired while the user is on this page; if you want events on another page to record this user ID too, you must call `setUserId` on the other page as well.

#### `setUserId`

`setUserId` is the simplest of the four methods. It sets the business user ID to a string of your choice:

**JavaScript (tag):**

```javascript
snowplow('setUserId', 'joe.blogs@email.com');
```

**Browser (npm):**

```javascript
setUserId('joe.blogs@email.com');
```

***

> **Note:** `setUserId` can also be called using the alias `identifyUser`.

#### `setUserIdFromLocation`

`setUserIdFromLocation` lets you set the user ID based on a querystring field of your choice. For example, if the URL is `http://www.mysite.com/home?id=user345`, then the following code would set the user ID to “user345”:

**JavaScript (tag):**

```javascript
snowplow('setUserIdFromLocation', 'id');
```

**Browser (npm):**

```javascript
setUserIdFromLocation('id');
```

***

#### `setUserIdFromReferrer`

`setUserIdFromReferrer` functions in the same way as `setUserIdFromLocation`, except that it uses the referrer querystring rather than the querystring of the current page.

**JavaScript (tag):**

```javascript
snowplow('setUserIdFromReferrer', 'id');
```

**Browser (npm):**

```javascript
setUserIdFromReferrer('id');
```

***

#### `setUserIdFromCookie`

Use `setUserIdFromCookie` to set the value of a cookie as the user ID. For example, if you have a cookie called “cookieid” whose value is “user123”, the following code would set the user ID to “user123”:

**JavaScript (tag):**

```javascript
snowplow('setUserIdFromCookie', 'cookieid');
```

**Browser (npm):**

```javascript
setUserIdFromCookie('cookieid');
```

***

### Custom page URL and referrer URL

The Snowplow JavaScript Tracker automatically tracks the page URL and referrer URL on any event tracked. However, in certain situations, you may want to override the one or both of these URLs with a custom value. For example, this might be desirable if your CMS spits out particularly ugly URLs that are hard to unpick at analysis time.

To set a custom page URL, use the `setCustomUrl` method:

**JavaScript (tag):**

```javascript
snowplow('setCustomUrl', 'http://mysite.com/checkout-page');
```

**Browser (npm):**

```javascript
setCustomUrl('http://mysite.com/checkout-page');
```

***

To set a custom referrer, use the `setReferrerUrl` method:

**JavaScript (tag):**

```javascript
snowplow('setReferrerUrl', 'http://custom-referrer.com');
```

**Browser (npm):**

```javascript
setReferrerUrl('http://custom-referrer.com');
```

***

> **Tip:** On an SPA, the page URL might change without the page being reloaded. Whenever an event is fired, the Tracker checks whether the page URL has changed since the last event. If it has, the page URL is updated and the URL at the time of the last event is used as the referrer. If you use `setCustomUrl`, the page URL will no longer be updated in this way. Similarly if you use `setReferrerUrl`, the referrer URL will no longer be updated in this way.
>
> To use `setCustomUrl` within an SPA, call it before all `trackPageView` calls.
>
> If you want to ensure that the original referrer is preserved even though your page URL can change without the page being reloaded, use `setReferrerUrl` like this before sending any events:
>
> **JavaScript (tag):**
>
> ```javascript
> snowplow('setReferrerUrl', document.referrer);
> ```
>
> **Browser (npm):**
>
> ```javascript
> setReferrerUrl(document.referrer);
> ```
>
> ***

### Custom timestamp

Snowplow events have several [timestamps](/docs/events/timestamps/).

Every `trackX...()` method in the tracker allows for a custom timestamp, called `trueTimestamp` to be set.

In certain circumstances you might want to set the timestamp yourself e.g. if the JS tracker is being used to process historical event data, rather than tracking the events live. In this case you can set the `true_timestamp` for the event.

To set the true timestamp add an extra argument to your track method: `{type: 'ttm', value: unixTimestampInMs}`.

This example shows how to set a true timestamp for a page view event:

**JavaScript (tag):**

```javascript
snowplow('trackPageView', {
  timestamp: { type: 'ttm', value: 1361553733371 }
});
```

**Browser (npm):**

```javascript
trackPageView({
  timestamp: { type: 'ttm', value: 1361553733371 }
});
```

***

E.g. to set a true timestamp for a self-describing event:

**JavaScript (tag):**

```javascript
snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/2-0-0',
    data: {
        productId: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        availableSince: new Date(2013,3,7)
    }
  },
  timestamp: { type: 'ttm', value: 1361553733371 }
});
```

**Browser (npm):**

```javascript
trackSelfDescribingEvent({
  event: {
    schema: 'iglu:com.acme_company/viewed_product/jsonschema/2-0-0',
    data: {
        productId: 'ASO01043',
        category: 'Dresses',
        brand: 'ACME',
        returning: true,
        price: 49.95,
        sizes: ['xs', 's', 'l', 'xl', 'xxl'],
        availableSince: new Date(2013,3,7)
    }
  },
  timestamp: { type: 'ttm', value: 1361553733371 }
});
```

***

## Get event properties

It's possible to retrieve certain identifiers and properties for use in your code. You'll need to use a callback for the JavaScript tracker.

**JavaScript (tag):**

If you call `snowplow` with a function as the argument, the function will be executed when `sp.js` loads:

```javascript
snowplow(function () {
  console.log("sp.js has loaded");
});
```

Or equivalently:

```javascript
snowplow(function (x) {
  console.log(x);
}, "sp.js has loaded");
```

The callback you provide is executed as a method on the internal `trackerDictionary` object. You can access the `trackerDictionary` using `this`.

```javascript
// Configure a tracker instance named "sp"
snowplow('newTracker', 'sp', '{{COLLECTOR_URL}', {
 appId: 'snowplowExampleApp'
});

// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var domainUserId = sp.getDomainUserId();
 console.log(domainUserId);
})
```

The callback function shouldn't be a method:

```javascript
// TypeError: Illegal invocation
snowplow(console.log, "sp.js has loaded");
```

This won't work because the value of `this` in the `console.log` function will be the `trackerDictionary`, rather than `console`.

You can get around this problem using `Function.prototoype.bind` as follows:

```javascript
snowplow(console.log.bind(console), "sp.js has loaded");
```

For more on execution context in JavaScript, see the [MDN page](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/this).

**Browser (npm):**

When initialising a tracker, you can use the returned `tracker` instance to access various properties from this tracker instance.

```javascript
// Configure a tracker instance named "sp"
const sp = newTracker('sp', '{{COLLECTOR_URL}', {
 appId: 'snowplowExampleApp'
});

// Access the tracker properties
const domainUserId = sp.getDomainUserId();
```

***

### Cookie values

You can [retrieve cookie values](/docs/sources/web-trackers/cookies-and-local-storage/getting-cookie-values/) using the `getDomainUserInfo` and other getters, or from the cookies directly.

### Page view ID

When the JavaScript Tracker loads on a page, it generates a new [page view UUID](/docs/sources/web-trackers/tracking-events/page-views/). To get this page view ID, use the `getPageViewId` method:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var pageViewId = sp.getPageViewId();
 console.log(pageViewId);
})
```

**Browser (npm):**

```javascript
const pageViewId = sp.getPageViewId();
console.log(pageViewId);
```

***

### Business user ID

The `getUserId` method returns the user ID which you configured using `setUserId()`:

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var userId = sp.getUserId();
 console.log(userId);
})
```

**Browser (npm):**

```javascript
const userId = sp.getUserId();
console.log(userId);
```

***

### Tab ID

If you've enabled the [`browser` entity](/docs/sources/web-trackers/tracking-events/browsers/), you can get the tab ID using the `getTabId` method. It's a UUID identifier for the specific browser tab the event is sent from.

**JavaScript (tag):**

```javascript
// Access the tracker instance inside a callback
snowplow(function () {
 var sp = this.sp;
 var tabId = sp.getTabId();
 console.log(tabId);
})
```

**Browser (npm):**

```javascript
const tabId = sp.getTabId();
console.log(tabId);
```

***

---

# Track link clicks on web
> Automatically track clicks on anchor elements with configurable filters, pseudo-click support, and optional content capture for href destinations.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/link-click/

Link click tracking enables click tracking for all anchor/link elements (HTML `<a>` elements).

Link clicks are tracked as self describing events with [the link\_click schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1). Each link click event captures the link’s href attribute. The event also has fields for the link’s id, classes, and target (where the linked document is opened, such as a new tab or new window).

Link click events are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                             |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                       |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-link-click-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-link-click-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

**Browser (npm):**

- `npm install @snowplow/browser-plugin-link-click-tracking`
- `yarn add @snowplow/browser-plugin-link-click-tracking`
- `pnpm add @snowplow/browser-plugin-link-click-tracking`

***

## Toggle link click tracking

Turn on link click tracking like this:

**JavaScript (tag):**

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-link-click-tracking@latest/dist/index.umd.min.js",
  ["snowplowLinkClickTracking", "LinkClickTrackingPlugin"]
);

snowplow('enableLinkClickTracking');
```

**Browser (npm):**

Initialize your tracker with the plugin.

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { LinkClickTrackingPlugin, enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ LinkClickTrackingPlugin() ],
});

enableLinkClickTracking();
```

***

Use this method once and the tracker will add click event listeners to the document to detect clicks on anchor elements.

An optional, but recommended, parameter is `pseudoClicks`. If this isn't turned on, Firefox won't recognize middle clicks. However, when configured, there is a small possibility of false positives (click events firing when they shouldn't).

**JavaScript (tag):**

```javascript
snowplow('enableLinkClickTracking', { pseudoClicks: true });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

enableLinkClickTracking({ pseudoClicks: true });
```

***

This is its signature (where `?` is an optional property):

**JavaScript (tag):**

```javascript
snowplow('enableLinkClickTracking', {
  options?: FilterCriterion,
  pseudoClicks?: boolean,
  trackContent?: boolean
  context?: SelfDescribingJson[]
});
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

enableLinkClickTracking({
  options?: FilterCriterion,
  pseudoClicks?: boolean,
  trackContent?: boolean
  context?: SelfDescribingJson[]
});
```

***

To stop tracking link events, call `disableLinkClickTracking`:

**JavaScript (tag):**

```javascript
snowplow('disableLinkClickTracking');
```

**Browser (npm):**

```javascript
import { disableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

disableLinkClickTracking();
```

***

## Refresh link click tracking

In previous versions, the `enableLinkClickTracking` method only tracked clicks on links that existed in the document at the time it was called. If new links were added to the page after that, you had to use `refreshLinkClickTracking` to add Snowplow click listeners to any new links.

From v4, this method is **deprecated** and has no effect; event listeners are now added directly to the document rather than to individual link elements and new links should automatically be tracked with no action required.

## Configuration

Control which links to track using the FilterCriterion object.

Where FilterCriterion is an object:

```javascript
interface FilterCriterion {
    /** A collection of class names to include */
    allowlist?: string[];
    /** A collector of class names to exclude */
    denylist?: string[];
    /** A callback which returns a boolean as to whether the element should be included */
    filter?: (elt: HTMLElement) => boolean;
}
```

You can control which links are tracked using the second argument. There are three ways to do this: a denylist, an allowlist, and a filter function.

### Denylist

This is an array of CSS classes which should be ignored by link click tracking. For example, the below code will stop link click events firing for links with the class "barred" or "untracked", but will fire link click events for all other links:

**JavaScript (tag):**

```javascript
snowplow('enableLinkClickTracking', {
  options: {
    denylist: ['barred', 'untracked']
  }
});

// If there is only one class name you wish to deny,
// you should still put it in an array
snowplow('enableLinkClickTracking', { options: { 'denylist': ['barred'] } });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

enableLinkClickTracking({
  options: {
    denylist: ['barred', 'untracked']
  }
});

// If there is only one class name you wish to deny,
// you should still put it in an array
enableLinkClickTracking({ options: { 'denylist': ['barred'] } });
```

***

### Allowlist

The opposite of a denylist. This is an array of the CSS classes of links which you do want to be tracked. Only clicks on links with a class in the list will be tracked.

**JavaScript (tag):**

```javascript
snowplow('enableLinkClickTracking', {
  options: {
    'allowlist': ['unbarred', 'tracked']
  }
});

 // If there is only one class name you wish to allow,
 // you should still put it in an array
snowplow('enableLinkClickTracking', { options: { 'allowlist': ['unbarred'] } });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

enableLinkClickTracking({
  options: {
    'allowlist': ['unbarred', 'tracked']
  }
});

 // If there is only one class name you wish to allow,
 // you should still put it in an array
enableLinkClickTracking({ options: { 'allowlist': ['unbarred'] } });
```

***

### Filter function

You can provide a filter function which determines which links should be tracked. The function should take one argument, the link element, and return either 'true' (in which case clicks on the link will be tracked) or 'false' (in which case they won't).

The following code will track clicks on those and only those links whose id contains the string "interesting":

**JavaScript (tag):**

```javascript
function myFilter (linkElement) {
  return linkElement.id.indexOf('interesting') > -1;
}

snowplow('enableLinkClickTracking', { options: { 'filter': myFilter } });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

function myFilter (linkElement) {
  return linkElement.id.indexOf('interesting') > -1;
}

enableLinkClickTracking({ options: { 'filter': myFilter } });
```

***

Another optional parameter is `trackContent`. Set it to `true` if you want link click events to capture the innerHTML of the clicked link:

**JavaScript (tag):**

```javascript
snowplow('enableLinkClickTracking', { trackContent: true });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

enableLinkClickTracking({ trackContent: true });
```

***

The innerHTML of a link is all the text between the `a` tags. Note that if you use a base 64 encoded image as a link, the entire base 64 string will be included in the event.

Each link click event will include (if available) the destination URL, id, classes and target of the clicked link. (The target attribute of a link specifies a window or frame where the linked document will be loaded.)

**Context**

`enableLinkClickTracking` can also be passed an array of custom context entities to attach to every link click event as an additional final parameter.

Link click tracking supports dynamic context entities. Callbacks passed in the context argument will be evaluated with the source element passed as the only argument. The self-describing JSON context object returned by the callback will be sent with the link click event.

A dynamic context could therefore look something like this for link click events:

**JavaScript (tag):**

```javascript
let dynamicContext = function (element) {
  // perform operations here to construct the context
  return context;
};

snowplow('enableLinkClickTracking', { context: [dynamicContext] });
```

**Browser (npm):**

```javascript
import { enableLinkClickTracking } from '@snowplow/browser-plugin-link-click-tracking';

let dynamicContext = function (element) {
  // perform operations here to construct the context
  return context;
};

enableLinkClickTracking({ context: [ dynamicContext ] });
```

***

See [this page](/docs/sources/web-trackers/custom-tracking-using-schemas/) for more information about tracking context entities.

## Manual link click tracking

You can manually track individual link click events with the `trackLinkClick` method. You do not need to call `enableLinkClickTracking` before using this method. This is its signature:

**JavaScript (tag):**

```javascript
snowplow('trackLinkClick, {
    /** The target URL of the link */
    targetUrl: string;
    /** The ID of the element clicked if present */
    elementId?: string;
    /** An array of class names from the element clicked */
    elementClasses?: Array<string>;
    /** The target value of the element if present */
    elementTarget?: string;
    /** The content of the element if present and enabled */
    elementContent?: string;
});
```

**Browser (npm):**

```javascript
import { trackLinkClick } from '@snowplow/browser-plugin-link-click-tracking';

trackLinkClick({
    /** The target URL of the link */
    targetUrl: string;
    /** The ID of the element clicked if present */
    elementId?: string;
    /** An array of class names from the element clicked */
    elementClasses?: Array<string>;
    /** The target value of the element if present */
    elementTarget?: string;
    /** The content of the element if present and enabled */
    elementContent?: string;
});
```

***

Of these arguments, only `targetUrl` is required. This is how to use `trackLinkClick`:

**JavaScript (tag):**

```javascript
snowplow('trackLinkClick', {
  targetUrl: 'http://www.example.com',
  elementId: 'first-link',
  elementClasses: ['class-1', 'class-2'],
  elementTarget: '',
  elementContent: 'this page'
});
```

**Browser (npm):**

```javascript
import { trackLinkClick } from '@snowplow/browser-plugin-link-click-tracking';

trackLinkClick({
  targetUrl: 'http://www.example.com',
  elementId: 'first-link',
  elementClasses: ['class-1', 'class-2'],
  elementTarget: '',
  elementContent: 'this page'
});
```

***

Rather than specify the values explicitly, you may also supply the link element directly (and optionally control whether to include the content or not):

**JavaScript (tag):**

```javascript
snowplow('trackLinkClick', {
  element: document.links[0],
  trackContent: false
});
```

**Browser (npm):**

```javascript
import { trackLinkClick } from '@snowplow/browser-plugin-link-click-tracking';

trackLinkClick({
  element: document.links[0],
  trackContent: false
});
```

***

---

# HTML5 media tracking on web
> Automatically track HTML5 video and audio elements with media events including play, pause, seek, buffer, and progress milestones.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/media/html5/

This plugin enables the automatic tracking of HTML5 media elements (`<video>` and `<audio>`) on a webpage.

It uses the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/) under the hood.

HTML5 media events and entities are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                        |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                  |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-media-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media-tracking@latest/dist/index.umd.min.js",
  ["snowplowMediaTracking", "MediaTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-media-tracking`
- `yarn add @snowplow/browser-plugin-media-tracking`
- `pnpm add @snowplow/browser-plugin-media-tracking`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { MediaTrackingPlugin, startHtml5MediaTracking } from '@snowplow/browser-plugin-media-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ MediaTrackingPlugin() ],
});

startHtml5MediaTracking(/* options */);
```

***

## Quick Start

To start tracking media with default settings:

**JavaScript (tag):**

**`main.js`**

```html
<video id='html-id' src='./video.mp4'></video>
<script>
  window.snowplow('addPlugin',
    "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media-tracking@latest/dist/index.umd.min.js",
    ["snowplowMediaTracking", "MediaTrackingPlugin"]
  );

  const sessionId = crypto.randomUUID();

  window.snowplow('startHtml5MediaTracking', {
    id: sessionId,
    video: 'html-id',
  })
</script>
```

**`index.html`**

```html
<video id='html-id' src='./video.mp4'></video>
```

**Browser (npm):**

**`main.js`**

```javascript
import { startHtml5MediaTracking } from '@snowplow/browser-plugin-media-tracking'

const sessionId = crypto.randomUUID();

startHtml5MediaTracking({
  id: sessionId,
  video: 'html-id',
})
```

**`index.html`**

```html
<video id='html-id' src='./video.mp4'></video>
```

***

## Start tracking

Start tracking HTML5 media by calling the `startHtml5MediaTracking` function with a configuration object. You must provide a unique `id` for the media session, along with the `video` ID.

**JavaScript (tag):**

```javascript
window.snowplow('startHtml5MediaTracking', {
  id: "unique-session-id",
  video: "myVideoElement",
  label: "Product Demo Video",
  captureEvents: ["play", "pause", "end"],
  boundaries: [25, 50, 75, 100],
  pings: {
    pingInterval: 20,
    maxPausedPings: 2,
  },
  updatePageActivityWhilePlaying: true,
  filterOutRepeatedEvents: {
    seekEvents: true,
    volumeChangeEvents: false,
    flushTimeoutMs: 5000
  },
  session: {
    // Custom session start time
    startedAt: new Date('2024-01-01T12:00:00Z'),
  },
});
```

**Browser (npm):**

```javascript
startHtml5MediaTracking({
  id: "unique-session-id",
  video: document.getElementById("myVideoElement"),
  label: "Product Demo Video",
  captureEvents: ["play", "pause", "end"],
  boundaries: [25, 50, 75, 100],
  pings: {
    pingInterval: 20,
    maxPausedPings: 2,
  },
  updatePageActivityWhilePlaying: true,
  filterOutRepeatedEvents: {
    seekEvents: true,
    volumeChangeEvents: false,
    flushTimeoutMs: 5000
  },
  session: {
    // Custom session start time
    startedAt: new Date('2024-01-01T12:00:00Z'),
  },
});
```

***

| Parameter                                       | Type                                                                                                                 | Description                                                                                    | Default / Required            |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------- |
| `id`                                            | `string`                                                                                                             | A unique session UUID for each media element, used to identify and end tracking.               | Required                      |
| `video`                                         | `string` or `HTMLMediaElement`                                                                                       | The ID of the media element (as a string) or the media element itself.                         | Required                      |
| `label`                                         | `string`                                                                                                             | A human-readable label for the media element.                                                  | `undefined`                   |
| `captureEvents`                                 | `HTML5MediaEventTypes`                                                                                               | A list of media events to track.                                                               | All events tracked by default |
| `boundaries`                                    | `number[]`                                                                                                           | Percentage thresholds (0-100) to trigger progress events.                                      | `[10, 25, 50, 75]`            |
| `context`                                       | [`DynamicContext`](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#global-contexts-methods) | Dynamic contexts attached to each tracking event.                                              | `undefined`                   |
| `updatePageActivityWhilePlaying`                | `boolean`                                                                                                            | Whether to update page activity while media is playing.                                        | `true`                        |
| `filterOutRepeatedEvents`                       | `FilterOutRepeatedEvents`                                                                                            | Whether to suppress consecutive identical events.                                              | Enabled by default            |
| `filterOutRepeatedEvents` `.seekEvents`         | `boolean`                                                                                                            | Whether to filter out seek start and end events tracked after each other.                      | `true`                        |
| `filterOutRepeatedEvents` `.volumeChangeEvents` | `boolean`                                                                                                            | Whether to filter out volume change events tracked after each other.                           | `true`                        |
| `filterOutRepeatedEvents` `.flushTimeoutMs`     | `number`                                                                                                             | Timeout in milliseconds after which to send the events that are queued for filtering.          | `5000` (milliseconds)         |
| `pings.pingInterval`                            | `number`                                                                                                             | Interval (in seconds) for sending ping events.                                                 | `30` (seconds)                |
| `pings.maxPausedPings`                          | `number`                                                                                                             | Maximum number of ping events sent while playback is paused.                                   | `1`                           |
| `session`                                       | `boolean` or `object`                                                                                                | Whether to track the media session entity for playback statistics, or set a custom start time. | `true`                        |

## Stop tracking

End tracking by calling `endHtml5MediaTracking`. This function disables auto tracking for the player registered with the provided session ID. It will remove any event listeners and poll intervals, and call [`endMediaTracking`](/docs/sources/web-trackers/tracking-events/media/snowplow/#usage) from the core plugin.

**JavaScript (tag):**

**`main.js`**

```html
<video id='html-id' src='./video.mp4'></video>
<script>
  window.snowplow('addPlugin',
    "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media-tracking@latest/dist/index.umd.min.js",
    ["snowplowMediaTracking", "MediaTrackingPlugin"]
  );

  const sessionId = crypto.randomUUID();

  window.snowplow('startHtml5MediaTracking', {
    id: sessionId,
    video: 'html-id',
  })

  // Tracking some video events...

  window.snowplow('endHtml5MediaTracking', sessionId)
</script>
```

**Browser (npm):**

**`main.js`**

```javascript
import { startHtml5MediaTracking, endHtml5MediaTracking } from '@snowplow/browser-plugin-media-tracking'

const sessionId = crypto.randomUUID();

startHtml5MediaTracking({
  id: "session-id",
  video: 'html-id',
})

// Tracking some video events...

endHtml5MediaTracking(sessionId)
```

**`index.html`**

```html
<video id='html-id' src='./video.mp4'></video>
```

***

## Events

The HTML5 media plugin can track the following events:

| Event Type             | Event String                | Description                                                                                   |
| ---------------------- | --------------------------- | --------------------------------------------------------------------------------------------- |
| Ready                  | `ready`                     | Fired when media tracking is successfully attached to the player and ready to track events.   |
| Play                   | `play`                      | Fired when the player transitions from a paused state to playing.                             |
| Pause                  | `pause`                     | Fired when the user pauses the playback.                                                      |
| End                    | `end`                       | Fired when playback stops because the media has reached its end or no more data is available. |
| SeekEnd                | `seek_end`                  | Fired when a seek operation is completed.                                                     |
| PlaybackRateChange     | `playback_rate_change`      | Fired when the playback rate (speed) of the media changes.                                    |
| VolumeChange           | `volume_change`             | Fired when the volume level of the media changes.                                             |
| FullscreenChange       | `fullscreen_change`         | Fired when the browser enters or exits full-screen mode.                                      |
| PictureInPictureChange | `picture_in_picture_change` | Fired when the browser enters or exits picture-in-picture mode.                               |
| BufferStart            | `buffer_start`              | Fired when the media player starts buffering (loading content to play).                       |
| BufferEnd              | `buffer_end`                | Fired when buffering ends, and playback resumes.                                              |
| Error                  | `error`                     | Fired when an error occurs during the loading or playback of the media.                       |
| Ping                   | `ping`                      | Fired periodically during media playback, regardless of other events.                         |
| PercentProgress        | `percent_progress`          | Fired when a specific percentage of the media content has been played (as set by boundaries). |

It's possible to only track a subset of these events by passing an array of event types to the `captureEvents` option:

**JavaScript (tag):**

```javascript
window.snowplow("startHtml5MediaTracking", {
  id: crypto.randomUUID(),
  video: 'example-video',
  captureEvents: ['play', 'pause'],
})
```

**Browser (npm):**

```javascript
import { startHtml5MediaTracking, HTML5MediaEventTypes } from '@snowplow/browser-plugin-media-tracking'

const sessionId = uuidv4();

startHtml5MediaTracking({
  id: crypto.randomUUID(),
  video: 'example-video',
  captureEvents: [HTML5MediaEventTypes.Play, HTML5MediaEventTypes.Pause],
})
```

***

## Additional entities

The event and entity schemas are [the same as](/docs/events/ootb-data/media-events/) for the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/).

Along with the standard Snowplow media entities, the HTML5 media plugin also tracks two HTML5 media-specific entities, `media_element` and `video_element`.

For `<video>` elements, both entities are added to all media events in the session. For `<audio>` elements, only the `media_element` entity is added.

If you've configured the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package to use the [v1 schemas](/docs/events/ootb-data/media-events/#media-api-versions), or are using an older version of the Media Player model, it will use fields from the `media_element` and `video_element` entities. The Media Player model version 0.6+ doesn't use them.

### HTML5 media element

### `media_element`

**Type:** Entity

Context Schema for a media player event

**Schema:** `iglu:org.whatwg/media_element/jsonschema/1-0-0`

**Example:**

```json
{
  "htmlId": "my-video",
  "mediaType": "VIDEO",
  "autoPlay": false,
  "buffered": [
    {
      "start": 0,
      "end": 20
    }
  ],
  "controls": true,
  "currentSrc": "http://example.com/video.mp4",
  "defaultMuted": true,
  "defaultPlaybackRate": 1,
  "disableRemotePlayback": false,
  "error": null,
  "networkState": "NETWORK_IDLE",
  "preload": "metadata",
  "readyState": "HAVE_ENOUGH_DATA",
  "seekable": [
    {
      "start": 0,
      "end": 20
    }
  ],
  "seeking": false,
  "src": "http://example.com/video.mp4",
  "textTracks": [
    {
      "label": "English",
      "language": "en",
      "kind": "captions",
      "mode": "showing"
    }
  ],
  "fileExtension": "mp4",
  "fullscreen": false,
  "pictureInPicture": false
}
```

**Properties:**

| Property                          | Description                                                                                                                                                     |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `autoPlay` _boolean_              | _Required._ If playback should automatically begin as soon as enough media is available to do so without interruption                                           |
| `buffered` _array_                | _Required._ An array of time ranges that have been buffered                                                                                                     |
| `controls` _boolean_              | _Required._ If the user agent should provide it's own set of controls                                                                                           |
| `crossOrigin` _string_            | _Optional._ CORS settings value of the media player                                                                                                             |
| `currentSrc` _string_             | _Required._ The absolute URL of the media resource                                                                                                              |
| `defaultMuted` _boolean_          | _Required._ If audio is muted by default                                                                                                                        |
| `defaultPlaybackRate` _number_    | _Required._ The default media playback rate of the player                                                                                                       |
| `disableRemotePlayback` _boolean_ | _Optional._ If the media element is allowed to have a remote playback UI                                                                                        |
| `error` _object_                  | _Required._ An object of the latest error to occur, or null if no errors                                                                                        |
| `fileExtension` _string_          | _Optional._ The media file format                                                                                                                               |
| `fullscreen` _boolean_            | _Optional._ If the video element is fullscreen                                                                                                                  |
| `mediaType` _string_              | _Required._ If the media is a video element, or audio Must be one of: `AUDIO`, `VIDEO`                                                                          |
| `networkState` _string_           | _Required._ The current state of the fetching of media over the network Must be one of: `NETWORK_EMPTY`, `NETWORK_IDLE`, `NETWORK_LOADING`, `NETWORK_NO_SOURCE` |
| `pictureInPicture` _boolean_      | _Optional._ If the video element is showing Picture-In-Picture                                                                                                  |
| `played` _array_                  | _Optional._ An array of time ranges played                                                                                                                      |
| `htmlId` _string_                 | _Required._ The HTML id of the element                                                                                                                          |
| `preload` _string_                | _Required._ The 'preload' HTML attribute of the media                                                                                                           |
| `readyState` _string_             | _Required._ The readiness of the media Must be one of: `HAVE_NOTHING`, `HAVE_METADATA`, `HAVE_CURRENT_DATA`, `HAVE_FUTURE_DATA`, `HAVE_ENOUGH_DATA`             |
| `seekable` _array_                | _Required._ Seekable time range(s)                                                                                                                              |
| `seeking` _boolean_               | _Required._ If the media is in the process of seeking to a new position                                                                                         |
| `src` _string_                    | _Optional._ The 'src' HTML attribute of the media element                                                                                                       |
| `textTracks` _array_              | _Optional._ An array of TextTrack objects on the media element                                                                                                  |

### HTML5 video element

### `video_element`

**Type:** Entity

Context Schema for a video player event

**Schema:** `iglu:org.whatwg/video_element/jsonschema/1-0-0`

**Example:**

```json
{
  "autoPictureInPicture": false,
  "disablePictureInPicture": false,
  "poster": "http://www.example.com/poster.jpg",
  "videoHeight": 300,
  "videoWidth": 400
}
```

**Properties:**

| Property                            | Description                                                                                                                                             |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `autoPictureInPicture` _boolean_    | _Optional._ A boolean value that is true if the video should enter or leave picture-in-picture mode automatically when changing tab and/or application  |
| `disablePictureInPicture` _boolean_ | _Optional._ The disablePictureInPicture property will hint the user agent to not suggest the picture-in-picture to users or to request it automatically |
| `poster` _string_                   | _Optional._ 'poster' HTML attribute, which specifies an image to show while no video data is available                                                  |
| `videoHeight` _integer_             | _Required._ A value indicating the intrinsic height of the resource in CSS pixels, or 0 if no media is available yet                                    |
| `videoWidth` _integer_              | _Required._ A value indicating the intrinsic width of the resource in CSS pixels, or 0 if no media is available yet                                     |

---

# Track media on web
> Choose from four media tracking plugins supporting HTML5, YouTube, Vimeo, and custom players with version 2 media schemas.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/media/

There are four media tracking plugins to choose from. Choosing the right plugin for you depends on the following questions:

1. What media player do you use in your app?
2. Do you prefer the tracker to automatically subscribe to and track events from the media player, or do you prefer to track the events manually by calling track event functions (e.g., you have a wrapper around the media player which listens to the player events and can send them to Snowplow)?

| Plugin name                                                                  | Plugin                            | [Media schemas version](/docs/events/ootb-data/media-events/#media-api-versions) | Provides auto-tracking? | Player  |
| ---------------------------------------------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------- | ----------------------- | ------- |
| [Snowplow media](/docs/sources/web-trackers/tracking-events/media/snowplow/) | `browser-plugin-media`            | v2                                                                               | ❌                       | Any     |
| [Vimeo](/docs/sources/web-trackers/tracking-events/media/vimeo/)             | `browser-plugin-vimeo-tracking`   | v2                                                                               | ✅                       | Vimeo   |
| [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/)             | `browser-plugin-media-tracking`   | v2                                                                               | ✅                       | HTML5   |
| [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/)         | `browser-plugin-youtube-tracking` | v2                                                                               | ✅                       | YouTube |

---

# Snowplow media tracking on web
> Manually track media events from any video player using flexible event tracking methods with custom entities and ad support.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/media/snowplow/

This plugin is the recommended way to manually track media events from video players. This plugin allows you to implement media tracking for any player.

See the [media tracking overview page](/docs/events/ootb-data/media-events/) for more details on schemas and using the plugin.

> **Info:** To illustrate the tracked [events](/docs/fundamentals/events/) and [entities](/docs/fundamentals/entities/), visit our [example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media) that showcases the tracked media events and entities live as you watch a video.
>
> Source code for the app is [available here](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples).

> **Note:** The plugin is available since version 3.12 of the tracker.

Snowplow media events and entities must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                               |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)         |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-media@latest/dist/index.umd.min.js',
    ['snowplowMedia', 'SnowplowMediaPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-media`
- `yarn add @snowplow/browser-plugin-media`
- `pnpm add @snowplow/browser-plugin-media`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { SnowplowMediaPlugin } from '@snowplow/browser-plugin-media';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SnowplowMediaPlugin() ],
});
```

***

## Usage

The Snowplow media tracking APIs enable you to track events from media playback on the web and in mobile apps. Track changes in playback state (play, pause, seek), playback position (ping and percentage progress events), and ad playback (ad breaks, ad progress, ad clicks).

For details on the events and entities tracked, see the [media tracking overview](/docs/events/ootb-data/media-events/).

> **Info:** See the tracked media events and entities live as you watch a video in our [React example app](https://snowplow-industry-solutions.github.io/snowplow-javascript-tracker-examples/media). [View the source code](https://github.com/snowplow-industry-solutions/snowplow-javascript-tracker-examples/tree/master/react).

## Quick start

A media tracking session follows this lifecycle:

1. **Start tracking** when the player loads (before playback begins)
2. **Update player state** every second during playback
3. **Track events** as they occur (play, pause, seek, etc.)
4. **End tracking** when the user leaves

**JavaScript (tag):**

```javascript
// 1. Start tracking when player loads
const id = crypto.randomUUID();
window.snowplow('startMediaTracking', {
  id,
  player: {
      duration: 300,
      label: 'My Video Title',
      mediaType: 'video',
      playerType: 'html5'
  }
});

// 2. Update player state every second
setInterval(() => {
  window.snowplow('updateMediaTracking', {
      id,
      player: { currentTime: player.currentTime }
  });
}, 1000);

// 3. Track events as they occur
window.snowplow('trackMediaPlay', { id });
window.snowplow('trackMediaPause', { id });
window.snowplow('trackMediaSeekStart', { id });
window.snowplow('trackMediaSeekEnd', { id });
window.snowplow('trackMediaEnd', { id });

// 4. End tracking when done
window.snowplow('endMediaTracking', { id });
```

**Browser (npm):**

```javascript
import {
  startMediaTracking,
  updateMediaTracking,
  endMediaTracking,
  trackMediaPlay,
  trackMediaPause,
  trackMediaSeekStart,
  trackMediaSeekEnd,
  trackMediaEnd,
  MediaType
} from "@snowplow/browser-plugin-media";

// 1. Start tracking when player loads
const id = crypto.randomUUID();
startMediaTracking({
  id,
  player: {
      duration: 300,
      label: 'My Video Title',
      mediaType: MediaType.Video,
      playerType: 'html5'
  }
});

// 2. Update player state every second
setInterval(() => {
  updateMediaTracking({
      id,
      player: { currentTime: player.currentTime }
  });
}, 1000);

// 3. Track events as they occur
trackMediaPlay({ id });
trackMediaPause({ id });
trackMediaSeekStart({ id });
trackMediaSeekEnd({ id });
trackMediaEnd({ id });

// 4. End tracking when done
endMediaTracking({ id });
```

***

## Configuration

Configure media tracking when you call `startMediaTracking`. All configuration is optional.

### Player properties

Set initial player properties to populate the [media player entity](/docs/events/ootb-data/media-events/#media-player). The `label` property is recommended as it helps identify content during analysis.

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', {
  id,
  player: {
      currentTime: 0,          // Current playback position in seconds
      duration: 300,           // Total duration in seconds
      ended: false,            // Whether playback has ended
      livestream: false,       // Whether this is a live stream
      label: 'My Video',       // Human-readable title (recommended)
      loop: false,             // Whether to restart after ending
      mediaType: 'video',      // 'video' or 'audio'
      muted: false,            // Whether audio is muted
      paused: true,            // Whether playback is paused
      pictureInPicture: false, // Whether in picture-in-picture mode
      playerType: 'html5',     // Player type identifier
      playbackRate: 1.0,       // Playback speed (1 = normal)
      quality: '1080p',        // Quality level
      volume: 100              // Volume percentage (0-100)
  }
});
```

**Browser (npm):**

```javascript
import { startMediaTracking, MediaType } from "@snowplow/browser-plugin-media";

startMediaTracking({
  id,
  player: {
      currentTime: 0,          // Current playback position in seconds
      duration: 300,           // Total duration in seconds
      ended: false,            // Whether playback has ended
      livestream: false,       // Whether this is a live stream
      label: 'My Video',       // Human-readable title (recommended)
      loop: false,             // Whether to restart after ending
      mediaType: MediaType.Video, // MediaType.Video or MediaType.Audio
      muted: false,            // Whether audio is muted
      paused: true,            // Whether playback is paused
      pictureInPicture: false, // Whether in picture-in-picture mode
      playerType: 'html5',     // Player type identifier
      playbackRate: 1.0,       // Playback speed (1 = normal)
      quality: '1080p',        // Quality level
      volume: 100              // Volume percentage (0-100)
  }
});
```

***

### Ping events

Media ping events are sent at regular intervals (default: 30 seconds) to report playback position. You can configure ping behavior in the starting configuration.

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', {
  id,
  pings: {
      pingInterval: 30,   // Seconds between pings (default: 30)
      maxPausedPings: 1   // Max pings while paused (default: 1)
  }
});

// Or turn off pings entirely
window.snowplow('startMediaTracking', { id, pings: false });
```

**Browser (npm):**

```javascript
startMediaTracking({
  id,
  pings: {
      pingInterval: 30,   // Seconds between pings (default: 30)
      maxPausedPings: 1   // Max pings while paused (default: 1)
  }
});

// Or turn off pings entirely
startMediaTracking({ id, pings: false });
```

***

> **Info:** Media ping events are separate from page ping events tracked by [activity tracking](/docs/events/ootb-data/page-activity-tracking). Media pings contain playback information (player and session entities), while page pings do not.

### Percentage progress

Track events when playback reaches specified percentage boundaries.

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', {
  id,
  boundaries: [10, 25, 50, 75, 90]  // Fire events at these percentages
});
```

**Browser (npm):**

```javascript
startMediaTracking({
  id,
  boundaries: [10, 25, 50, 75, 90]  // Fire events at these percentages
});
```

***

### Session tracking

The [media session entity](/docs/events/ootb-data/media-events/#media-session) is attached to all media events by default. It's optional for the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) data model, so you could choose not to include it.

We recommend tracking this entity.

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', { id, session: true });
```

**Browser (npm):**

```javascript
startMediaTracking({ id, session: true });
```

***

To set a custom session start time, pass an object with a `startedAt` property instead of a boolean:

```javascript
session: { startedAt: new Date('2024-01-15T10:30:00Z') }
```

### Custom entities

You can attach custom entities to all events for this media tracking instance.

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', {
  id,
  context: [
      {
          schema: 'iglu:com.example/video_metadata/jsonschema/1-0-0',
          data: { contentId: 'abc123', category: 'tutorial' }
      }
  ]
});
```

**Browser (npm):**

```javascript
startMediaTracking({
  id,
  context: [
      {
          schema: 'iglu:com.example/video_metadata/jsonschema/1-0-0',
          data: { contentId: 'abc123', category: 'tutorial' }
      }
  ]
});
```

***

### Filter events

**JavaScript (tag):**

Only track specific event types by providing an allowlist:

```javascript
window.snowplow('startMediaTracking', {
  id,
  captureEvents: ['play', 'pause', 'end']  // Only track these events
});
```

When users scrub through video, media players can generate many rapid seek or volume change events. The tracker automatically coalesces these into just the first and last event. This behavior is optional:

```javascript
// Turn off filtering entirely
window.snowplow('startMediaTracking', { id, filterOutRepeatedEvents: false });

// Or configure separately for seek and volume events
window.snowplow('startMediaTracking', {
  id,
  filterOutRepeatedEvents: {
      seekEvents: true,
      volumeChangeEvents: false,
      flushTimeoutMs: 5000  // Time before flushing buffered events (default: 5000)
  }
});
```

**Browser (npm):**

Only track specific event types by providing an allowlist:

```javascript
import { MediaEventType } from "@snowplow/browser-plugin-media";

startMediaTracking({
  id,
  captureEvents: [MediaEventType.Play, MediaEventType.Pause, MediaEventType.End]
});
```

When users scrub through video, media players can generate many rapid seek or volume change events. The tracker automatically coalesces these into just the first and last event. This behavior is optional:

```javascript
// Turn off filtering entirely
startMediaTracking({ id, filterOutRepeatedEvents: false });

// Or configure separately for seek and volume events
startMediaTracking({
  id,
  filterOutRepeatedEvents: {
      seekEvents: true,
      volumeChangeEvents: false,
      flushTimeoutMs: 5000  // Time before flushing buffered events (default: 5000)
  }
});
```

***

## Tracking events

Track events by calling the appropriate function when player events occur. For a complete list of available events and their properties, see the [media events reference](/docs/events/ootb-data/media-events/#playback-lifecycle-events).

### Playback events

**JavaScript (tag):**

```javascript
window.snowplow('trackMediaReady', { id });        // Player is ready
window.snowplow('trackMediaPlay', { id });          // Playback started
window.snowplow('trackMediaPause', { id });         // Playback paused
window.snowplow('trackMediaEnd', { id });           // Playback ended
window.snowplow('trackMediaSeekStart', { id });     // Seek operation started
window.snowplow('trackMediaSeekEnd', { id });       // Seek operation ended
window.snowplow('trackMediaBufferStart', { id });   // Buffering started
window.snowplow('trackMediaBufferEnd', { id });     // Buffering ended
```

**Browser (npm):**

```javascript
import {
  trackMediaReady, trackMediaPlay, trackMediaPause, trackMediaEnd,
  trackMediaSeekStart, trackMediaSeekEnd, trackMediaBufferStart, trackMediaBufferEnd
} from "@snowplow/browser-plugin-media";

trackMediaReady({ id });        // Player is ready
trackMediaPlay({ id });         // Playback started
trackMediaPause({ id });        // Playback paused
trackMediaEnd({ id });          // Playback ended
trackMediaSeekStart({ id });    // Seek operation started
trackMediaSeekEnd({ id });      // Seek operation ended
trackMediaBufferStart({ id });  // Buffering started
trackMediaBufferEnd({ id });    // Buffering ended
```

***

### Player state events

**JavaScript (tag):**

```javascript
window.snowplow('trackMediaPlaybackRateChange', { id, newRate: 1.5 });
window.snowplow('trackMediaVolumeChange', { id, newVolume: 80 });
window.snowplow('trackMediaFullscreenChange', { id, fullscreen: true });
window.snowplow('trackMediaPictureInPictureChange', { id, pictureInPicture: true });
window.snowplow('trackMediaQualityChange', {
  id,
  newQuality: '1080p',
  bitrate: 5000,
  framesPerSecond: 30,
  automatic: true
});
window.snowplow('trackMediaError', {
  id,
  errorCode: '500',
  errorName: 'NetworkError',
  errorDescription: 'Failed to load media'
});
```

**Browser (npm):**

```javascript
import {
  trackMediaPlaybackRateChange, trackMediaVolumeChange, trackMediaFullscreenChange,
  trackMediaPictureInPictureChange, trackMediaQualityChange, trackMediaError
} from "@snowplow/browser-plugin-media";

trackMediaPlaybackRateChange({ id, newRate: 1.5 });
trackMediaVolumeChange({ id, newVolume: 80 });
trackMediaFullscreenChange({ id, fullscreen: true });
trackMediaPictureInPictureChange({ id, pictureInPicture: true });
trackMediaQualityChange({
  id,
  newQuality: '1080p',
  bitrate: 5000,
  framesPerSecond: 30,
  automatic: true
});
trackMediaError({
  id,
  errorCode: '500',
  errorName: 'NetworkError',
  errorDescription: 'Failed to load media'
});
```

***

### Ad events

Track advertising events with ad and ad break information. See the [media ad](/docs/events/ootb-data/media-events/#media-ad) and [media ad break](/docs/events/ootb-data/media-events/#media-ad-break) entity schemas for property details.

**JavaScript (tag):**

```javascript
// Ad break events
window.snowplow('trackMediaAdBreakStart', {
  id,
  adBreak: {
      breakId: 'break-1',
      name: 'pre-roll',
      breakType: 'linear',
      podSize: 3
  }
});
window.snowplow('trackMediaAdBreakEnd', { id });

// Ad events
window.snowplow('trackMediaAdStart', {
  id,
  ad: {
      adId: 'ad-1',
      name: 'Product Ad',
      creativeId: 'creative-1',
      duration: 30,
      skippable: true
  }
});
window.snowplow('trackMediaAdFirstQuartile', { id });
window.snowplow('trackMediaAdMidpoint', { id });
window.snowplow('trackMediaAdThirdQuartile', { id });
window.snowplow('trackMediaAdComplete', { id });
window.snowplow('trackMediaAdSkip', { id });
window.snowplow('trackMediaAdClick', { id });
window.snowplow('trackMediaAdPause', { id });
window.snowplow('trackMediaAdResume', { id });
```

**Browser (npm):**

```javascript
import {
  trackMediaAdBreakStart, trackMediaAdBreakEnd, trackMediaAdStart,
  trackMediaAdFirstQuartile, trackMediaAdMidpoint, trackMediaAdThirdQuartile,
  trackMediaAdComplete, trackMediaAdSkip, trackMediaAdClick,
  trackMediaAdPause, trackMediaAdResume, MediaPlayerAdBreakType
} from "@snowplow/browser-plugin-media";

// Ad break events
trackMediaAdBreakStart({
  id,
  adBreak: {
      breakId: 'break-1',
      name: 'pre-roll',
      breakType: MediaPlayerAdBreakType.Linear,
      podSize: 3
  }
});
trackMediaAdBreakEnd({ id });

// Ad events
trackMediaAdStart({
  id,
  ad: {
      adId: 'ad-1',
      name: 'Product Ad',
      creativeId: 'creative-1',
      duration: 30,
      skippable: true
  }
});
trackMediaAdFirstQuartile({ id });
trackMediaAdMidpoint({ id });
trackMediaAdThirdQuartile({ id });
trackMediaAdComplete({ id });
trackMediaAdSkip({ id });
trackMediaAdClick({ id });
trackMediaAdPause({ id });
trackMediaAdResume({ id });
```

***

### Custom events

Track custom self-describing events within the media session. The tracker automatically attaches media entities.

**JavaScript (tag):**

```javascript
window.snowplow('trackMediaSelfDescribingEvent', {
  id,
  event: {
      schema: 'iglu:com.example/video_interaction/jsonschema/1-0-0',
      data: { action: 'share', platform: 'twitter' }
  }
});
```

**Browser (npm):**

```javascript
import { trackMediaSelfDescribingEvent } from "@snowplow/browser-plugin-media";

trackMediaSelfDescribingEvent({
  id,
  event: {
      schema: 'iglu:com.example/video_interaction/jsonschema/1-0-0',
      data: { action: 'share', platform: 'twitter' }
  }
});
```

***

## Update player state

**JavaScript (tag):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```javascript
window.snowplow('updateMediaTracking', {
  id,
  player: { currentTime: player.currentTime }
});
```

**Browser (npm):**

Update the player state every second during playback. This ensures accurate metrics in ping events and the session entity. This function does not send any events.

```javascript
import { updateMediaTracking } from "@snowplow/browser-plugin-media";

updateMediaTracking({
  id,
  player: { currentTime: player.currentTime }
});
```

***

## Page activity during playback

When users watch video, they may not interact with the page, causing page pings to stop. The media plugin automatically keeps page activity alive during playback by calling `updatePageActivity` whenever media events are tracked.

You can turn off this behavior if not needed:

**JavaScript (tag):**

```javascript
window.snowplow('startMediaTracking', {
    id,
    updatePageActivityWhilePlaying: false
});
```

**Browser (npm):**

```javascript
startMediaTracking({
    id,
    updatePageActivityWhilePlaying: false
});
```

***

---

# Vimeo media tracking on web
> Automatically track Vimeo video players embedded via iframe or Vimeo Player SDK with comprehensive playback and interaction events.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/media/vimeo/

This plugin enables the automatic tracking of a Vimeo video. It uses the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/) under the hood.

Vimeo media events and entities are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                        |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                  |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-vimeo-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-vimeo-tracking@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-vimeo-tracking@latest/dist/index.umd.min.js',
    ['snowplowVimeoTracking', 'VimeoTrackingPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-vimeo-tracking`
- `yarn add @snowplow/browser-plugin-vimeo-tracking`
- `pnpm add @snowplow/browser-plugin-vimeo-tracking`

```javascript
import { VimeoTrackingPlugin } from '@snowplow/browser-plugin-vimeo-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ VimeoTrackingPlugin() ],
});
```

***

## Quick Start

The snippets below show how to get started with the plugin, after [setting up your tracker](/docs/sources/web-trackers/tracker-setup/).

Using an `iFrame`:

**JavaScript (tag):**

```html
<iframe
  id='vimeo-iframe'
  src='https://player.vimeo.com/video/535907279?h=db7ea8b89c'
></iframe>
```

```javascript
const id = 'XXXXX'; // randomly generated ID
const video = document.getElementById('vimeo-iframe')

window.snowplow('startVimeoTracking', { id, video });

// Vimeo events are tracked...

window.snowplow('endVimeoTracking', id);
```

**Browser (npm):**

```html
<iframe
  id='vimeo-iframe'
  src='https://player.vimeo.com/video/535907279?h=db7ea8b89c'
></iframe>
```

```javascript
import { startVimeoTracking, endVimeoTracking } from '@snowplow/browser-plugin-vimeo-tracking'

const id = 'XXXXX'; // randomly generated ID
const video = document.getElementById('vimeo-iframe')

startVimeoTracking({ id, video })

// Vimeo events are tracked...

endVimeoTracking(id)
```

***

Using `Vimeo.Player`:

**JavaScript (tag):**

```html
<div id='vimeo-player'></div>
```

```javascript
const id = 'XXXXX'; // randomly generated ID
const video = new Vimeo.Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

window.snowplow('startVimeoTracking', { id, video });

// Vimeo events are tracked...

window.snowplow('endVimeoTracking', id);
```

**Browser (npm):**

```html
<div id='vimeo-player'></div>
```

```javascript
import { Player } from '@vimeo/player'
import { startVimeoTracking, endVimeoTracking } from '@snowplow/browser-plugin-vimeo-tracking'

const id = 'XXXXX'; // randomly generated ID
const video = new Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

startVimeoTracking({ id, video })

// Vimeo events are tracked...

endVimeoTracking(id)
```

***

## Start tracking

Use the `startVimeoTracking` function to begin tracking Vimeo media events. You must provide a unique `id` for the media session, along with the `video` element or `Vimeo.Player` instance.

| Parameter                                       | Type                                                                                                                 | Description                                                                                                                        | Default / Required            |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| `id`                                            | `string`                                                                                                             | A unique session UUID for each media element, used to identify and end tracking.                                                   | Required                      |
| `video`                                         | `HTMLIFrameElement` or `Player`                                                                                      | An `iframe` element, or an instance of `Vimeo.Player` created with the [Vimeo Player SDK](https://developer.vimeo.com/player/sdk). | Required                      |
| `label`                                         | `string`                                                                                                             | A custom human-readable label for the media element.                                                                               | `undefined`                   |
| `captureEvents`                                 | `VimeoEvent[]`                                                                                                       | A list of media events to track.                                                                                                   | All events tracked by default |
| `boundaries`                                    | `number[]`                                                                                                           | Percentage thresholds (0-100) to trigger progress events.                                                                          | `[10, 25, 50, 75]`            |
| `context`                                       | [`DynamicContext`](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#global-contexts-methods) | Dynamic contexts attached to each tracking event.                                                                                  | `undefined`                   |
| `updatePageActivityWhilePlaying`                | `boolean`                                                                                                            | Whether to update page activity while media is playing.                                                                            | `true`                        |
| `filterOutRepeatedEvents`                       | `FilterOutRepeatedEvents`                                                                                            | Whether to suppress consecutive identical events.                                                                                  | Enabled by default            |
| `filterOutRepeatedEvents` `.seekEvents`         | `boolean`                                                                                                            | Whether to filter out seek start and end events tracked after each other.                                                          | `true`                        |
| `filterOutRepeatedEvents` `.volumeChangeEvents` | `boolean`                                                                                                            | Whether to filter out volume change events tracked after each other.                                                               | `true`                        |
| `filterOutRepeatedEvents` `.flushTimeoutMs`     | `number`                                                                                                             | Timeout in milliseconds after which to send the events that are queued for filtering.                                              | `5000` (milliseconds)         |
| `pings.pingInterval`                            | `number`                                                                                                             | Interval (in seconds) for sending ping events.                                                                                     | `30` (seconds)                |
| `pings.maxPausedPings`                          | `number`                                                                                                             | Maximum number of ping events sent while playback is paused.                                                                       | `1`                           |
| `session`                                       | `boolean` or `object`                                                                                                | Whether to track the media session entity for playback statistics, or set a custom start time.                                     | `true`                        |

## Stop tracking

It's important to call `endVimeoTracking` when you finish tracking. This will end any recurring ping events, clear all listeners set by the Vimeo plugin, along with resetting statistics counters used by the Snowplow Media plugin.

## Events

The plugin provides automatic tracking of the following events. Note that some events are specific to this plugin, and not part of the standard Snowplow Media events.

| Vimeo Event Name                 | Description                                                                                        | Vimeo only |
| -------------------------------- | -------------------------------------------------------------------------------------------------- | ---------- |
| `Ready`                          | Sent when the media tracking is successfully attached to the player and can track events.          |            |
| `Play`                           | Sent when the player changes state to playing from previously being paused.                        |            |
| `Pause`                          | Sent when the user pauses the playback.                                                            |            |
| `End`                            | Sent when playback stops when end of the media is reached or because no further data is available. |            |
| `SeekStart`                      | Sent when a seek operation begins.                                                                 |            |
| `SeekEnd`                        | Sent when a seek operation completes.                                                              |            |
| `PlaybackRateChange`             | Sent when the playback rate has changed.                                                           |            |
| `VolumeChange`                   | Sent when the volume has changed.                                                                  |            |
| `FullscreenChange`               | Sent immediately after the browser switches into or out of full-screen mode.                       |            |
| `PictureInPictureChange`         | Sent immediately after the browser switches into or out of picture-in-picture mode.                |            |
| `BufferStart`                    | Sent when the player goes into the buffering state and begins to buffer content.                   |            |
| `BufferEnd`                      | Sent when the player finishes buffering content and resumes playback.                              |            |
| `QualityChange`                  | Sent when the video playback quality changes automatically.                                        |            |
| `Error`                          | Sent when the Vimeo player encounters an error                                                     |            |
| `Ping`                           | Fires periodically during playback                                                                 |            |
| `PercentProgress`                | Fires at progress milestones defined by `boundaries` option                                        |            |
| `CuePoint`                       | Sent when a cue point is reached.                                                                  | ✅          |
| `ChapterChange`                  | Sent when the chapter changes.                                                                     | ✅          |
| `TextTrackChange`                | Sent when the text track changes.                                                                  | ✅          |
| `InteractiveHotspotClicked`      | Sent when an interactive hotspot is clicked.                                                       | ✅          |
| `InteractiveOverlayPanelClicked` | Sent when an interactive overlay panel is clicked.                                                 | ✅          |

If you wish to track only a subset of these events, pass an array of `VimeoEvent`s to the `startVimeoTracking` function:

**JavaScript (tag):**

```javascript
window.snowplow('startVimeoTracking',  {
  id,
  video,
  captureEvents: ['play', 'pause'],
})
```

**Browser (npm):**

```javascript
import { VimeoEvent } from '@snowplow/browser-plugin-vimeo-tracking'

startVimeoTracking({
  id,
  video,
  captureEvents: [VimeoEvent.Play, VimeoEvent.Pause],
})
```

***

### Track events manually

For event types provided by the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/) but not automatically tracked by the Vimeo plugin, such as advertising events, you can use the corresponding `track*Event` functions from the Snowplow Media plugin.

**JavaScript (tag):**

```javascript
const id = 'XXXXX'; // randomly generated ID

const video = new Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

window.snowplow('startVimeoTracking', { id, video });

...

// When your ad break starts
window.snowplow('trackMediaAdBreakStart', { id });

// When your ad break ends
window.snowplow('trackMediaAdBreakEnd', { id });

...

window.snowplow('endVimeoTracking', id);;
```

**Browser (npm):**

```javascript
import { trackMediaAdBreakStart, trackMediaAdBreakEnd } from '@snowplow/browser-plugin-media';

const id = 'XXXXX'; // randomly generated ID

const video = new Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

startVimeoTracking({ id, video });

...

// When your ad break starts
trackMediaAdBreakStart({ id });

// When your ad break ends
trackMediaAdBreakEnd({ id });

...

endVimeoTracking(id);
```

***

## Vimeo-specific event schemas

The `CuePoint`, `ChapterChange`, `TextTrackChange`, `InteractiveHotspotClicked`, and `InteractiveOverlayPanelClicked` events are specific to the Vimeo plugin, and not provided by the Snowplow Media plugin.

They're not used by the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package, but you can use them for your own analysis.

### Cue point

Fired when a cue point added via `addCuePoint()` is reached during playback.

### `cue_point_event`

**Type:** Event

Schema for a Vimeo cue point event, fired when a cue point is reached in a video.

**Schema:** `iglu:com.vimeo/cue_point_event/jsonschema/1-0-0`

**Example:**

```json
{
  "id": "fb819c48-5760-4d94-9c5b-4fa52f61a998",
  "cuePointTime": 30.5,
  "data": {
    "label": "Chapter 1"
  }
}
```

**Properties:**

| Property                | Description                                                                  |
| ----------------------- | ---------------------------------------------------------------------------- |
| `id` _string_           | _Required._ The ID of the cue point.                                         |
| `cuePointTime` _number_ | _Required._ The location of the cue point in seconds.                        |
| `data` _object_         | _Optional._ The custom data from the addCuePoint() call, or an empty object. |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_vimeo_cue_point_event_1 cue_point_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'cue_point_event'
and events.event_vendor = 'com.vimeo'
```

### Chapter change

Fired when the current chapter changes during playback.

### `chapter_change_event`

**Type:** Event

Schema for a Vimeo chapter change event fired when the current chapter changes.

**Schema:** `iglu:com.vimeo/chapter_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "index": 2,
  "startTime": 120,
  "title": "Getting Started"
}
```

**Properties:**

| Property              | Description                                              |
| --------------------- | -------------------------------------------------------- |
| `index` _integer_     | _Optional._ The chapter number.                          |
| `startTime` _integer_ | _Optional._ The time in seconds when the chapter begins. |
| `title` _string_      | _Optional._ The chapter title.                           |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_vimeo_chapter_change_event_1 chapter_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'chapter_change_event'
and events.event_vendor = 'com.vimeo'
```

### Text track change

Fired when the active text track (captions or subtitles) changes.

### `text_track_change_event`

**Type:** Event

Schema for a Vimeo text track event, fired when the active text track of the captions or subtitle kind changes.

**Schema:** `iglu:com.vimeo/text_track_change_event/jsonschema/1-0-0`

**Example:**

```json
{
  "kind": "captions",
  "language": "en",
  "label": "English",
  "mode": "showing"
}
```

**Properties:**

| Property            | Description                                                                         |
| ------------------- | ----------------------------------------------------------------------------------- |
| `kind` _string_     | _Optional._ The kind of the text track: 'captions' or 'subtitles'.                  |
| `language` _string_ | _Optional._ The ISO code of the text track's language.                              |
| `label` _string_    | _Optional._ The human-readable label of the text track for identification purposes. |
| `mode` _string_     | _Optional._ The mode of the text track                                              |

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_vimeo_text_track_change_event_1 text_track_change_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'text_track_change_event'
and events.event_vendor = 'com.vimeo'
```

### Interactive hotspot clicked

Fired when a user clicks an interactive hotspot in a Vimeo interactive video. The interaction details are provided in the `interaction` entity.

### `interactive_hotspot_click_event`

**Type:** Event

Schema for a Vimeo interactive video hotspot click

**Schema:** `iglu:com.vimeo/interactive_hotspot_click_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_vimeo_interactive_hotspot_click_event_1 interactive_hotspot_click_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'interactive_hotspot_click_event'
and events.event_vendor = 'com.vimeo'
```

### Interactive overlay panel clicked

Fired when a user clicks an interactive overlay panel in a Vimeo interactive video. The interaction details are provided in the `interaction` entity.

### `interactive_overlay_panel_click_event`

**Type:** Event

Schema for a Vimeo interactive overlay panel click

**Schema:** `iglu:com.vimeo/interactive_overlay_panel_click_event/jsonschema/1-0-0`

**Properties:**

**Example warehouse query (Snowflake):**

```sql
select
unstruct_event_com_vimeo_interactive_overlay_panel_click_event_1 interactive_overlay_panel_click_event_1
from
atomic.events
where
events.collector_tstamp > getdate() - interval '1 hour'
and events.event = 'unstruct'
and events.event_name = 'interactive_overlay_panel_click_event'
and events.event_vendor = 'com.vimeo'
```

## Vimeo entities

The event and entity schemas are [the same as](/docs/events/ootb-data/media-events/) for the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/), plus the Vimeo-specific events detailed above.

Along with the standard Snowplow media entities, the Vimeo media plugin also tracks two Vimeo media-specific entities, `meta` and `interaction`. They're not used by the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package, but you can use them for your own analysis.

### Video metadata

The `meta` entity provides additional metadata about the video. It's added to the `Ready` event only.

### `meta`

**Type:** Entity

Schema for a Vimeo video metadata.

**Schema:** `iglu:com.vimeo/meta/jsonschema/1-0-0`

**Example:**

```json
{
  "videoId": 535907279,
  "videoTitle": "Introduction to Analytics",
  "videoUrl": "https://vimeo.com/535907279",
  "videoWidth": 1920,
  "videoHeight": 1080
}
```

**Properties:**

| Property               | Description                                                                                           |
| ---------------------- | ----------------------------------------------------------------------------------------------------- |
| `videoId` _number_     | _Required._ The Vimeo ID of the video.                                                                |
| `videoTitle` _string_  | _Required._ The title of the video.                                                                   |
| `videoUrl` _string_    | _Optional._ The URL of the video on vimeo.com.                                                        |
| `videoWidth` _number_  | _Required._ The native width of the video as the width of the video's highest available resolution.   |
| `videoHeight` _number_ | _Required._ The native height of the video as the height of the video's highest available resolution. |

### Interaction

The `interaction` entity provides details of user interactions for `InteractiveHotspotClicked` and `InteractiveOverlayPanelClicked` events.

### `interaction`

**Type:** Entity

Context Schema attached to interactive\_hotspot\_click\_event and interactive\_overlay\_panel\_click\_event.

**Schema:** `iglu:com.vimeo/interaction/jsonschema/1-0-0`

**Example:**

```json
{
  "action": "clickthrough",
  "actionPreference": {
    "pauseOnAction": true,
    "url": "https://example.com/product"
  },
  "hotspotId": "hs-product-link",
  "currentTime": 45.2
}
```

**Properties:**

| Property                     | Description                                                                                              |
| ---------------------------- | -------------------------------------------------------------------------------------------------------- |
| `action` _string_            | _Required._ The action type of the hotspot.                                                              |
| `actionPreference` _object_  | _Required._ The user's preferred action type for the hotspot.                                            |
| `customPayloadData` _object_ | _Optional._ The custom payload data of the interactive hotspot                                           |
| `currentTime` _number_       | _Optional._ The current time of the video when the interaction occurs is clicked.                        |
| `hotspotId` _string_         | _Optional._ The unique ID for the hotspot.                                                               |
| `panelId` _string_           | _Optional._ The unique ID for a panel within the overlay.                                                |
| `overlayId` _string_         | _Optional._ When action is overlay, the displayed unique ID for the overlay when the hotspot is clicked. |

## Advanced usage

As the Vimeo plugin uses Snowplow Media internally, for more granular control over events, you can utilise any of the functions provided by the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/).

For example, if you wish to include additional behavior when a video is paused, you can create callback on the `pause` event of an instance of a Vimeo player.

> **Note:** In the following example, ensure you aren't passing the `pause` event to the `startVimeoTracking` function, as this will result in the event being tracked twice.

**JavaScript (tag):**

```javascript
const id = 'XXXXX'; // randomly generated ID

const video = new Vimeo.Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

window.snowplow('startVimeoTracking', { id, video, captureEvents: ['play']});

video.on('pause', () => {
  // Do something when the video is paused
  window.snowplow('trackPauseEvent', { id });
});
```

**Browser (npm):**

```javascript
import { Player } from '@vimeo/player'
import { trackPauseEvent } from '@snowplow/browser-plugin-media'

const id = 'XXXXX'; // randomly generated ID

const video = new Player('vimeo-player', {
  videoId: 'zSM4ZyVe8xs'
});

startVimeoTracking({ id, video, captureEvents: [VimeoEvent.Play]});

video.on('pause', () => {
  // Do something when the video is paused
  trackPauseEvent({ id })
});
```

***

You can also provide `player` configuration when calling `startVimeoTracking`, to set options on the underlying Vimeo player instance.

---

# YouTube media tracking on web
> Automatically track YouTube video players embedded via iframe or YouTube IFrame API with playback events, quality changes, and buffer tracking.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/media/youtube/

This plugin enables the automatic tracking of an embedded YouTube iFrame video, using the [YouTube Player API](https://developers.google.com/youtube/iframe_api_reference).

It uses the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/) under the hood.

YouTube media events and entities are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                          |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                    |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-youtube-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-youtube-tracking@latest/dist/index.umd.min.js) (latest)               |

**Browser (npm):**

- `npm install @snowplow/browser-plugin-youtube-tracking`
- `yarn add @snowplow/browser-plugin-youtube-tracking`
- `pnpm add @snowplow/browser-plugin-youtube-tracking`

***

## Quick Start

The snippets below show how to get started with the plugin, after [setting up your tracker](/docs/sources/web-trackers/tracker-setup/).

Call `startYouTubeTracking` to begin tracking a YouTube video player, and `endYouTubeTracking` to stop tracking it.

**JavaScript (tag) - iFrame:**

```html
<!DOCTYPE html>
<html>
  <body>
    <iframe
      id="yt-player"
      src="https://www.youtube.com/embed/zSM4ZyVe8xs"
    ></iframe>

    <script>
      window.snowplow(
        'addPlugin',
        'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-youtube-tracking@latest/dist/index.umd.min.js',
        ['snowplowYouTubeTracking', 'YouTubeTrackingPlugin']
      );

      var mediaSessionId = crypto.randomUUID();

      window.snowplow('startYouTubeTracking', {
        id: mediaSessionId,
        video: 'yt-player' // or: document.getElementById('yt-player')
      });

      window.snowplow('endYouTubeTracking', mediaSessionId);
    </script>
  </body>
</html>
```

**Browser (npm) - iFrame:**

```html
<iframe
  id="yt-player"
  src="https://www.youtube.com/embed/zSM4ZyVe8xs"
></iframe>
```

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { YouTubeTrackingPlugin, startYouTubeTracking } from '@snowplow/browser-plugin-youtube-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ YouTubeTrackingPlugin() ],
});

const mediaSessionId = startYouTubeTracking({
  id: crypto.randomUUID(),
  video: 'yt-player' // or: document.getElementById('yt-player')
})

endYouTubeTracking(mediaSessionId);
```

**JavaScript (tag) - YT.Player:**

```html
<!DOCTYPE html>
<html>
  <body>
    <div id="yt-player"></div>

    <script>
      window.snowplow(
        'addPlugin',
        'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-youtube-tracking@latest/dist/index.umd.min.js',
        ['snowplowYouTubeTracking', 'YouTubeTrackingPlugin']
      );

      const player = new YT.Player('yt-player', {
        videoId: 'zSM4ZyVe8xs'
      });

      const mediaSessionId = crypto.randomUUID();

      window.snowplow('startYouTubeTracking', {
          id: mediaSessionId,
          video: player
      });

      window.snowplow('endYouTubeTracking', mediaSessionId);
    </script>
  </body>
</html>
```

**Browser (npm) - YT.Player:**

```html
<div id="yt-player"></div>
```

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { YouTubeTrackingPlugin, startYouTubeTracking } from '@snowplow/browser-plugin-youtube-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ YouTubeTrackingPlugin() ],
});

const player = new YT.Player('yt-player', {
  videoId: 'zSM4ZyVe8xs'
});

const mediaSessionId = startYouTubeTracking({
  id: crypto.randomUUID(),
  video: player
})

endYouTubeTracking(mediaSessionId);
```

***

## Start tracking

The `startYouTubeTracking` function is used to begin auto-tracking a YouTube Player instance. You must provide a unique `id` for the media session, along with the `video` ID.

It installs the iFrame API if necessary, adds event listeners and any poll intervals, and then calls [`startMediaTracking`](/docs/sources/web-trackers/tracking-events/media/snowplow/#usage) from the underlying Snowplow Media plugin.

The function returns the `mediaSessionId` used for the events that will be generated.

| Parameter                                       | Type                                                                                                                 | Description                                                                                                                                                                                                           | Default / Required            |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| `id`                                            | `string`                                                                                                             | A unique session UUID for each media element, used to identify and end tracking.                                                                                                                                      | Required                      |
| `video`                                         | `VideoLocation`                                                                                                      | A DOM ID for the iFrame element hosting the player, an iFrame element, or a pre-existing [YT.Player](https://developers.google.com/youtube/iframe_api_reference#Loading_a_Video_Player) instance to track events for. | Required                      |
| `label`                                         | `string`                                                                                                             | A custom human-readable label for the media element.                                                                                                                                                                  | `undefined`                   |
| `captureEvents`                                 | `CapturableEvents[]`                                                                                                 | A list of media events to track.                                                                                                                                                                                      | All events tracked by default |
| `boundaries`                                    | `number[]`                                                                                                           | Percentage thresholds (0-100) to trigger progress events.                                                                                                                                                             | `[10, 25, 50, 75]`            |
| `context`                                       | [`DynamicContext`](/docs/sources/web-trackers/custom-tracking-using-schemas/global-context/#global-contexts-methods) | Dynamic contexts attached to each tracking event.                                                                                                                                                                     | `undefined`                   |
| `updatePageActivityWhilePlaying`                | `boolean`                                                                                                            | Whether to update page activity while media is playing.                                                                                                                                                               | `true`                        |
| `filterOutRepeatedEvents`                       | `FilterOutRepeatedEvents`                                                                                            | Whether to suppress consecutive identical events.                                                                                                                                                                     | Enabled by default            |
| `filterOutRepeatedEvents` `.seekEvents`         | `boolean`                                                                                                            | Whether to filter out seek start and end events tracked after each other.                                                                                                                                             | `true`                        |
| `filterOutRepeatedEvents` `.volumeChangeEvents` | `boolean`                                                                                                            | Whether to filter out volume change events tracked after each other.                                                                                                                                                  | `true`                        |
| `filterOutRepeatedEvents` `.flushTimeoutMs`     | `number`                                                                                                             | Timeout in milliseconds after which to send the events that are queued for filtering.                                                                                                                                 | `5000` (milliseconds)         |
| `pings.pingInterval`                            | `number`                                                                                                             | Interval (in seconds) for sending ping events.                                                                                                                                                                        | `30` (seconds)                |
| `pings.maxPausedPings`                          | `number`                                                                                                             | Maximum number of ping events sent while playback is paused.                                                                                                                                                          | `1`                           |
| `session`                                       | `boolean` or `object`                                                                                                | Whether to track the media session entity for playback statistics, or set a custom start time.                                                                                                                        | `true`                        |

## Stop tracking

The `endYouTubeTracking` function disables auto tracking for the player registered with the provided session ID.

It will remove any event listeners and poll intervals, and call [`endMediaTracking`](/docs/sources/web-trackers/tracking-events/media/snowplow/#usage) from the core plugin.

## Events

Below is a table of all the core Snowplow Media plugin events that can be used in the `captureEvents` option:

| Name                   | Fire condition                                                   |
| ---------------------- | ---------------------------------------------------------------- |
| `ready`                | The video player has loaded                                      |
| `play`                 | The video is played                                              |
| `pause`                | The video is paused                                              |
| `end`                  | When playback stops at the end of the video                      |
| `seek_start`           | When seeking begins to jump to another position of the video     |
| `seek_end`             | When seeking ends and another position of the video is jumped to |
| `playback_rate_change` | Playback rate has changed                                        |
| `volume_change`        | Volume has changed                                               |
| `ping`                 | Fires periodically during playback                               |
| `percent_progress`     | Fires at progress milestones defined by `boundaries` option      |
| `buffer_start`         | Fires when playback pauses because content is not yet buffered   |
| `buffer_end`           | Fires when playback resumes after content has been fetched       |
| `quality_change`       | Playback quality has changed                                     |
| `error`                | An error occurs in the player                                    |

In addition, the following event names are also accepted for compatibility with earlier tracker versions, mapped to the equivalent event from above:

| Name                    | Fire condition                                                    |
| ----------------------- | ----------------------------------------------------------------- |
| `seek`                  | On seek                                                           |
| `volumechange`          | Volume has changed                                                |
| `ended`                 | When playback stops at the end of the video                       |
| `percentprogress`       | When a percentage boundary set in `options.boundaries` is reached |
| `playbackratechange`    | Playback rate has changed                                         |
| `playbackqualitychange` | Playback quality has changed                                      |

You can also use a pre-made event group names in `captureEvents`:

| Name            | Events                                                                                                                                               |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DefaultEvents` | `['ready', 'play', 'pause', 'ping', 'end', 'seek_start', 'seek_end', 'volume_change', 'percent_progress', 'playback_rate_change', 'quality_change']` |
| `AllEvents`     | Every supported event in the Events table                                                                                                            |

It's possible to extend an event group with any event in the Events table above. This could be useful if you want, for example, all the events contained in the `DefaultEvents` group, along with the `error` event. This is expressed in the following way:

**JavaScript (tag):**

```javascript
window.snowplow('startYouTubeTracking', {
  id: crypto.randomUUID(),
  video: "example-video",
  captureEvents: ["DefaultEvents", "error"],
})
```

**Browser (npm):**

```javascript
startYouTubeTracking({
  id: crypto.randomUUID(),
  video: 'example-video',
  captureEvents: ['DefaultEvents', 'error'],
})
```

***

### Unsupported events

The following events are defined by the core [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/) but aren't supported by the YouTube API, and so cannot be captured by this plugin:

| Name                        | Fire condition                                  |
| --------------------------- | ----------------------------------------------- |
| `fullscreen_change`         | Full screen state toggled                       |
| `picture_in_picture_change` | Picture-in-picture state toggled                |
| `ad_break_start`            | Beginning of an ad break                        |
| `ad_break_end`              | End of an ad break                              |
| `ad_start`                  | Beginning of an ad within an ad break           |
| `ad_first_quartile`         | 25% progress through an ad                      |
| `ad_midpoint`               | 50% progress through an ad                      |
| `ad_third_quartile`         | 75% progress through an ad                      |
| `ad_complete`               | 100% progress through an ad                     |
| `ad_skip`                   | User has opted to skip the ad before completion |
| `ad_click`                  | User has clicked the playing ad                 |
| `ad_pause`                  | User has paused the playing ad                  |
| `ad_resume`                 | User has resumed the paused ad                  |

### Add media entities to custom events

Use the `trackYouTubeSelfDescribingEvent` function to track custom event payloads that have the media-related entities attached as if they were generated by this or the core plugin directly.

**JavaScript (tag) - iFrame:**

```html
<!DOCTYPE html>
<html>
  <body>
    <iframe
      id="yt-player"
      src="https://www.youtube.com/embed/zSM4ZyVe8xs"
    ></iframe>

    <script>
      window.snowplow(
        'addPlugin',
        'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-youtube-tracking@latest/dist/index.umd.min.js',
        ['snowplowYouTubeTracking', 'YouTubeTrackingPlugin']
      );

      var mediaSessionId = crypto.randomUUID();

      window.snowplow('startYouTubeTracking', {
        id: mediaSessionId,
        video: 'yt-player' // or: document.getElementById('yt-player')
      });

      window.snowplow('trackYouTubeSelfDescribingEvent', {event: {schema: 'iglu:com_example/my_event/jsonschema/1-0-0', data: { video: true }}});

      window.snowplow('endYouTubeTracking', mediaSessionId);
    </script>
  </body>
</html>
```

**Browser (npm) - iFrame:**

```html
<iframe
  id="yt-player"
  src="https://www.youtube.com/embed/zSM4ZyVe8xs"
></iframe>
```

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { YouTubeTrackingPlugin, endYouTubeTracking, startYouTubeTracking } from '@snowplow/browser-plugin-youtube-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ YouTubeTrackingPlugin() ],
});

const mediaSessionId = startYouTubeTracking({
  id: crypto.randomUUID(),
  video: 'yt-player' // or: document.getElementById('yt-player')
});

trackYouTubeSelfDescribingEvent({event: {schema: 'iglu:com_example/my_event/jsonschema/1-0-0', data: { video: true }}});

endYouTubeTracking(mediaSessionId);
```

***

## YouTube player entity

The event and entity schemas are [the same as](/docs/events/ootb-data/media-events/) for the [Snowplow Media plugin](/docs/sources/web-trackers/tracking-events/media/snowplow/).

Along with the standard Snowplow media entities, the YouTube media plugin also adds a `youtube` entity to all media events.

If you've configured the [Media Player](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-media-player-data-model/) dbt package to use the [v1 schemas](/docs/events/ootb-data/media-events/#media-api-versions), or are using an older version of the Media Player model, it will use fields from the `youtube` entity. The Media Player model version 0.6+ doesn't use this entity.

### `youtube`

**Type:** Entity

Context Schema for a youtube player event

**Schema:** `iglu:com.youtube/youtube/jsonschema/1-0-0`

**Example:**

```json
{
  "autoPlay": false,
  "avaliablePlaybackRates": [
    0.25,
    0.5,
    0.75,
    1,
    1.25,
    1.5,
    1.75,
    2
  ],
  "buffering": false,
  "controls": true,
  "cued": false,
  "loaded": 17,
  "playbackQuality": "hd1080",
  "playerId": "example-id",
  "unstarted": false,
  "url": "https://www.youtube.com/watch?v=zSM4ZyVe8xs",
  "yaw": 0,
  "pitch": 0,
  "roll": 0,
  "fov": 100.00004285756798,
  "avaliableQualityLevels": [
    "hd2160",
    "hd1440",
    "hd1080",
    "hd720",
    "large",
    "medium",
    "small",
    "tiny",
    "auto"
  ]
}
```

**Properties:**

| Property                         | Description                                                                                                                                                        |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `avaliablePlaybackRates` _array_ | _Required._ An array of playback rates in which the current video is available                                                                                     |
| `avaliableQualityLevels` _array_ | _Optional._ An array of quality levels in which the current video is available                                                                                     |
| `cued` _boolean_                 | _Required._ If the video is cued                                                                                                                                   |
| `playerId` _string_              | _Required._ The HTML id of the video element                                                                                                                       |
| `autoPlay` _boolean_             | _Required._ This specifies whether the initial video will automatically start to play when the player loads.                                                       |
| `buffering` _boolean_            | _Required._ If the player is buffering                                                                                                                             |
| `controls` _boolean_             | _Required._ Whether the video player controls are displayed                                                                                                        |
| `error` _string_                 | _Optional._ A string of the latest error to occur, or null if no errors Must be one of: `INVALID_PARAMETER`, `HTML5_PLAYER_ERROR`, `NOT_FOUND`, `EMBED_DISALLOWED` |
| `loaded` _integer_               | _Required._ The percentage of the video that the player shows as buffered                                                                                          |
| `origin` _string_                | _Optional._ The origin domain of the embed                                                                                                                         |
| `playbackQuality` _string_       | _Required._ The quality level of the current video                                                                                                                 |
| `playlist` _array_               | _Optional._ An array of the video IDs in the playlist as they are currently ordered.                                                                               |
| `playlistIndex` _number_         | _Optional._ The index of the playlist video that is currently playing                                                                                              |
| `unstarted` _boolean_            | _Required._ If the player hasn't started                                                                                                                           |
| `url` _string_                   | _Required._ The YouTube embed URL of the media resource                                                                                                            |
| `fov` _number_                   | _Optional._ The field-of-view of the view in degrees, as measured along the longer edge of the viewport                                                            |
| `roll` _number_                  | _Optional._ The clockwise or counterclockwise rotational angle of the view in degrees                                                                              |
| `pitch` _number_                 | _Optional._ The vertical angle of the view in degrees                                                                                                              |
| `yaw` _number_                   | _Optional._ The horizontal angle of the view in degrees                                                                                                            |

## Legacy API

In addition to the above, the following methods are provided for compatibility with earlier versions of the plugin.

These APIs are deprecated in tracker version 4, and wrap the newer API methods above.

### Enable tracking

The `enableYouTubeTracking` function is similar to `startYouTubeTracking`, except:

- An `id` option is used to identify the player instead of `video`
- The additional options to `startMediaTracking` are nested within an `options` object
- The media session ID value will be generated automatically if not provided

> **Info:** The plugin's `id` option is equivalent to `video` in the new API and will accept:- The `id` of an `iframe` element
> - An `iframe` element directly
> - An existing instance of `YT.Player`, created with the [YouTube Iframe API](https://developers.google.com/youtube/iframe_api_reference)

The `enableYouTubeTracking` function takes the form:

**JavaScript (tag):**

```javascript
window.snowplow("enableYouTubeTracking", { id, options?: { label?, captureEvents?, boundaries?, updateRate? } })
```

**Browser (npm):**

```javascript
enableYouTubeTracking({ id, options?: { label?, captureEvents?, boundaries?, updateRate? } })
```

***

| Parameter               | Type                    | Default             | Description                                                                                                    | Required |
| ----------------------- | ----------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------- | -------- |
| `id`                    | `string` or `YT.Player` | -                   | The HTML id attribute of the media element                                                                     | Yes      |
| `options.label`         | `string`                | -                   | An identifiable custom label sent with the event                                                               | No       |
| `options.captureEvents` | `string[]`              | `['DefaultEvents']` | The events or Event Group to capture. For a full list of events and groups, check the [section below](#events) | No       |
| `options.boundaries`    | `number[]`              | `[10, 25, 50, 75]`  | The progress percentages to fire an event at (valid values 1 - 99 inclusive)                                   | No       |
| `options.*`             | `any`                   |                     | Any other options are passed through to `startMediaTracking`                                                   | No       |

Below is an example of the full `enableYouTubeTracking` function:

**JavaScript (tag):**

```javascript
window.snowplow('enableYouTubeTracking', {
  id: 'example-video',
  options: {
    label: 'My Custom Video Label',
    captureEvents: ['play', 'pause', 'ended'],
    boundaries: [20, 80],
    updateRate: 500,
  }
})
```

**Browser (npm):**

```javascript
enableYouTubeTracking({
  id: 'example-video',
  options: {
    label: 'My Custom Video Label',
    captureEvents: ['play', 'pause', 'ended'],
    boundaries: [20, 80],
    updateRate: 200,
  }
})
```

***

### Disable tracking

The `disableYouTubeTracking` function is an equivalent counterpart to `endYouTubeTracking` for `enableYouTubeTracking`.

Instead of requiring the session ID to be provided, it will remove the oldest of any sessions started with `enableYouTubeTracking` so the session ID doesn't need to be known and passed explicitly.

---

# Integrate Optimizely with the web trackers
> Automatically capture Optimizely X experiment and variation data as context entities on all tracked events for A/B testing analysis.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/optimizely/

The web tracker supports [Optimizely X](https://www.optimizely.com/) (Next Generation Platform) for campaign and A/B test management.

The Optimizely entity is **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                      |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-optimizely-x@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-optimizely-x@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-optimizely-x@latest/dist/index.umd.min.js",
  ["snowplowOptimizelyX", "OptimizelyXPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-optimizely-x`
- `yarn add @snowplow/browser-plugin-optimizely-x`
- `pnpm add @snowplow/browser-plugin-optimizely-x`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { OptimizelyXPlugin } from '@snowplow/browser-plugin-optimizely-x';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ OptimizelyXPlugin() ],
});
```

***

Check out the [third-party sources overview page](/docs/events/ootb-data/third-party-sources/) to see the entity schema.

If you're planning on leveraging the entity's `variationName`, you'll have to untick "Mask descriptive names in project code and third-party integrations" in the **OptimizelyX menu** > **Settings** > **Privacy**. Otherwise, all variation names will be null.

## Legacy plugin

The `browser-plugin-optimizely-x` plugin supersedes the earlier `browser-plugin-optimizely` plugin that was added in version 3.0. We deprecated and fully [removed it in version 4.0](/docs/sources/web-trackers/migration-guides/v3-to-v4-migration-guide/).

---

# Track page views on web
> Track page views with automatically captured URL, referrer, and title along with page view UUID for session analysis and single page apps.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/page-views/

Page view events are one of the most fundamental types of events to track on a website. They're tracked using `trackPageView()`.

Page view events must be **manually tracked**.

**JavaScript (tag):**

```javascript
snowplow('trackPageView');
```

**Browser (npm):**

```javascript
import { trackPageView } from '@snowplow/browser-tracker';

trackPageView();
```

***

This method automatically captures the URL, referrer URL, and page title. The first page view tracked uses `document.referrer` for the referrer URL, while for subsequent page views the referrer is the previous page URL.

It's possible to [override the URL and referrer URL](/docs/sources/web-trackers/tracking-events/#custom-page-url-and-referrer-url).

By default, the tracker infers the page title from the `<title>` tag. If you wish, you can also override the title with a custom value:

**JavaScript (tag):**

```javascript
snowplow('trackPageView', { title: 'my custom page title' });
```

**Browser (npm):**

```javascript
import { trackPageView } from '@snowplow/browser-tracker';

trackPageView({ title: 'my custom page title' });
```

***

## Page view ID and `web_page` entity

When the tracker loads on a page, or when `trackPageView()` is called, it generates a new page view UUID. This page view ID is attached to **all events** tracked on that page, as a [`web_page` entity](/docs/events/ootb-data/page-and-screen-view-events/#web-page-entity), until the next page view event is tracked. At that point, a new page view ID is generated and used for all subsequent events.

From version 3, the `web_page` entity is [enabled by default](/docs/sources/web-trackers/tracker-setup/initialization-options/). We advise you leave this enabled so you can use the [Snowplow Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) dbt package.

By default, the tracker lazily generates the page view ID when it's needed for the first time. For example:

1. The tracker is initialized when the page loads
2. `trackPageView()` is called for the first time
3. Tracker calls `getPageViewId()` to get the current page view ID
4. Since there is no page view ID yet, it generates a new one
5. The page view event is tracked with the generated page view ID
6. Subsequent events use the same page view ID

The next time `trackPageView()` is called, the tracker generates a new page view ID.

If you track an event before the first page view, the tracker will generate the page view ID at that point:

1. The tracker is initialized when the page loads
2. A custom event is tracked
3. Tracker calls `getPageViewId()` to get the current page view ID
4. Since there is no page view ID yet, it generates a new one
5. The custom event is tracked with the generated page view ID
6. Subsequent events, including the first page view, use the same page view ID

The second page view will generate a new page view ID as usual.

You can get the current page view ID using [the `getPageViewId` function](/docs/sources/web-trackers/tracking-events/#page-view-id).

### Change ID behavior for SPAs

In a single-page app (SPA), the tracker stays in memory as the user navigates between URLs. If you track events after navigating to a new URL but before calling `trackPageView()`, those events will use the existing page view ID from the previous page. The ID only resets when `trackPageView()` is called.

Use the `preservePageViewIdForUrl` configuration option to bind the page view ID generation to URL changes instead of page view events. The options are:

- `false` (default): URL changes don't trigger a new ID when reading `getPageViewId()`. Only `trackPageView()` triggers a new ID (on second+ call).
- `true` or `'full'`: generate a new ID when reading `getPageViewId()` if the full URL changed. `trackPageView()` still generates a new ID on second+ call even for the same URL.
- `'pathname'`: generate a new ID when reading `getPageViewId()` if pathname changed. Search params or fragment changes don't trigger a new ID.
- `'pathnameAndSearch'`: generate a new ID when reading `getPageViewId()` if pathname or search params changed. Fragment changes don't trigger a new ID.
- `preservePageViewId`: never regenerate the ID at all. Ignores `preservePageViewIdForUrl`.

You can set these options at initialization or during runtime:

**JavaScript (tag):**

```javascript
// At initialization
snowplow('newTracker', 'sp', 'collector.example.com', {
  appId: 'my-app',
  preservePageViewIdForUrl: 'pathname'
});

// At runtime
snowplow('preservePageViewIdForUrl', 'pathname');
```

**Browser (npm):**

```javascript
// At initialization
import { newTracker, preservePageViewIdForUrl } from '@snowplow/browser-tracker';

const tracker = newTracker('sp', 'collector.example.com', {
  appId: 'my-app',
  preservePageViewIdForUrl: 'pathname'
});

// At runtime
tracker.preservePageViewIdForUrl('pathname');
```

***

## Reset page activity on page view

By default, tracking a page view using `trackPageView()`resets [activity tracking](/docs/sources/web-trackers/tracking-events/activity-page-pings/).

Read more about this in the [activity tracking documentation](/docs/sources/web-trackers/tracking-events/activity-page-pings/#reset-page-pings-on-page-view).

## Add entities dynamically

As with all `trackX` methods, `trackPageView` can be passed an array of [custom entities](/docs/sources/web-trackers/custom-tracking-using-schemas/) as an additional parameter.

Additionally, you can add entities to page view and [page ping](/docs/sources/web-trackers/tracking-events/activity-page-pings/) events dynamically using the `contextCallback` option.

Pass it a function that returns an array of zero or more entities. The function will fire for the page view and for all subsequent [page pings](/docs/sources/web-trackers/tracking-events/activity-page-pings/) on the page. The returned entities will be added to the events.

For example:

**JavaScript (tag):**

```javascript
// Turn on page pings every 10 seconds
snowplow('enableActivityTracking', {
  minimumVisitLength: 10,
  heartbeatDelay: 10
});

snowplow('trackPageView', {
  // The usual array of static entities
  context: [{
    schema: 'iglu:com.acme/static_context/jsonschema/1-0-0',
    data: {
      staticValue: new Date().toString()
    }
  }],
  // Function which returns an array of custom entities
  // Gets called once per page view / page ping
  contextCallback: function() {
    return [{
      schema: 'iglu:com.acme/dynamic_context/jsonschema/1-0-0',
      data: {
        dynamicValue: new Date().toString()
      }
    }];
  }
});
```

**Browser (npm):**

```javascript
import {
  enableActivityTracking,
  trackPageView
} from '@snowplow/browser-tracker';

// Turn on page pings every 10 seconds
enableActivityTracking({
  minimumVisitLength: 10,
  heartbeatDelay: 10
});

trackPageView({
  // The usual array of static entities
  context: [{
    schema: 'iglu:com.acme/static_context/jsonschema/1-0-0',
    data: {
      staticValue: new Date().toString()
    }
  }],
  // Function which returns an array of custom entities
  // Gets called once per page view / page ping
  contextCallback: function() {
    return [{
      schema: 'iglu:com.acme/dynamic_context/jsonschema/1-0-0',
      data: {
        dynamicValue: new Date().toString()
      }
    }];
  }
});
```

***

In this example, the tracked page view and every subsequent page ping will have both a `static_context` and a `dynamic_context` attached. The `static_context` will all have the same `staticValue`, but the `dynamic_context` will have different `dynamicValue` values.

---

# Track Privacy Sandbox browser data with the web trackers
> Capture Privacy Sandbox data including Topics API information for privacy-preserving interest-based advertising and content personalization.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/privacy-sandbox/

> **Warning:** Privacy Sandbox has been deprecated by Google. This plugin remains available but the underlying browser API may no longer be supported.

The plugin allows for adding Privacy Sandbox [Topics API](https://developer.chrome.com/docs/privacy-sandbox/topics/overview/) data to your Snowplow tracking. To learn more about Privacy Sandbox visit the official [website](https://www.privacysandbox.com/).

> **Note:** Some of the APIs and data will not be available by default in all users. This is commonly due to these APIs being dependent on browser support, user privacy preferences, browser feature-flags or ad-blocking software. The plugin will not modify or request access explicitly to any of these features if not available by default.

> **Note:** The plugin is available since version 3.14 of the tracker.

The Privacy Sandbox entity is **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                         |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                   |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-privacy-sandbox@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-privacy-sandbox@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-privacy-sandbox@latest/dist/index.umd.min.js",
  ["snowplowPrivacySandbox", "PrivacySandboxPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-privacy-sandbox`
- `yarn add @snowplow/browser-plugin-privacy-sandbox`
- `pnpm add @snowplow/browser-plugin-privacy-sandbox`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { PrivacySandboxPlugin } from '@snowplow/browser-plugin-privacy-sandbox';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ PrivacySandboxPlugin() ],
});
```

***

## Context entity

Adding this plugin will automatically capture [this](https://github.com/snowplow/iglu-central/blob/master/schemas/com.google.privacysandbox/topics/jsonschema/1-0-0) entity.

---

# Track screen views on web
> Track screen views with mobile-style tracking on web including screen engagement metrics for time spent and scroll depth analysis.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/screen-views/

Screen view tracking is the recommended way to track users opening a screen in mobile apps. They are the default option for tracking views in our mobile trackers. Check out the [screen tracking overview page](/docs/events/ootb-data/page-and-screen-view-events/) for more details and schemas.

On web, [we recommend using page views](/docs/sources/web-trackers/tracking-events/page-views/) to track users visiting a page. However, using the screen view tracking plugin is also an option on web.

> **Note:** The plugin is available from Version 4.2 of the tracker.

Screen tracking events must be **manually tracked**. The plugin will add the relevant entities automatically.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                         |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                   |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-screen-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-screen-tracking@latest/dist/index.umd.min.js) (latest)               |

**Browser (npm):**

- `npm install @snowplow/browser-plugin-screen-tracking`
- `yarn add @snowplow/browser-plugin-screen-tracking`
- `pnpm add @snowplow/browser-plugin-screen-tracking`

***

In order to make use of the plugin, you will need to register it with the tracker:

**JavaScript (tag):**

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-screen-tracking@latest/dist/index.umd.min.js',
    ['snowplowScreenTracking', 'ScreenTrackingPlugin']
);
```

**Browser (npm):**

```javascript
import { ScreenTrackingPlugin } from '@snowplow/browser-plugin-screen-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [
    ScreenTrackingPlugin()
   ],
});
```

***

## Track a screen view event

To track a [screen view event](/docs/events/ootb-data/page-and-screen-view-events/#screen-views), use the `trackScreenView` function.

**JavaScript (tag):**

This example shows the required properties only.

```javascript
window.snowplow(
  'trackScreenView',
  {
    name: 'my-screen-name',
    id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e', // generated automatically if not provided
  }
);
```

**Browser (npm):**

This example shows the required properties only.

```javascript
import { trackScreenView } from '@snowplow/browser-plugin-screen-tracking';

trackScreenView({
  name: 'my-screen-name',
  id: '5d79770b-015b-4af8-8c91-b2ed6faf4b1e', // generated automatically if not provided
});
```

***

## Screen entity

By default, the tracker will automatically attach [a `screen` entity](/docs/events/ootb-data/page-and-screen-view-events/#screen-entity) to **all** events tracked.

However, if you haven't tracked any screen views, no `screen` entity will be attached. The `screen` entity contains information about the last screen viewed by the user, based on the last `trackScreenView` call.

**JavaScript (tag):**

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-screen-tracking@latest/dist/index.umd.min.js',
    ['snowplowScreenTracking', 'ScreenTrackingPlugin'],
    [
      {
        screenContext: true, // enabled by default
      }
    ]
);
```

**Browser (npm):**

```javascript
import { ScreenTrackingPlugin } from '@snowplow/browser-plugin-screen-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [
    ScreenTrackingPlugin({
      screenContext: true, // enabled by default
    })
   ],
});
```

***

## Screen engagement tracking

By default, the screen tracking plugin enables [screen engagement tracking](/docs/events/ootb-data/page-activity-tracking/#screen-engagement).

The tracker will automatically track a screen end event, with `screen_summary` entity, just before tracking a new screen view event.

Because this feature was designed for mobile platforms, not all the functionality is applicable to web. The page doesn't distinguish between foreground and background, so the `foreground_sec` time in the `screen_summary` entity will track the total time spent on the screen. The `background_sec` field will always be `null`.

**JavaScript (tag):**

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-screen-tracking@latest/dist/index.umd.min.js',
    ['snowplowScreenTracking', 'ScreenTrackingPlugin'],
    [
      {
        screenEngagementAutotracking: true, // enabled by default
      }
    ]
);
```

**Browser (npm):**

```javascript
import { ScreenTrackingPlugin } from '@snowplow/browser-plugin-screen-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [
    ScreenTrackingPlugin({
      screenEngagementAutotracking: true, // enabled by default
    })
   ],
});
```

***

### Updating list item view and scroll depth information

To update the [list item viewed and scroll depth information](/docs/events/ootb-data/page-activity-tracking/#screen-engagement) tracked in the `screen_summary` entity, track the `ListItemView` and `ScrollChanged` events with this information.

> **Note:** We designed the screen summary entity to work with mobile platforms. On web, you can track scroll offsets automatically using [page pings](/docs/sources/web-trackers/tracking-events/activity-page-pings/).
>
> For fine-grained tracking of page element visibility, consider using the [element visibility tracking plugin](/docs/sources/web-trackers/tracking-events/element-tracking/) instead.

When tracked and `screenEngagementAutotracking` is enabled, the tracker won't send these events to the Collector, but will process the information into the next screen summary entity. This means that tracking these events by themselves has no effect if you don't also track screen views.

If you've set `screenEngagementAutotracking: false`, the list item view and scroll depth events are treated as regular events and sent to the Collector.

You may want to track the events every time a new list item is viewed on the screen, or whenever the scroll position changes.

To update the list items viewed information:

**JavaScript (tag):**

```javascript
window.snowplow(
  'trackListItemView',
  {
    index: 1,
    itemsCount: 10,
  }
);
```

**Browser (npm):**

```javascript
import { trackListItemView } from '@snowplow/browser-plugin-screen-tracking';

trackListItemView({
  index: 1,
  itemsCount: 10,
});
```

***

To update the scroll depth information:

**JavaScript (tag):**

```javascript
window.snowplow(
  'trackScrollChanged',
  {
    yOffset: 10,
    xOffset: 20,
    viewHeight: 100,
    viewWidth: 200,
    contentHeight: 300,
    contentWidth: 400,
  }
);
```

**Browser (npm):**

```javascript
import { trackScrollChanged } from '@snowplow/browser-plugin-screen-tracking';

trackScrollChanged({
  yOffset: 10,
  xOffset: 20,
  viewHeight: 100,
  viewWidth: 200,
  contentHeight: 300,
  contentWidth: 400,
});
```

***

---

# Track sessions on web
> Automatically track session information including session IDs, event indexes, and references to previous sessions with configurable session timeouts.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/session/

The web tracker can add a context entity to events with information about the current session. The context entity repeats some of the session information stored in canonical event properties (e.g. `domain_userid`, `domain_sessionid`), but also adds new information.

It adds a reference to the previous session (`previousSessionId`) and first event in the current session (`firstEventId`, `firstEventTimestamp`). It also adds an index of the event in the session, useful for ordering events as they were tracked (`eventIndex`).

> **Note:** Because session tracking requires the session cookie, no session context entities will be added to events if anonymous tracking is enabled.

The default session expiry time is 30 minutes.

## Tracking the session entity

The session entity is **automatically tracked** once [configured](/docs/sources/web-trackers/tracker-setup/initialization-options/).

**Client session entity properties**

The [client\_session](https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2/) context entity consists of the following properties:

| Attribute             | Description                                                                                                   | Required? |
| --------------------- | ------------------------------------------------------------------------------------------------------------- | --------- |
| `userId`              | An identifier for the user of the session (same as `domain_userid`).                                          | Yes       |
| `sessionId`           | An identifier (UUID) for the session (same as `domain_sessionid`).                                            | Yes       |
| `sessionIndex`        | The index of the current session for this user (same as `domain_sessionidx`).                                 | Yes       |
| `eventIndex`          | Optional index of the current event in the session. Signifies the order of events in which they were tracked. | No        |
| `previousSessionId`   | The previous session identifier (UUID) for this user.                                                         | No        |
| `storageMechanism`    | The mechanism that the session information has been stored on the device.                                     | Yes       |
| `firstEventId`        | The optional identifier (UUID) of the first event id for this session.                                        | No        |
| `firstEventTimestamp` | Optional date-time timestamp of when the first event in the session was tracked.                              | No        |

> **Note:** Please note that the session context entity is only available since version 3.5 of the tracker.

## Starting a new session

The tracker automatically tracks session information and implements session timeouts after which the session is reset.

However, in case you want to trigger a new session manually (e.g., after logging out a user), you can use the `newSession` call to do that.

This will expire the current session and start a new session.

**JavaScript (tag):**

```javascript
snowplow('newSession');
```

**Browser (npm):**

```javascript
import { newSession } from '@snowplow/browser-tracker';

newSession();
```

***

## On session update callback

The `onSessionUpdateCallback` option, allows you to supply a callback function to be executed whenever a new session is generated on the tracker.

The callback's signature is: `(clientSession: ClientSession) => void` where clientSession includes the same values as you would expect on the [`client_session`](http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-2) context.

_Note:_ The callback is **not** called whenever a session is expired, but only when a new one is generated.

> **Note:** Please note that the session update callback is only available since version 3.11 of the tracker.

## Session cookie duration

Whenever an event fires, the tracker creates a session cookie. If the cookie didn’t previously exist, the tracker interprets this as the start of a new session.

By default the session cookie expires after 30 minutes. This means that a user leaving the site and returning in under 30 minutes does not change the session. You can override this default by setting `sessionCookieTimeout` to a duration (in seconds) in the initial tracker configuration object. For example,

```json
{
  ...
  sessionCookieTimeout: 3600
  ...
}
```

would set the session cookie lifespan to an hour.

---

# Track site search on web
> Track internal site search queries with search terms, filters, and result counts to analyze user search behavior and content discovery.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/site-search/

Site search tracking are provided as part of the Site tracking plugin. This plugin also provides events for [social media interactions](/docs/sources/web-trackers/tracking-events/social-media/) and [timings](/docs/sources/web-trackers/tracking-events/timings/generic/).

Site search events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                       |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                 |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js",
  ["snowplowSiteTracking", "SiteTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-site-tracking`
- `yarn add @snowplow/browser-plugin-site-tracking`
- `pnpm add @snowplow/browser-plugin-site-tracking`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { SiteTrackingPlugin, trackSiteSearch } from '@snowplow/browser-plugin-site-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SiteTrackingPlugin() ],
});
```

***

## Event

Use the `trackSiteSearch` method to track users searching your website. Here are its arguments:

| **Name**       | **Required?** | **Description**                 | **Type** |
| -------------- | ------------- | ------------------------------- | -------- |
| `terms`        | Yes           | Search terms                    | array    |
| `filters`      | No            | Search filters                  | JSON     |
| `totalResults` | No            | Results found                   | number   |
| `pageResults`  | No            | Results displayed on first page | number   |

An example:

**JavaScript (tag):**

```javascript
snowplow('trackSiteSearch', {
    terms: ['unified', 'log'],
    filters: {'category': 'books', 'sub-category': 'non-fiction'},
    totalResults: 14,
    pageResults: 8
});
```

**Browser (npm):**

```javascript
import { trackSiteSearch } from '@snowplow/browser-plugin-site-tracking';

trackSiteSearch({
    terms: ['unified', 'log'],
    filters: {'category': 'books', 'sub-category': 'non-fiction'},
    totalResults: 14,
    pageResults: 8
});
```

***

Site search events are implemented as Snowplow self-describing events. [Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/site_search/jsonschema/1-0-0) is the schema for a `site_search` event.

---

# Track social media interactions with the web trackers
> Track user interactions with social media widgets including likes, shares, and retweets across Facebook, Twitter, and other platforms.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/social-media/

Social interaction tracking is provided as part of the Site tracking plugin. This plugin also provides events for [site search](/docs/sources/web-trackers/tracking-events/site-search/) and [timings](/docs/sources/web-trackers/tracking-events/timings/generic/).

Social media interaction events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                       |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                 |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js",
  ["snowplowSiteTracking", "SiteTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-site-tracking`
- `yarn add @snowplow/browser-plugin-site-tracking`
- `pnpm add @snowplow/browser-plugin-site-tracking`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { SiteTrackingPlugin, trackSiteSearch } from '@snowplow/browser-plugin-site-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SiteTrackingPlugin() ],
});
```

***

## Event

Social tracking is used to track the way users interact with Facebook, Twitter and Google + widgets, e.g. to capture "like this" or "tweet this" events.

The `trackSocialInteraction` method takes three parameters:

| **Parameter** | **Description**                                               | **Required?** | **Example value**     |
| ------------- | ------------------------------------------------------------- | ------------- | --------------------- |
| `action`      | Social action performed                                       | Yes           | 'like', 'retweet'     |
| `network`     | Social network                                                | Yes           | 'facebook', 'twitter' |
| `target`      | Object social action is performed on e.g. page ID, product ID | No            | '19.99'               |

The method is executed in as:

**JavaScript (tag):**

```javascript
snowplow('trackSocialInteraction', {
  action: string,
  network: string,
  target: string
});
```

For example:

```javascript
snowplow('trackSocialInteraction', {
  action: 'like',
  network: 'facebook',
  target: 'pbz00123'
});
```

**Browser (npm):**

```javascript
import { trackSocialInteraction } from '@snowplow/browser-plugin-site-tracking';

trackSocialInteraction({
  action: string,
  network: string,
  target: string
});
```

For example:

```javascript
import { trackSocialInteraction } from '@snowplow/browser-plugin-site-tracking';

trackSocialInteraction({
  action: 'like',
  network: 'facebook',
  target: 'pbz00123'
});
```

***

---

# Track timezone and geolocation on web
> Capture user timezone information and geolocation coordinates with accuracy data through browser APIs for location-based analysis.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/timezone-geolocation/

Track users' timezone and geolocation with these configuration options.

## Timezone

Since version 4 of the JavaScript tracker, the tracker automatically captures the user's timezone. It populates the `os_timezone` [atomic field](/docs/fundamentals/canonical-event/). This feature uses the `Intl.DateTimeFormat` function in modern browsers.

If you're using an older version of the tracker, or your users use older browsers, use the timezone plugin to capture timezone information. It uses the `jstimezonedetect` library to determine the user's timezone and populate the `os_timezone` field.

The timezone property is **automatically tracked**.

### Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                  |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)            |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-timezone@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-timezone@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-timezone@latest/dist/index.umd.min.js",
  ["snowplowTimezone", "TimezonePlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-timezone`
- `yarn add @snowplow/browser-plugin-timezone`
- `pnpm add @snowplow/browser-plugin-timezone`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { TimezonePlugin } from '@snowplow/browser-plugin-timezone';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ TimezonePlugin() ],
});
```

***

Once configured, all subsequent events will contain this property.

## Geolocation

If this plugin is enabled, the tracker will attempt to create a entity from the visitor's geolocation information.

If the visitor hasn't already given or denied the website permission to use their geolocation information, a prompt will appear. If they give permission, then all events from that moment on will include their geolocation information, as a context entity.

If the geolocation entity isn't enabled at tracker initialization, you can enable it at a later time by calling `enableGeolocationContext`. This is useful if you have other areas of your site where you require requesting geolocation access, as you can defer enabling this on your Snowplow events until you have permission to read the users geolocation for your other use case.

For more information on the geolocation API, see [the specification](http://dev.w3.org/geo/api/spec-source.html).

Check out the [geolocation tracking overview](/docs/events/ootb-data/geolocation/) for the entity schema.

The geolocation entity is **automatically tracked** once configured.

### Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

This plugin was included in `sp.js` in version 3, but removed from the default bundle in version 4.

**Download:**

|                                             |                                                                                                                                                                                                                   |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                                                                                                             |
| Available on jsDelivr                       | [jsDeliv](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-geolocation@latest/dist/index.umd.min.js)[r](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-consent@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-geolocation@latest/dist/index.umd.min.js) (latest)                                                                                                             |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

To enable after initialization:

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-geolocation@latest/dist/index.umd.min.js",
  ["snowplowGeolocation", "GeolocationPlugin"],
);

// Enable when appropriate e.g. after user consents
// This prompts the browser for location permission
window.snowplow('enableGeolocationContext');
```

To enable at initialization:

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-geolocation@latest/dist/index.umd.min.js",
  ["snowplowGeolocation", "GeolocationPlugin"],
  [true] // Prompts for permission immediately
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-geolocation`
- `yarn add @snowplow/browser-plugin-geolocation`
- `pnpm add @snowplow/browser-plugin-geolocation`

To enable after initialization:

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { GeolocationPlugin, enableGeolocationContext } from '@snowplow/browser-plugin-geolocation';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ GeolocationPlugin() ], // Inactive by default
});

// Enable when appropriate e.g. after user consents
// This prompts the browser for location permission
enableGeolocationContext();
```

To enable at initialization:

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { GeolocationPlugin } from '@snowplow/browser-plugin-geolocation';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ GeolocationPlugin(true) ], // Prompts for permission immediately
});
```

***

> **Note:** There's no API to turn off geolocation tracking once enabled.

If you have multiple tracker instances on your site, you can choose to enable geolocation tracking on a per-tracker basis. Provide a list of tracker namespaces to the `enableGeolocationContext` function.

**JavaScript (tag):**

```javascript
window.snowplow('enableGeolocationContext', ['sp1']);
```

**Browser (npm):**

```javascript
enableGeolocationContext(['sp1']);
```

***

---

# Track generic site timings on web
> Track custom timing events to measure how long resources take to load or user actions take to complete with category and label organization.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/timings/generic/

Use the `trackTiming` method to track user timing events such as how long resources take to load. This method is provided as part of the `site-tracking` plugin.

Timing events must be **manually tracked**.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                       |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                 |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-site-tracking@latest/dist/index.umd.min.js",
  ["snowplowSiteTracking", "SiteTrackingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-site-tracking`
- `yarn add @snowplow/browser-plugin-site-tracking`
- `pnpm add @snowplow/browser-plugin-site-tracking`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { SiteTrackingPlugin, trackSiteSearch } from '@snowplow/browser-plugin-site-tracking';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ SiteTrackingPlugin() ],
});
```

***

## Timing event

Use the `trackTiming` method to track how long something took. Here are its arguments:

| **Name**   | **Required?** | **Description**                | **Type** |
| ---------- | ------------- | ------------------------------ | -------- |
| `category` | Yes           | Timing category                | string   |
| `variable` | Yes           | Timed variable                 | string   |
| `timing`   | Yes           | Number of milliseconds elapsed | number   |
| `label`    | No            | Label for the event            | string   |

An example:

**JavaScript (tag):**

```javascript
snowplow('trackTiming', {
  category: 'load',
  variable: 'map_loaded',
  timing: 50,
  label: 'Map loading time'
});
```

**Browser (npm):**

```javascript
import { trackTiming } from '@snowplow/browser-plugin-site-tracking';

trackTiming({
  category: 'load',
  variable: 'map_loaded',
  timing: 50,
  label: 'Map loading time'
});
```

***

Timing events are implemented as Snowplow self describing events. [Here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/timing/jsonschema/1-0-0) is the schema for a `timing` event.

---

# Track performance navigation timings on web
> Track page load performance metrics using the PerformanceNavigationTiming API including DNS lookup, connection times, and resource loading duration.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/timings/

To add performance timing context entities to your Snowplow tracking, use this Performance navigation timing plugin.

By default all its metrics are relative to the page load rather than absolute time stamps, making it easy to analyze and aggregate. To learn more about the properties tracked, you can visit the [specification](https://www.w3.org/TR/navigation-timing-2/) or MDN [documentation site](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming).

The following diagram shows the ResourceTiming and PerformanceNavigationTiming properties and how they connect to the navigation of the page main document.

![performance navigation timeline](/assets/images/performance_navigation_timeline-2abc2f691ceff27976a60f6e68678be2.png)

_Performance navigation timeline from the [W3C specification](https://www.w3.org/TR/navigation-timing-2/)._

> **Note:** The plugin is available since version 3.10 of the tracker.

Adding this plugin will automatically capture [this](https://github.com/snowplow/iglu-central/blob/master/schemas/org.w3/PerformanceNavigationTiming/jsonschema/1-0-0) context entity.

Performance navigation timing context entities are **automatically tracked** once configured.

You can also create and track general timing events using the [site tracking plugin](/docs/sources/web-trackers/tracking-events/timings/generic/).

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                                       |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                                 |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-performance-navigation-timing@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-performance-navigation-timing@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-performance-navigation-timing@latest/dist/index.umd.min.js",
  ["snowplowPerformanceNavigationTiming", "PerformanceNavigationTimingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-performance-navigation-timing`
- `yarn add @snowplow/browser-plugin-performance-navigation-timing`
- `pnpm add @snowplow/browser-plugin-performance-navigation-timing`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { PerformanceNavigationTimingPlugin } from '@snowplow/browser-plugin-performance-navigation-timing';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ PerformanceNavigationTimingPlugin() ],
});
```

***

## More detailed analysis for Single Page Applications (SPA)

As these metrics are primarily related to the initial page serve and load, after the window\.onload handler ends the metrics will likely stay static for the life of the SPA page. If there's a pattern to the API requests the SPA makes for new content, the [PerformanceObserver](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceObserver) could be used to capture the network metrics for that request from the performance API and re-use the schema.

For actual rendering performance, the application will have to use the PerformanceMark/PerformanceMeasure [User timing](https://developer.mozilla.org/en-US/docs/Web/API/Performance_API/User_timing) APIs. These allow custom timing milestones, but since it's completely custom and there are no common conventions for using these APIs, currently there is no automatic support in the tracker for tracking these. Therefore we encourage you to build your custom schema in case you believe you would benefit from these additional metrics.

## Performance timing plugin (original)

> **Warning:** This plugin has been deprecated and superseded by the Performance Navigation Timing plugin described above.

This older plugin has been superseded by the Performance Navigation Timing plugin, which has a newer API and additional metrics such as the compressed/decompressed page size, and information about the navigation that can contextualise cache usage that can impact the measured metrics, as well as server-side metrics, etc.

If this context entity is enabled, the JavaScript Tracker will use the create a context JSON from the `window.performance.timing` object, along with the Chrome `firstPaintTime` field (renamed to `chromeFirstPaint`) if it exists. This data can be used to calculate page performance metrics.

Note that if you fire a page view event as soon as the page loads, the `domComplete`, `loadEventStart`, `loadEventEnd`, and `chromeFirstPaint` metrics in the Navigation Timing API may be set to zero. This is because those properties are only known once all scripts on the page have finished executing.

**Details**

Advanced PerformanceTiming usage

The `domComplete`, `loadEventStart`, and `loadEventEnd` metrics in the NavigationTiming API are set to 0 until after every script on the page has finished executing, including sp.js. This means that the corresponding fields in the PerformanceTiming reported by the tracker will be 0.

To get around this limitation, you can wrap all Snowplow code in a `setTimeout` call:

```javascript
setTimeout(function () {

// Load Snowplow and call tracking methods here

}, 0);
```

This delays its execution until after those NavigationTiming fields are set.

Additionally the `redirectStart`, `redirectEnd`, and `secureConnectionStart` are set to 0 if there is no redirect or a secure connection is not requested.

For more information on the Navigation Timing API, see [the specification](http://www.w3.org/TR/2012/REC-navigation-timing-20121217/#sec-window.performance-attribute).

Performance timing context entities are **automatically tracked** once configured. The schema is [here](https://github.com/snowplow/iglu-central/blob/master/schemas/org.w3/PerformanceTiming/jsonschema/1-0-0).

### Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

This plugin was included in `sp.js` in version 3, but removed in version 4 in favor of the performance navigation timing plugin.

**Download:**

|                                             |                                                                                                                            |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)                      |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-performance-timing@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-performance-timing@latest/dist/index.umd.min.js) (latest)               |

**Note:** The links to the CDNs above point to the current latest version. You should pin to a specific version when integrating this plugin on your website if you are using a third party CDN in production.

```javascript
window.snowplow('addPlugin',
  "https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-performance-timing@latest/dist/index.umd.min.js",
  ["snowplowPerformanceTiming", "PerformanceTimingPlugin"]
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-performance-timing`
- `yarn add @snowplow/browser-plugin-performance-timing`
- `pnpm add @snowplow/browser-plugin-performance-timing`

```javascript
import { newTracker, trackPageView } from '@snowplow/browser-tracker';
import { PerformanceTimingPlugin } from '@snowplow/browser-plugin-performance-timing';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ PerformanceTimingPlugin() ],
});
```

***

---

# Track core web vitals performance metrics with the web trackers
> Automatically track Core Web Vitals metrics including LCP, FID, and CLS using the web-vitals library for user experience analysis.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/web-vitals/

The plugin adds the capability to track web performance metrics categorized as [Web Vitals](https://web.dev/vitals/). These metrics are tracked with an event based on the [web\_vitals schema](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/web_vitals/jsonschema/). To make sure it collects the most complete values for the web vital metrics, the plugin uses a set of browser APIs to detect and send the event as the visitor is leaving the current page in the browser.

To collect the web vitals data, the plugin loads the [web-vitals](https://github.com/GoogleChrome/web-vitals) open source library dynamically on your page.

> **Note:** The plugin is available since version 3.13 of the tracker.

Web vitals events are **automatically tracked** once configured.

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ✅        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                    |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Download from GitHub Releases (Recommended) | [Github Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)              |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-web-vitals@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-web-vitals@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-web-vitals@latest/dist/index.umd.min.js',
    ['snowplowWebVitals', 'WebVitalsPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-web-vitals`
- `yarn add @snowplow/browser-plugin-web-vitals`
- `pnpm add @snowplow/browser-plugin-web-vitals`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { WebVitalsPlugin } from '@snowplow/browser-plugin-web-vitals';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ WebVitalsPlugin() ],
});
```

***

## Configuration

The Web Vitals plugin can be initialized with a couple of options allowing for customizing its behavior:

| Option                | Type             | Description                                                                                                                                                                        | Default value                                            |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `loadWebVitalsScript` | `boolean`        | Should the plugin immediately load the Core Web Vitals measurement script from UNPKG CDN.                                                                                          | `true`                                                   |
| `webVitalsSource`     | `string`         | The URL endpoint the Web Vitals script should be loaded from. Defaults to the UNPKG CDN.                                                                                           | `https://unpkg.com/web-vitals@4/dist/web-vitals.iife.js` |
| `context`             | `DynamicContext` | Context entities to add to the tracked event. Can be provided either as an array of self-describing JSONs or function that returns a context entity. (available from version 3.19) |                                                          |

**JavaScript (tag):**

Pass configuration options as an array in the fourth parameter of `addPlugin`:

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-web-vitals@latest/dist/index.umd.min.js',
    ['snowplowWebVitals', 'WebVitalsPlugin'],
    [{
        loadWebVitalsScript: true,
        webVitalsSource: 'https://cdn.jsdelivr.net/npm/web-vitals@3/dist/web-vitals.iife.js',
        context: [
            function(vitals) {
                return {
                    schema: 'iglu:com.example/page_performance/jsonschema/1-0-0',
                    data: { lcpRating: vitals.lcp < 2500 ? 'good' : 'needs-improvement' }
                };
            }
        ]
    }]
);
```

**Browser (npm):**

Pass configuration options directly to the plugin function:

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { WebVitalsPlugin } from '@snowplow/browser-plugin-web-vitals';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [
       WebVitalsPlugin({
           loadWebVitalsScript: true,
           webVitalsSource: 'https://cdn.jsdelivr.net/npm/web-vitals@3/dist/web-vitals.iife.js',
           context: [
               function(vitals) {
                   return {
                       schema: 'iglu:com.example/page_performance/jsonschema/1-0-0',
                       data: { lcpRating: vitals.lcp < 2500 ? 'good' : 'needs-improvement' }
                   };
               }
           ]
       })
   ],
});
```

***

### Using an already existing Web Vitals library source

There could be cases where your page or one of the loaded JavaScript bundles already include the [web-vitals](https://github.com/GoogleChrome/web-vitals) library. In those cases, there is no need to load it an additional time from the plugin. If this is the case, you have to make sure that the library APIs are exposed in the `window` object properly as well.

### Choosing a Web Vitals measurement source

The default Web Vitals measurement script is loaded from the [UNPKG](https://www.unpkg.com/) CDN. This choice is chosen as a default but you should consider your own setup when choosing the script source. Selecting a script source from a CDN which might already be used in your website might save you from yet another connection startup time (_Queueing_,_DNS lookup_,_TCP_, _SSL_).

Another reasonable choice could be [jsDelivr](https://cdn.jsdelivr.net/npm/web-vitals@3/dist/web-vitals.iife.js).

---

# Track WebViews and hybrid applications using the web trackers
> Integrate web tracking in hybrid mobile apps by forwarding events from web views to native iOS, Android, or React Native trackers.
> Source: https://docs.snowplow.io/docs/sources/web-trackers/tracking-events/webview/

This plugin provides integration for hybrid apps using the Snowplow native mobile ([Android and iOS v6.1+](/docs/sources/mobile-trackers/hybrid-apps/)) or [React Native v4.2+](/docs/sources/react-native-tracker/hybrid-apps/) trackers. Hybrid apps are mobile apps that in addition to a native interface, provide part of the UI through an embedded web view.

If your web app will run both separately and as part of a hybrid app, with this plugin you only need one tracking implementation for your web app. You still have to implement the mobile trackers too.

When the plugin is active, for every event the web tracker checks if it's running in a web view with at least one of the mobile interfaces available. If it is, the web tracker forwards the event to the mobile tracker, and doesn't track it itself. If not, the web tracker tracks the event as normal.

> **Note:** The plugin is available since version 4.3 of the tracker.

The WebView integration is **automatic** once configured.

The diagram below shows the interaction of the WebView plugin and mobile trackers in hybrid apps.

```mermaid
flowchart TB

subgraph hybridApp[Hybrid Mobile App]

    subgraph webView[Web View]
        webViewCode[App logic]
        webViewTracker[JS tracker with WebView plugin]

        webViewCode -- "Tracks events" --> webViewTracker
    end

    subgraph nativeCode[Native Code]
        nativeAppCode[App logic]
        nativeTracker[Snowplow iOS/Android/React Native tracker]

        nativeAppCode -- "Tracks events" --> nativeTracker
    end

    webViewTracker -- "Forwards events" --> nativeTracker
end

subgraph cloud[Cloud]
    collector[Snowplow Collector]
end

nativeTracker -- "Sends tracked events" --> collector
```

To use the WebView plugin, you must have a mobile tracker initialized and configured.

The supported trackers are:

- Android v6.1+
- iOS v6.1+
- React Native v4.2+

How to set up hybrid app tracking:

1. Implement the Snowplow [iOS, Android](/docs/sources/mobile-trackers/), or [React Native](/docs/sources/react-native-tracker/) tracker in your mobile codebase.
2. Create a web view based on your web app, with Snowplow web tracking and WebView plugin instrumented.
3. Subscribe to the web view. Read how to do this for [native mobile](/docs/sources/mobile-trackers/hybrid-apps/) or [React Native](/docs/sources/react-native-tracker/hybrid-apps/).
4. Track events from web and mobile.

This plugin uses the [Snowplow WebView tracker](/docs/sources/webview-tracker/) as a dependency.

## What do the forwarded events look like?

The forwarded hybrid events will have all the information tracked by the web tracker. This includes all entities, whether configured by the tracker [automatically](/docs/sources/web-trackers/tracking-events/#add-contextual-data-with-entities) or by you as a global context. Baked-in (non-entity) properties such as user agent or URL are also included.

Additionally, any configured mobile entities will also be added. Again, this includes [auto-tracked entities](/docs/sources/mobile-trackers/tracking-events/#auto-tracked-events-and-entities) such as the screen, session, or platform entities, as well as any global context entities.

The forwarded events will have the web tracker version, e.g. "js-4.3.0", but the `namespace` and `appId` from the mobile tracker.

Hybrid events are still compatible with the Snowplow [Unified Digital dbt model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/).

## Install plugin

**JavaScript (tag):**

| Tracker Distribution | Included |
| -------------------- | -------- |
| `sp.js`              | ❌        |
| `sp.lite.js`         | ❌        |

**Download:**

|                                             |                                                                                                                 |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Download from GitHub Releases (Recommended) | [GitHub Releases (plugins.umd.zip)](https://github.com/snowplow/snowplow-javascript-tracker/releases)           |
| Available on jsDelivr                       | [jsDelivr](https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-webview@latest/dist/index.umd.min.js) (latest) |
| Available on unpkg                          | [unpkg](https://unpkg.com/@snowplow/browser-plugin-webview@latest/dist/index.umd.min.js) (latest)               |

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-webview@latest/dist/index.umd.min.js',
    ['snowplowWebViewTracking', 'WebViewPlugin']
);
```

**Browser (npm):**

- `npm install @snowplow/browser-plugin-webview`
- `yarn add @snowplow/browser-plugin-webview`
- `pnpm add @snowplow/browser-plugin-webview`

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { WebViewPlugin } from '@snowplow/browser-plugin-webview';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ WebViewPlugin() ],
});
```

***

## Configuration

By default, the plugin will forward events to the default initialized Snowplow tracker on each platform. To specify a different tracker instance, or multiple trackers, pass in a list of tracker namespaces at setup.

**JavaScript (tag):**

```javascript
window.snowplow(
    'addPlugin',
    'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-webview@latest/dist/index.umd.min.js',
    ["snowplowWebViewTracking", "WebViewPlugin"],
    [
      {
        trackerNamespaces: ["sp1", "sp2"],
      }
    ]
);
```

**Browser (npm):**

```javascript
import { newTracker } from '@snowplow/browser-tracker';
import { WebViewPlugin } from '@snowplow/browser-plugin-webview';

newTracker('sp1', '{{collector_url}}', {
   appId: 'my-app-id',
   plugins: [ WebViewPlugin({ trackerNamespaces: ['sp1', 'sp2'] }) ],
});
```

***

> **Warning:** When a mobile interface is available, the web tracker will forward the event rather than tracking it. If you specify namespaces in the configuration, and no mobile trackers actually exist with those namespaces, the event will be lost.

---

# Adjust webhook
> Track mobile attribution data including app installation and reattribution events from Adjust callbacks using the Iglu webhook adapter.
> Source: https://docs.snowplow.io/docs/sources/webhooks/adjust-webhook/

[Adjust](https://www.adjust.com/) provides a mobile attribution platform. It enables users to track what marketing channels drive mobile app installations.

Many Snowplow users who have mobile apps use Adjust to capture the sources (marketing channels) that drive app downloads. By integrating Adjust with Snowplow, it is straightforward to:

1. Have your mobile attribution data (at the app installation event-level), with the rest of your event-level data, in Snowplow
2. Join app installation data collected from Adjust, with other user event data collected through Snowplow from mobile, web or other platforms, in order to analyze the performance of those marketing channels and return on ad spend. This is typically used to drive future ad spend.

Integrating Adjust with Snowplow is straightforward. In this guide we will walk through the process.

## How the integration works: an overview

The integration uses the [Adjust Callback](https://docs.adjust.com/en/callbacks/). We will configure this to send an app install event to the Snowplow collector when an app is installed, with the attribution data attached. In addition, we will set this up to send a reattribution event every time this occurs in Adjust to Snowplow. Again, this will contain all the data we need to figure out what drove an earlier app installation.

On the Snowplow side, we will use the [Iglu webhook adapter](/docs/sources/webhooks/iglu-webhook/) to ensure that the event is correctly received and processed. We will create a schema for the event that matches the structure of the data sent from Adjust.

The guide is illustrative. One of the nice things about Adjust is that it gives you a huge amount of flexibility about what data you want to send into Snowplow and how. One of the great things about Snowplow is that it is flexible enough to work with very many structures of data, just so long as it knows the schema that the data adheres to. So there are lots of ways you can adapt the following setup. It should be a good start for most users however.

## Step-by-step guide

### 1. Setup the Adjust webhook

Log into Adjust and navigate to Settings -> Events screen in the dashboard. Hover over the install event, click the “edit” icon that appears and enter the following, substituting your own collector URL for `mycollector.mydomain.com`:

```markup
http://mycollector.mydomain.com/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.adjust%2Finstall%2Fjsonschema%2F1-0-0&app_id={app_id}&app_name={app_name}&app_name_dashboard={app_name_dashboard}&store={store}&tracker={tracker}&tracker_name={tracker_name}&network_name={network_name}&campaign_name={campaign_name}&adgroup_name={adgroup_name}&creative_name={creative_name}&impression_based={impression_based}&is_organic={is_organic}&gclid={gclid}&rejection_reason={rejection_reason}&click_referer={click_referer}&click_attribution_window={click_attribution_window}&impression_attribution_window={impression_attribution_window}&reattribution_attribution_window={reattribution_attribution_window}&inactive_user_definition={inactive_user_definition}&adid={adid}&idfa={idfa}&android_id={android_id}&android_id_md5={android_id_md5}&mac_sha1={mac_sha1}&mac_md5={mac_md5}&idfa-android-id={idfa||android_id}&idfa-or-gps-adid={idfa||gps_adid}&idfa_md5={idfa_md5}&idfa_md5_hex={idfa_md5_hex}&idfv={idfv}&gps_adid={gps_adid}&gps_adid_md5={gps_adid_md5}&win_udid={win_udid}&win_hwid={win_hwid}&win_naid={win_naid}&win_adid={win_adid}&match_type={match_type}&reftag={reftag}&referrer={referrer}&user_agent={user_agent}&ip_address={ip_address}&click_time={click_time}&engagement_time={engagement_time}&installed_at={installed_at}&installed_at_hour={installed_at_hour}&created_at={created_at}&reattributed_at={reattributed_at}&connection_type={connection_type}&isp={isp}&city={city}&country={country}&language={language}&device_name={device_name}&device_type={device_type}&os_name={os_name}&api_level={api_level}&sdk_version={sdk_version}&os_version={os_version}&environment={environment}&tracking_enabled={tracking_enabled}&timezone={timezone}&fb_campaign_group_name={fb_campaign_group_name}&fb_campaign_group_id={fb_campaign_group_id}&fb_campaign_name={fb_campaign_name}&fb_campaign_id={fb_campaign_id}&fb_adgroup_name={fb_adgroup_name}&fb_adgroup_id={fb_adgroup_id}&tweet_id={tweet_id}&twitter_line_item_id={twitter_line_item_id}&label={label}
```

We are collecting the data by adding key-value pairs to request in the format `data_point={data_point}`. The first part of each pair is the name of the field in the collected event and it must match a property name in your schema. The second part, in curly braces, is a placeholder that Adjust replaces with an actual value before making the callback. For example, `app_id={app_id}` will be transformed to something like `app_id=C013FJP3WF`.

In our example webhook setup above we’re using the schema that is defined further down in this post, but you are a free to use any custom compatible schema. There is no requirement that the key in each key-value pair must match the spelling of the respective Adjust placeholder, so your schema can have properties with names that better match your own business logic, e.g. `adjust_tracker={tracker}`.

We have elected to grab _all_ the data that Adjust makes available with an install event (at the time of writing).

For a complete list of data Adjust can send, see the [Adjust placeholder documentation](https://partners.adjust.com/placeholders/).

At the end of this post we’ll describe how you can tailor your setup to grab just a subset.

### 2. Create the corresponding jsonschema

In order for Snowplow to process the data sent to the Iglu webhook, we need to schema it.

_Since first publishing this guide, we’ve added an Adjust `install` event schema to our [Iglu Central](https://github.com/snowplow/iglu-central) schema repository. You can find it [here](https://github.com/snowplow/iglu-central/blob/master/schemas/com.adjust/install/jsonschema/1-0-0). We recommend using that schema for out-of-the-box tracking of all the data points associated with an `install` event (at the time the schema was written), rather than the example schema used further down. If you’re using Redshift you might also find the [table definition](https://github.com/snowplow/iglu-central/blob/master/sql/com.adjust/install_1.sql) and [JSONpaths file](https://github.com/snowplow/iglu-central/blob/master/jsonpaths/com.adjust/install_1.json) from Iglu Central useful. For Snowflake and BigQuery users, the loader app in the pipeline will figure those out itself._

_Refer to the rest of this guide to see how you can write your own custom schemas._

Upload the following schema to your Iglu repo as `com.adjust.snowplow/install/jsonschema/1-0-0`:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for Adjust install event",
  "self": {
    "vendor": "com.adjust.snowplow",
    "name": "install",
    "format": "jsonschema",
    "version": "1-0-0"
  },

  "type": "object",
  "properties": {
    "app_id": {
      "type": "string"
    },
    "app_name": {
      "type": "string",
      "maxLength": 256
    },
    "app_name_dashboard": {
      "type": "string",
      "maxLength": 256
    },
    "store": {
      "type": "string",
      "maxLength": 256
    },
    "tracker": {
      "type": "string",
      "maxLength": 256
    },
    "tracker_name": {
      "type": "string",
      "maxLength": 256
    },
    "network_name": {
      "type": "string",
      "maxLength": 256
    },
    "campaign_name": {
      "type": "string",
      "maxLength": 256
    },
    "adgroup_name": {
      "type": "string",
      "maxLength": 256
    },
    "creative_name": {
      "type": "string",
      "maxLength": 256
    },
    "impression_based": {
      "type": "string",
      "maxLength": 256
    },
    "is_organic": {
      "type": "string",
      "maxLength": 256
    },
    "gclid": {
      "type": "string",
      "maxLength": 256
    },
    "rejection_reason": {
      "type": "string",
      "maxLength": 256
    },
    "click_referer": {
      "type": "string",
      "maxLength": 256
    },
    "click_attribution_window": {
      "type": "string",
      "maxLength": 256
    },
    "impression_attribution_window": {
      "type": "string",
      "maxLength": 256
    },
    "reattribution_attribution_window": {
      "type": "string",
      "maxLength": 256
    },
    "inactive_user_definition": {
      "type": "string",
      "maxLength": 256
    },
    "adid": {
      "type": "string",
      "maxLength": 256
    },
    "idfa": {
      "type": "string",
      "maxLength": 256
    },
    "android_id": {
      "type": "string",
      "maxLength": 256
    },
    "android_id_md5": {
      "type": "string",
      "maxLength": 256
    },
    "mac_sha1": {
      "type": "string",
      "maxLength": 256
    },
    "mac_md5": {
      "type": "string",
      "maxLength": 256
    },
    "idfa-android-id": {
      "type": "string",
      "maxLength": 256
    },
    "idfa-or-gps-adid": {
      "type": "string",
      "maxLength": 256
    },
    "idfa_md5": {
      "type": "string",
      "maxLength": 256
    },
    "idfa_md5_hex": {
      "type": "string",
      "maxLength": 256
    },
    "idfv": {
      "type": "string",
      "maxLength": 256
    },
    "gps_adid": {
      "type": "string",
      "maxLength": 256
    },
    "gps_adid_md5": {
      "type": "string",
      "maxLength": 256
    },
    "win_udid": {
      "type": "string",
      "maxLength": 256
    },
    "win_hwid": {
      "type": "string",
      "maxLength": 256
    },
    "win_naid": {
      "type": "string",
      "maxLength": 256
    },
    "win_adid": {
      "type": "string",
      "maxLength": 256
    },
    "match_type": {
      "type": "string",
      "maxLength": 256
    },
    "reftag": {
      "type": "string",
      "maxLength": 256
    },
    "referrer": {
      "type": "string",
      "maxLength": 256
    },
    "user_agent": {
      "type": "string",
      "maxLength": 256
    },
    "ip_address": {
      "type": "string",
      "maxLength": 256
    },
    "click_time": {
      "type": "string",
      "maxLength": 256
    },
    "engagement_time": {
      "type": "string",
      "maxLength": 256
    },
    "installed_at": {
      "type": "string",
      "maxLength": 256
    },
    "installed_at_hour": {
      "type": "string",
      "maxLength": 256
    },
    "created_at": {
      "type": "string",
      "maxLength": 256
    },
    "reattributed_at": {
      "type": "string",
      "maxLength": 256
    },
    "connection_type": {
      "type": "string",
      "maxLength": 256
    },
    "isp": {
      "type": "string",
      "maxLength": 256
    },
    "city": {
      "type": "string",
      "maxLength": 256
    },
    "country": {
      "type": "string",
      "maxLength": 256
    },
    "language": {
      "type": "string",
      "maxLength": 256
    },
    "device_name": {
      "type": "string",
      "maxLength": 256
    },
    "device_type": {
      "type": "string",
      "maxLength": 256
    },
    "os_name": {
      "type": "string",
      "maxLength": 256
    },
    "api_level": {
      "type": "string",
      "maxLength": 256
    },
    "sdk_version": {
      "type": "string",
      "maxLength": 256
    },
    "os_version": {
      "type": "string",
      "maxLength": 256
    },
    "environment": {
      "type": "string",
      "maxLength": 256
    },
    "tracking_enabled": {
      "type": "string",
      "maxLength": 256
    },
    "timezone": {
      "type": "string",
      "maxLength": 256
    },
    "fb_campaign_group_name": {
      "type": "string",
      "maxLength": 256
    },
    "fb_campaign_group_id": {
      "type": "string",
      "maxLength": 256
    },
    "fb_campaign_name": {
      "type": "string",
      "maxLength": 256
    },
    "fb_campaign_id": {
      "type": "string",
      "maxLength": 256
    },
    "fb_adgroup_name": {
      "type": "string",
      "maxLength": 256
    },
    "fb_adgroup_id": {
      "type": "string",
      "maxLength": 256
    },
    "tweet_id": {
      "type": "string",
      "maxLength": 256
    },
    "twitter_line_item_id": {
      "type": "string",
      "maxLength": 256
    },
    "label": {
      "type": "string",
      "maxLength": 256
    }
  },
  "additionalProperties": true
}
```

We have included a field for each placeholder that we included in our Adjust callback. Although here we’ve used the same spelling for property names as the placeholders, there is no requirement to do so in your own schema.

### 3. Create the table in Redshift

Create a corresponding Redshift table for the schema. We recommend autogenerating this [Schema Guru](https://github.com/snowplow/schema-guru), e.g. by executing the following in the root of your schema registry:

```bash
/path/to/schema-guru-0.6.2 ddl --with-json-paths schemas/com.adjust.snowplow/install
```

Or with [Igluctl](/docs/api-reference/iglu/igluctl-2/):

```bash
/path/to/igluctl static generate --with-json-paths schemas/com.adjust.snowplow/install
```

In our case this auto-generated the following table definition. (_Please note that this example might be outdated. Since first publishing this guide, Redshift has introduced the more efficient ZSTD encoding for column compression, which we have adopted as standard in Igluctl._)

```sql
CREATE TABLE IF NOT EXISTS atomic.com_adjust_snowplow_install_1 (
    "schema_vendor"                    VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_name"                      VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_format"                    VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_version"                   VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "root_id"                          CHAR(36)      ENCODE RAW       NOT NULL,
    "root_tstamp"                      TIMESTAMP     ENCODE LZO       NOT NULL,
    "ref_root"                         VARCHAR(255)  ENCODE RUNLENGTH NOT NULL,
    "ref_tree"                         VARCHAR(1500) ENCODE RUNLENGTH NOT NULL,
    "ref_parent"                       VARCHAR(255)  ENCODE RUNLENGTH NOT NULL,
    "adgroup_name"                     VARCHAR(256)  ENCODE LZO,
    "adid"                             VARCHAR(256)  ENCODE LZO,
    "android_id"                       VARCHAR(256)  ENCODE LZO,
    "android_id_md5"                   VARCHAR(256)  ENCODE LZO,
    "api_level"                        VARCHAR(256)  ENCODE LZO,
    "app_id"                           VARCHAR(4096) ENCODE LZO,
    "app_name"                         VARCHAR(256)  ENCODE LZO,
    "app_name_dashboard"               VARCHAR(256)  ENCODE LZO,
    "campaign_name"                    VARCHAR(256)  ENCODE LZO,
    "city"                             VARCHAR(256)  ENCODE LZO,
    "click_attribution_window"         VARCHAR(256)  ENCODE LZO,
    "click_referer"                    VARCHAR(256)  ENCODE LZO,
    "click_time"                       VARCHAR(256)  ENCODE LZO,
    "connection_type"                  VARCHAR(256)  ENCODE LZO,
    "country"                          VARCHAR(256)  ENCODE LZO,
    "created_at"                       VARCHAR(256)  ENCODE LZO,
    "creative_name"                    VARCHAR(256)  ENCODE LZO,
    "device_name"                      VARCHAR(256)  ENCODE LZO,
    "device_type"                      VARCHAR(256)  ENCODE LZO,
    "engagement_time"                  VARCHAR(256)  ENCODE LZO,
    "environment"                      VARCHAR(256)  ENCODE LZO,
    "fb_adgroup_id"                    VARCHAR(256)  ENCODE LZO,
    "fb_adgroup_name"                  VARCHAR(256)  ENCODE LZO,
    "fb_campaign_group_id"             VARCHAR(256)  ENCODE LZO,
    "fb_campaign_group_name"           VARCHAR(256)  ENCODE LZO,
    "fb_campaign_id"                   VARCHAR(256)  ENCODE LZO,
    "fb_campaign_name"                 VARCHAR(256)  ENCODE LZO,
    "gclid"                            VARCHAR(256)  ENCODE LZO,
    "gps_adid"                         VARCHAR(256)  ENCODE LZO,
    "gps_adid_md5"                     VARCHAR(256)  ENCODE LZO,
    "idfa"                             VARCHAR(256)  ENCODE LZO,
    "idfa_android_id"                  VARCHAR(256)  ENCODE LZO,
    "idfa_or_gps_adid"                 VARCHAR(256)  ENCODE LZO,
    "idfa_md5"                         VARCHAR(256)  ENCODE LZO,
    "idfa_md5_hex"                     VARCHAR(256)  ENCODE LZO,
    "idfv"                             VARCHAR(256)  ENCODE LZO,
    "impression_attribution_window"    VARCHAR(256)  ENCODE LZO,
    "impression_based"                 VARCHAR(256)  ENCODE LZO,
    "inactive_user_definition"         VARCHAR(256)  ENCODE LZO,
    "installed_at"                     VARCHAR(256)  ENCODE LZO,
    "installed_at_hour"                VARCHAR(256)  ENCODE LZO,
    "ip_address"                       VARCHAR(256)  ENCODE LZO,
    "is_organic"                       VARCHAR(256)  ENCODE LZO,
    "isp"                              VARCHAR(256)  ENCODE LZO,
    "label"                            VARCHAR(256)  ENCODE LZO,
    "language"                         VARCHAR(256)  ENCODE LZO,
    "mac_md5"                          VARCHAR(256)  ENCODE LZO,
    "mac_sha1"                         VARCHAR(256)  ENCODE LZO,
    "match_type"                       VARCHAR(256)  ENCODE LZO,
    "network_name"                     VARCHAR(256)  ENCODE LZO,
    "os_name"                          VARCHAR(256)  ENCODE LZO,
    "os_version"                       VARCHAR(256)  ENCODE LZO,
    "reattributed_at"                  VARCHAR(256)  ENCODE LZO,
    "reattribution_attribution_window" VARCHAR(256)  ENCODE LZO,
    "referrer"                         VARCHAR(256)  ENCODE LZO,
    "reftag"                           VARCHAR(256)  ENCODE LZO,
    "rejection_reason"                 VARCHAR(256)  ENCODE LZO,
    "sdk_version"                      VARCHAR(256)  ENCODE LZO,
    "store"                            VARCHAR(256)  ENCODE LZO,
    "timezone"                         VARCHAR(256)  ENCODE LZO,
    "tracker"                          VARCHAR(256)  ENCODE LZO,
    "tracker_name"                     VARCHAR(256)  ENCODE LZO,
    "tracking_enabled"                 VARCHAR(256)  ENCODE LZO,
    "tweet_id"                         VARCHAR(256)  ENCODE LZO,
    "twitter_line_item_id"             VARCHAR(256)  ENCODE LZO,
    "user_agent"                       VARCHAR(256)  ENCODE LZO,
    "win_adid"                         VARCHAR(256)  ENCODE LZO,
    "win_hwid"                         VARCHAR(256)  ENCODE LZO,
    "win_naid"                         VARCHAR(256)  ENCODE LZO,
    "win_udid"                         VARCHAR(256)  ENCODE LZO,
    FOREIGN KEY (root_id) REFERENCES atomic.events(event_id)
)
DISTSTYLE KEY
DISTKEY (root_id)
SORTKEY (root_tstamp);

COMMENT ON TABLE com_adjust_snowplow_install_1 IS 'iglu:com.adjust.snowplow/install/jsonschema/1-0-0';
```

### 4. Create the correpsonding JSONpaths file

Finally add the following JSONpaths file to your jsonpaths folder (as `com.adjust.snowplow/install_1.json`). Your JSONpaths file should have already been auto-generated using schema-guru or Igluctl:

```json
{
    "jsonpaths": [
        "$.schema.vendor",
        "$.schema.name",
        "$.schema.format",
        "$.schema.version",
        "$.hierarchy.rootId",
        "$.hierarchy.rootTstamp",
        "$.hierarchy.refRoot",
        "$.hierarchy.refTree",
        "$.hierarchy.refParent",
        "$.data.adgroup_name",
        "$.data.adid",
        "$.data.android_id",
        "$.data.android_id_md5",
        "$.data.api_level",
        "$.data.app_id",
        "$.data.app_name",
        "$.data.app_name_dashboard",
        "$.data.campaign_name",
        "$.data.city",
        "$.data.click_attribution_window",
        "$.data.click_referer",
        "$.data.click_time",
        "$.data.connection_type",
        "$.data.country",
        "$.data.created_at",
        "$.data.creative_name",
        "$.data.device_name",
        "$.data.device_type",
        "$.data.engagement_time",
        "$.data.environment",
        "$.data.fb_adgroup_id",
        "$.data.fb_adgroup_name",
        "$.data.fb_campaign_group_id",
        "$.data.fb_campaign_group_name",
        "$.data.fb_campaign_id",
        "$.data.fb_campaign_name",
        "$.data.gclid",
        "$.data.gps_adid",
        "$.data.gps_adid_md5",
        "$.data.idfa",
        "$.data.idfa-android-id",
        "$.data.idfa-or-gps-adid",
        "$.data.idfa_md5",
        "$.data.idfa_md5_hex",
        "$.data.idfv",
        "$.data.impression_attribution_window",
        "$.data.impression_based",
        "$.data.inactive_user_definition",
        "$.data.installed_at",
        "$.data.installed_at_hour",
        "$.data.ip_address",
        "$.data.is_organic",
        "$.data.isp",
        "$.data.label",
        "$.data.language",
        "$.data.mac_md5",
        "$.data.mac_sha1",
        "$.data.match_type",
        "$.data.network_name",
        "$.data.os_name",
        "$.data.os_version",
        "$.data.reattributed_at",
        "$.data.reattribution_attribution_window",
        "$.data.referrer",
        "$.data.reftag",
        "$.data.rejection_reason",
        "$.data.sdk_version",
        "$.data.store",
        "$.data.timezone",
        "$.data.tracker",
        "$.data.tracker_name",
        "$.data.tracking_enabled",
        "$.data.tweet_id",
        "$.data.twitter_line_item_id",
        "$.data.user_agent",
        "$.data.win_adid",
        "$.data.win_hwid",
        "$.data.win_naid",
        "$.data.win_udid"
    ]
}
```

### 5. Extending the setup for reattribution events

To extend the setup for reattribution events you work through the same process as for installation events:

#### 5.1 Setup the callback

E.g. by creating the following callback for a reattribution event:

```text
http://mycollector.mydomain.com/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.adjust.snowplow%2Freattribute%2Fjsonschema%2F1-0-0&app_id={app_id}&app_name={app_name}&app_name_dashboard={app_name_dashboard}&store={store}&tracker={tracker}&tracker_name={tracker_name}&last_tracker={last_tracker}&last_tracker_name={last_tracker_name}&network_name={network_name}&campaign_name={campaign_name}&adgroup_name={adgroup_name}&creative_name={creative_name}&impression_based={impression_based}&is_organic={is_organic}&gclid={gclid}&click_attribution_window={click_attribution_window}&impression_attribution_window={impression_attribution_window}&reattribution_attribution_window={reattribution_attribution_window}&inactive_user_definition={inactive_user_definition}&adid={adid}&idfa={idfa}&android_id={android_id}&android_id_md5={android_id_md5}&mac_sha1={mac_sha1}&mac_md5={mac_md5}&idfa-android-id={idfa||android_id}&idfa-or-gps-adid={idfa||gps_adid}&idfa_md5={idfa_md5}&idfa_md5_hex={idfa_md5_hex}&idfv={idfv}&gps_adid={gps_adid}&gps_adid_md5={gps_adid_md5}&win_udid={win_udid}&win_hwid={win_hwid}&win_naid={win_hwid}&win_adid={win_adid}&match_type={match_type}&reftag={reftag}&referrer={referrer}&user_agent={user_agent}&ip_address={ip_address}&click_time={click_time}&engagement_time={engagement_time}&installed_at={installed_at}&installed_at_hour={installed_at_hour}&created_at={created_at}&reattributed_at={reattributed_at}&connection_type={connection_type}&isp={isp}&city={city}&country={country}&language={language}&device_name={device_name}&device_type={device_type}&os_name={os_name}&api_level={api_level}&sdk_version={sdk_version}&os_version={os_version}&environment={environment}&tracking_enabled={tracking_enabled}&timezone={timezone}&time_spent={time_spent}&lifetime_session_count={lifetime_session_count}&deeplink={deeplink}&fb_campaign_group_name={fb_campaign_group_name}&fb_campaign_group_id={fb_campaign_group_id}&fb_campaign_name={fb_campaign_name}&fb_campaign_id={fb_campaign_id}&fb_adgroup_name={fb_adgroup_name}&fb_adgroup_id={fb_adgroup_id}&tweet_id={tweet_id}&twitter_line_item_id={twitter_line_item_id}&label={label}
```

As before, keep in mind that there is no requirement that the key in each key-value pair must match the spelling of the associated Adjust placeholder. For example, you might decide to use a property name for the `{impression_based}` placeholder that makes it clearer that this field contains a Boolean value and makes the naming more consistent with other Boolean fields: `is_impression_based={impression_based}`. The only requirement is that the keys must match the property names in your schema.

#### 5.2 Create a corresponding jsonschema for the reattribution event

E.g.:

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for Adjust reattribute event",
  "self": {
    "vendor": "com.adjust.snowplow",
    "name": "reattribute",
    "format": "jsonschema",
    "version": "1-0-0"
  },

  "type": "object",
  "properties": {
    "app_id": {
      "type": "string"
    },
    "app_name": {
      "type": "string",
      "maxLength": 1024
    },
    "app_name_dashboard": {
      "type": "string",
      "maxLength": 1024
    },
    "store": {
      "type": "string",
      "maxLength": 1024
    },
    "tracker": {
      "type": "string",
      "maxLength": 1024
    },
    "tracker_name": {
      "type": "string",
      "maxLength": 1024
    },
    "last_tracker": {
      "type": "string",
      "maxLength": 1024
    },
    "last_tracker_name": {
      "type": "string",
      "maxLength": 1024
    },
    "network_name": {
      "type": "string",
      "maxLength": 1024
    },
    "campaign_name": {
      "type": "string",
      "maxLength": 1024
    },
    "adgroup_name": {
      "type": "string",
      "maxLength": 1024
    },
    "creative_name": {
      "type": "string",
      "maxLength": 1024
    },
    "impression_based": {
      "type": "string",
      "maxLength": 1024
    },
    "is_organic": {
      "type": "string",
      "maxLength": 1024
    },
    "gclid": {
      "type": "string",
      "maxLength": 1024
    },
    "click_attribution_window": {
      "type": "string",
      "maxLength": 1024
    },
    "impression_attribution_window": {
      "type": "string",
      "maxLength": 1024
    },
    "reattribution_attribution_window": {
      "type": "string",
      "maxLength": 1024
    },
    "inactive_user_definition": {
      "type": "string",
      "maxLength": 1024
    },
    "adid": {
      "type": "string",
      "maxLength": 1024
    },
    "idfa": {
      "type": "string",
      "maxLength": 1024
    },
    "android_id": {
      "type": "string",
      "maxLength": 1024
    },
    "android_id_md5": {
      "type": "string",
      "maxLength": 1024
    },
    "mac_sha1": {
      "type": "string",
      "maxLength": 1024
    },
    "mac_md5": {
      "type": "string",
      "maxLength": 1024
    },
    "idfa-android-id": {
      "type": "string",
      "maxLength": 1024
    },
    "idfa-or-gps-adid": {
      "type": "string",
      "maxLength": 1024
    },
    "idfa_md5": {
      "type": "string",
      "maxLength": 1024
    },
    "idfa_md5_hex": {
      "type": "string",
      "maxLength": 1024
    },
    "idfv": {
      "type": "string",
      "maxLength": 1024
    },
    "gps_adid": {
      "type": "string",
      "maxLength": 1024
    },
    "gps_adid_md5": {
      "type": "string",
      "maxLength": 1024
    },
    "win_udid": {
      "type": "string",
      "maxLength": 1024
    },
    "win_hwid": {
      "type": "string",
      "maxLength": 1024
    },
    "win_naid": {
      "type": "string",
      "maxLength": 1024
    },
    "win_adid": {
      "type": "string",
      "maxLength": 1024
    },
    "match_type": {
      "type": "string",
      "maxLength": 1024
    },
    "reftag": {
      "type": "string",
      "maxLength": 1024
    },
    "referrer": {
      "type": "string",
      "maxLength": 1024
    },
    "user_agent": {
      "type": "string",
      "maxLength": 1024
    },
    "ip_address": {
      "type": "string",
      "maxLength": 1024
    },
    "click_time": {
      "type": "string",
      "maxLength": 1024
    },
    "engagement_time": {
      "type": "string",
      "maxLength": 1024
    },
    "installed_at": {
      "type": "string",
      "maxLength": 1024
    },
    "installed_at_hour": {
      "type": "string",
      "maxLength": 1024
    },
    "created_at": {
      "type": "string",
      "maxLength": 1024
    },
    "reattributed_at": {
      "type": "string",
      "maxLength": 1024
    },
    "connection_type": {
      "type": "string",
      "maxLength": 1024
    },
    "isp": {
      "type": "string",
      "maxLength": 1024
    },
    "city": {
      "type": "string",
      "maxLength": 1024
    },
    "country": {
      "type": "string",
      "maxLength": 1024
    },
    "language": {
      "type": "string",
      "maxLength": 1024
    },
    "device_name": {
      "type": "string",
      "maxLength": 1024
    },
    "device_type": {
      "type": "string",
      "maxLength": 1024
    },
    "os_name": {
      "type": "string",
      "maxLength": 1024
    },
    "api_level": {
      "type": "string",
      "maxLength": 1024
    },
    "sdk_version": {
      "type": "string",
      "maxLength": 1024
    },
    "os_version": {
      "type": "string",
      "maxLength": 1024
    },
    "environment": {
      "type": "string",
      "maxLength": 1024
    },
    "tracking_enabled": {
      "type": "string",
      "maxLength": 1024
    },
    "timezone": {
      "type": "string",
      "maxLength": 1024
    },
    "time_spent": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_campaign_group_name": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_campaign_group_id": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_campaign_name": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_campaign_id": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_adgroup_name": {
      "type": "string",
      "maxLength": 1024
    },
    "fb_adgroup_id": {
      "type": "string",
      "maxLength": 1024
    },
    "tweet_id": {
      "type": "string",
      "maxLength": 1024
    },
    "twitter_line_item_id": {
      "type": "string",
      "maxLength": 1024
    },
    "label": {
      "type": "string",
      "maxLength": 1024
    }
  },
  "additionalProperties": true
}
```

#### 5.3 Implement the corresponding SQL table definitions and jsonpaths

Do this using [Schema Guru](https://github.com/snowplow/schema-guru) e.g. the SQL table definition by executing the following from the root of your schema registry:

```text
$ /path/to/schema-guru-0.6.2 ddl --with-json-paths schemas/com.adjust.snowplow/reattribute
```

Or with [Igluctl](/docs/api-reference/iglu/igluctl-2/):

```bash
$ /path/to/igluctl static generate --with-json-paths schemas/com.adjust.snowplow/reattribute
```

The generated SQL file (_Please note that this example might be outdated. Since first publishing this guide, Redshift has introduced the more efficient ZSTD encoding for column compression, which we have adopted as standard in Igluctl._):

```sql
CREATE TABLE IF NOT EXISTS atomic.com_adjust_snowplow_reattribute_1 (
    "schema_vendor"                    VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_name"                      VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_format"                    VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "schema_version"                   VARCHAR(128)  ENCODE RUNLENGTH NOT NULL,
    "root_id"                          CHAR(36)      ENCODE RAW       NOT NULL,
    "root_tstamp"                      TIMESTAMP     ENCODE LZO       NOT NULL,
    "ref_root"                         VARCHAR(255)  ENCODE RUNLENGTH NOT NULL,
    "ref_tree"                         VARCHAR(1500) ENCODE RUNLENGTH NOT NULL,
    "ref_parent"                       VARCHAR(255)  ENCODE RUNLENGTH NOT NULL,
    "adgroup_name"                     VARCHAR(1024)  ENCODE LZO,
    "adid"                             VARCHAR(1024)  ENCODE LZO,
    "android_id"                       VARCHAR(1024)  ENCODE LZO,
    "android_id_md5"                   VARCHAR(1024)  ENCODE LZO,
    "api_level"                        VARCHAR(1024)  ENCODE LZO,
    "app_id"                           VARCHAR(4096) ENCODE LZO,
    "app_name"                         VARCHAR(1024)  ENCODE LZO,
    "app_name_dashboard"               VARCHAR(1024)  ENCODE LZO,
    "campaign_name"                    VARCHAR(1024)  ENCODE LZO,
    "city"                             VARCHAR(1024)  ENCODE LZO,
    "click_attribution_window"         VARCHAR(1024)  ENCODE LZO,
    "click_time"                       VARCHAR(1024)  ENCODE LZO,
    "connection_type"                  VARCHAR(1024)  ENCODE LZO,
    "country"                          VARCHAR(1024)  ENCODE LZO,
    "created_at"                       VARCHAR(1024)  ENCODE LZO,
    "creative_name"                    VARCHAR(1024)  ENCODE LZO,
    "device_name"                      VARCHAR(1024)  ENCODE LZO,
    "device_type"                      VARCHAR(1024)  ENCODE LZO,
    "engagement_time"                  VARCHAR(1024)  ENCODE LZO,
    "environment"                      VARCHAR(1024)  ENCODE LZO,
    "fb_adgroup_id"                    VARCHAR(1024)  ENCODE LZO,
    "fb_adgroup_name"                  VARCHAR(1024)  ENCODE LZO,
    "fb_campaign_group_id"             VARCHAR(1024)  ENCODE LZO,
    "fb_campaign_group_name"           VARCHAR(1024)  ENCODE LZO,
    "fb_campaign_id"                   VARCHAR(1024)  ENCODE LZO,
    "fb_campaign_name"                 VARCHAR(1024)  ENCODE LZO,
    "gclid"                            VARCHAR(1024)  ENCODE LZO,
    "gps_adid"                         VARCHAR(1024)  ENCODE LZO,
    "gps_adid_md5"                     VARCHAR(1024)  ENCODE LZO,
    "idfa"                             VARCHAR(1024)  ENCODE LZO,
    "idfa_android_id"                  VARCHAR(1024)  ENCODE LZO,
    "idfa_or_gps_adid"                 VARCHAR(1024)  ENCODE LZO,
    "idfa_md5"                         VARCHAR(1024)  ENCODE LZO,
    "idfa_md5_hex"                     VARCHAR(1024)  ENCODE LZO,
    "idfv"                             VARCHAR(1024)  ENCODE LZO,
    "impression_attribution_window"    VARCHAR(1024)  ENCODE LZO,
    "impression_based"                 VARCHAR(1024)  ENCODE LZO,
    "inactive_user_definition"         VARCHAR(1024)  ENCODE LZO,
    "installed_at"                     VARCHAR(1024)  ENCODE LZO,
    "installed_at_hour"                VARCHAR(1024)  ENCODE LZO,
    "ip_address"                       VARCHAR(1024)  ENCODE LZO,
    "is_organic"                       VARCHAR(1024)  ENCODE LZO,
    "isp"                              VARCHAR(1024)  ENCODE LZO,
    "label"                            VARCHAR(1024)  ENCODE LZO,
    "language"                         VARCHAR(1024)  ENCODE LZO,
    "last_tracker"                     VARCHAR(1024)  ENCODE LZO,
    "last_tracker_name"                VARCHAR(1024)  ENCODE LZO,
    "mac_md5"                          VARCHAR(1024)  ENCODE LZO,
    "mac_sha1"                         VARCHAR(1024)  ENCODE LZO,
    "match_type"                       VARCHAR(1024)  ENCODE LZO,
    "network_name"                     VARCHAR(1024)  ENCODE LZO,
    "os_name"                          VARCHAR(1024)  ENCODE LZO,
    "os_version"                       VARCHAR(1024)  ENCODE LZO,
    "reattributed_at"                  VARCHAR(1024)  ENCODE LZO,
    "reattribution_attribution_window" VARCHAR(1024)  ENCODE LZO,
    "referrer"                         VARCHAR(1024)  ENCODE LZO,
    "reftag"                           VARCHAR(1024)  ENCODE LZO,
    "sdk_version"                      VARCHAR(1024)  ENCODE LZO,
    "store"                            VARCHAR(1024)  ENCODE LZO,
    "time_spent"                       VARCHAR(1024)  ENCODE LZO,
    "timezone"                         VARCHAR(1024)  ENCODE LZO,
    "tracker"                          VARCHAR(1024)  ENCODE LZO,
    "tracker_name"                     VARCHAR(1024)  ENCODE LZO,
    "tracking_enabled"                 VARCHAR(1024)  ENCODE LZO,
    "tweet_id"                         VARCHAR(1024)  ENCODE LZO,
    "twitter_line_item_id"             VARCHAR(1024)  ENCODE LZO,
    "user_agent"                       VARCHAR(1024)  ENCODE LZO,
    "win_adid"                         VARCHAR(1024)  ENCODE LZO,
    "win_hwid"                         VARCHAR(1024)  ENCODE LZO,
    "win_naid"                         VARCHAR(1024)  ENCODE LZO,
    "win_udid"                         VARCHAR(1024)  ENCODE LZO,
    FOREIGN KEY (root_id) REFERENCES atomic.events(event_id)
)
DISTSTYLE KEY
DISTKEY (root_id)
SORTKEY (root_tstamp);

COMMENT ON TABLE atomic.com_adjust_snowplow_reattribute_1 IS 'iglu:com.adjust.snowplow/reattribute/jsonschema/1-0-0';
```

and the generated jsonpath file:

```text
{
    "jsonpaths": [
        "$.schema.vendor",
        "$.schema.name",
        "$.schema.format",
        "$.schema.version",
        "$.hierarchy.rootId",
        "$.hierarchy.rootTstamp",
        "$.hierarchy.refRoot",
        "$.hierarchy.refTree",
        "$.hierarchy.refParent",
        "$.data.adgroup_name",
        "$.data.adid",
        "$.data.android_id",
        "$.data.android_id_md5",
        "$.data.api_level",
        "$.data.app_id",
        "$.data.app_name",
        "$.data.app_name_dashboard",
        "$.data.campaign_name",
        "$.data.city",
        "$.data.click_attribution_window",
        "$.data.click_time",
        "$.data.connection_type",
        "$.data.country",
        "$.data.created_at",
        "$.data.creative_name",
        "$.data.device_name",
        "$.data.device_type",
        "$.data.engagement_time",
        "$.data.environment",
        "$.data.fb_adgroup_id",
        "$.data.fb_adgroup_name",
        "$.data.fb_campaign_group_id",
        "$.data.fb_campaign_group_name",
        "$.data.fb_campaign_id",
        "$.data.fb_campaign_name",
        "$.data.gclid",
        "$.data.gps_adid",
        "$.data.gps_adid_md5",
        "$.data.idfa",
        "$.data.idfa-android-id",
        "$.data.idfa-or-gps-adid",
        "$.data.idfa_md5",
        "$.data.idfa_md5_hex",
        "$.data.idfv",
        "$.data.impression_attribution_window",
        "$.data.impression_based",
        "$.data.inactive_user_definition",
        "$.data.installed_at",
        "$.data.installed_at_hour",
        "$.data.ip_address",
        "$.data.is_organic",
        "$.data.isp",
        "$.data.label",
        "$.data.language",
        "$.data.last_tracker",
        "$.data.last_tracker_name",
        "$.data.mac_md5",
        "$.data.mac_sha1",
        "$.data.match_type",
        "$.data.network_name",
        "$.data.os_name",
        "$.data.os_version",
        "$.data.reattributed_at",
        "$.data.reattribution_attribution_window",
        "$.data.referrer",
        "$.data.reftag",
        "$.data.sdk_version",
        "$.data.store",
        "$.data.time_spent",
        "$.data.timezone",
        "$.data.tracker",
        "$.data.tracker_name",
        "$.data.tracking_enabled",
        "$.data.tweet_id",
        "$.data.twitter_line_item_id",
        "$.data.user_agent",
        "$.data.win_adid",
        "$.data.win_hwid",
        "$.data.win_naid",
        "$.data.win_udid"
    ]
}
```

You're now all set for capturing reattribution events as well as installation events.

### 6. Tailoring the setup

#### 6.1 Maximalist vs minimalist setup

The above setup is a maximalist setup: we've decided to grab as much data from Adjust as Adjust make available with their installation and reattribution events. Our corresponding callback, schema, SQL table definition and jsonpath file are all very long. (Because they contain all the different fields.)

You can choose to set this up for a more limited set of fields. In this case you'd simply remove the name value pairs from the callback, remove them from the schema, jsonpath and sql table definitions.

You might also want to setup Callbacks for other Adjust events, if you want to add more than the attribution data to your Snowplow data set.

#### 6.2 Adding custom parameters

Adjust supports sending your own [custom parameters](https://docs.adjust.com/en/callbacks/#reference-custom-sdk-parameters) with each callback.

To include these custom parameters in your Adjust-Snowplow integration, all you need to do is extend the schemas for each Adjust event types with the custom parameters you want to record into Snowplow. Note that you **do not need to update the webhook itself**. Whilst you do need to list each Adjust placeholder that you want passed into Snowplow on the webhook, Adjust will automatically add every custom parameter you record in it onto the callback. So as long as your event schema includes those custom parameters, they will automatically be fetched.

Once you've updated your schema to include the custom parameters, you'll need to similarly make sure that your corresponding Redshift table definition and jsonpath files have the additional parameters in too.

### 7. Integrating callbacks from other third parties

Adjust is not the only company to provide event-level data via a Callback or Webhook. Zendesk is another example. The same approach / methodology should work for Zendesk and any similar provider that lets you specify the Callback URL and individual data points passed on the querystring.

---

# Iglu webhook
> Track self-describing events via GET or POST requests with custom JSON schemas for flexible third-party webhook integrations.
> Source: https://docs.snowplow.io/docs/sources/webhooks/iglu-webhook/

This webhook adapter lets you track events sent via a `GET` or `POST` request containing an [Iglu](https://github.com/snowplow/iglu)-compatible event payload.

You can use this adapter with vendors who allow you define your **own** event types for "postback" events or custom Webhook events.

## Setup

Integrating Iglu-compatible webhooks into Snowplow is a two-stage process:

1. Configure your third-party system to send Iglu-compatible events to Snowplow
2. Setup the appropriate JSON Schema for each Iglu-compatible event you are sending through

## Your webhook

The Iglu webhook adapter supports events send in as `GET` and `POST` requests.

### Path

We use a special path to tell Snowplow that these events should be parsed as Iglu self-describing JSON events:

```markup
http://{{COLLECTOR_URL}}/com.snowplowanalytics.iglu/v1?schema=<iglu schema uri>&...
```

### Required fields

You can send in whatever name-value pairs on the querystring that make sense for your event, but you **must** also include a `schema` parameter, which is set to a valid Iglu self-describing schema URI.

The below examples all use a schema available on Iglu Central. However, you will likely want to create your own schema that describes the event for the platform you are receiving data from.

The below examples use the `social_interaction` schema:

```text
iglu:com.snowplowanalytics.snowplow/social_interaction/jsonschema/1-0-0
```

### Optional fields

If you want to specify which app these events belong to, add an `aid` parameter as taken from the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields):

```text
...&aid=<company code>&...
```

You can also manually override the event's `platform` parameter like so:

```text
...&p=<platform code>&...
```

Supported platform codes can again be found in the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields); if not set, then the value for `platform` will default to `srv` for a server-side application.

### Example `GET` request

Here is an example of an Iglu-compatible event sent as a `GET` request, for a [Social Interacton event](https://github.com/snowplow/iglu-central/blob/master/schemas/com.snowplowanalytics.snowplow/social_interaction/jsonschema/1-0-0), broken out onto multiple lines to make it easier to read:

```markup
http://{{COLLECTOR_URL}}/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.snowplowanalytics.snowplow%2Fsocial_interaction%2Fjsonschema%2F1-0-0
  &aid=mobile-attribution
  &p=mob
  &network=twitter
  &action=retweet
```

This will be converted by the Iglu webhook adapter into a self-describing JSON looking like this:

```json
{
  "schema":"iglu:com.snowplowanalytics.snowplow/social_interaction/jsonschema/1-0-0",
  "data": {
    "network": "twitter",
    "action": "retweet"
  }
}
```

The Snowplow enriched event containing this JSON will include `app_id` set to "mobile-attribution" and `platform` set to "mob".

### Example `POST` requests

POST requests can be compiled in three different ways for the Iglu webhook:

- As a full Self Describing JSON in the body
- With a `?schema=<iglu schema uri>` in the querystring and JSON in the body
- As a `x-www-form-urlencoded` payload

**NOTE**: For the event to be accepted the `Content-Type` must be either:

- `application/json`
- `application/json; charset=utf-8`
- `application/x-www-form-urlencoded`

The below examples are written as `curl` requests as an example, however you can send the events using any tool or technology that supports sending POST requests.

To send as a full Self Describing JSON in the body and a Content-Type of `application/json`:

```bash
curl --request POST \
   --url http://{{COLLECTOR_URL}}/com.snowplowanalytics.iglu/v1 \
   --header 'Content-Type: application/json' \
   --data '{
   "schema":"iglu:com.snowplowanalytics.snowplow/social_interaction/jsonschema/1-0-0",
   "data": {
         "network": "twitter",
         "action": "retweet"
     }
 }'
```

To send with a `?schema=<iglu schema uri>` in the querystring and a data JSON in the body and Content-Type of `application/json`:

```bash
curl --request POST \
   --url 'http://{{COLLECTOR_URL}}/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.snowplowanalytics.snowplow%2Fsocial_interaction%2Fjsonschema%2F1-0-0' \
   --header 'Content-Type: application/json' \
   --data '{
   "network": "twitter",
   "action": "retweet"
 }'
```

To send as a `x-www-form-urlencoded` payload:

```bash
curl --request POST \
   --url 'http://{{COLLECTOR_URL}}/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.snowplowanalytics.snowplow%2Fsocial_interaction%2Fjsonschema%2F1-0-0' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data 'network=twitter&action=retweet'
```

As with the `GET` request above you can also attach extra information into the querystring to help describe your event. The following parameters can be added:

- `aid=` : The application ID
- `p=` : The platform
- `nuid=` : The network user ID
- `eid=` : A custom event ID
- `ttm=` : The true timestamp
- `url=` : The page URL
- `cv=` : The context vendor (deprecated)

---

# Collect data from third parties with webhooks
> Receive third-party event streams through webhooks for Adjust, Iglu, Iterable, MailGun, Mandrill, SendGrid, and Zendesk integrations.
> Source: https://docs.snowplow.io/docs/sources/webhooks/

Snowplow enables you to collect events via the webhooks of supported third-party software.

Webhooks allow the third-party software to send their own internal event streams to your Snowplow Collector for further processing. Webhooks are sometimes referred to as "streaming APIs" or "HTTP response APIs".

| Webhook                                          | Track                                                                |
| ------------------------------------------------ | -------------------------------------------------------------------- |
| [Adjust](/docs/sources/webhooks/adjust-webhook/) | Which marketing channels are driving mobile app installations        |
| [Iglu](/docs/sources/webhooks/iglu-webhook/)     | Any [Iglu](/docs/api-reference/iglu/)-compatible GET or POST request |
| [Iterable](/docs/sources/webhooks/iterable/)     | Events provided by Iterable                                          |
| [MailGun](/docs/sources/webhooks/mailgun/)       | Email activity logged by MailGun                                     |
| [Mandrill](/docs/sources/webhooks/mandrill/)     | Email activity logged by Mandrill                                    |
| [SendGrid](/docs/sources/webhooks/sendgrid/)     | Email activity logged by SendGrid                                    |
| [Zendesk](/docs/sources/webhooks/zendesk/)       | Events logged by Zendesk                                             |

---

# Iterable webhook
> Track email opens, clicks, SMS, push notifications, and other Iterable system webhook events directly into Snowplow.
> Source: https://docs.snowplow.io/docs/sources/webhooks/iterable/

This webhook integration lets you track a variety of events provided by Iterable through their [System Webhooks](https://support.iterable.com/hc/en-us/articles/208013936-System-Webhooks). The event types include email opens and clicks, sent SMS or push notifications, and more.

## Configure Iterable to send events to Snowplow

First, login to Iterable. Under **Integrations** in the top toolbar, click on **System webhooks**.

Then select **Create webhook** above the list of webhooks on the right and enter the endpoint URL. The endpoint URL field is the URI to your Snowplow Collector. We use a special path to tell Snowplow that these events are generated by Iterable:

```markup
https://<collector host>/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.iterable%2Fsystem_webhook%2Fjsonschema%2F1-0-0
```

You may want to set the snowplow `aid=` parameter in your URL query string to the company for which this webhook has been configured; this is the `app_id` parameter taken from the [Snowplow Tracker Protocol](/docs/events/), however this is optional. The company is also contained in the received messages.

If you want, you can also manually override the event's `platform` parameter by appending a query string to the end of the URL, in combination or separately with aid above. Here is what the final URL would look like for a configured aid and platform:

```markup
https://<collector host>/com.snowplowanalytics.iglu/v1?schema=iglu%3Acom.iterable%2Fsystem_webhook%2Fjsonschema%2F1-0-0&aid=<company>&p=<platform code>
```

Supported platform codes can again be found in the [Snowplow Tracker Protocol](/docs/fundamentals/canonical-event/#application-fields); if not set, then the value for `platform` will default to `srv` for a server-side application.

Having entered the endpoint URL, click on **Create**. This will open a form where you can set up authentication of the callbacks – leave that to "None". In the bottom part of the page, you will be able to select which events to subscribe to as shown in this screenshot:

![](/assets/images/iterable-events-810x1024-9c02a317d11b6989116049079f224476.png)

That’s it – when you save the configuration, Iterable events should automatically flow through into your data warehouse.

---

# MailGun webhook
> Track MailGun email events including delivered messages, hard bounces, spam complaints, unsubscribes, clicks, and opens.
> Source: https://docs.snowplow.io/docs/sources/webhooks/mailgun/

This webhook integration lets you track a variety of events logged by [Mailgun](https://www.mailgun.com/).

Available events are:

- Delivered Messages
- Dropped Messages
- Hard Bounces
- Spam Complaints
- Unsubscribes
- Clicks
- Opens

## Compatibility

- [R97 Knossos](https://github.com/snowplow/snowplow/releases/tag/r97-knossos)+ (`POST`-capable collectors only)
- [Mailgun webhook API](https://documentation.mailgun.com/en/latest/user_manual.html#webhooks)

## Setup

Integrating Mailgun's webhooks into Snowplow is a two-stage process:

1. Configure Mailgun to send events to Snowplow
2. (Optional) Create the Mailgun events tables into Amazon Redshift

## Mailgun

First login to Mailgun. Select **Webhooks** from the top panel.

![](/assets/images/mailgun-1-89f9f315ff609950647cb2558bfeff0d.png)

Then select the domain for which you want to configure snowplow.

![](/assets/images/mailgun-2-c0145bb44af8cad166858be686c3b45b.png)

Once you have the desired domain selected, you can configure snowplow for the type of events that interest you.

Following is an example configuring the "Spam Complaints" event. The process is identical for all events.

Click on the cross next to the event type you woudl like to configure. In this case "Spam Complaints":

![](/assets/images/mailgun-3-3ce9ff3a625c2511140b49e8a9bbb9b5.png)

Then set the collector URL:

```markup
http://<collector host>/com.mailgun/v1
```

Finally click on **Set Webhook URL**. You can optionally click on **Test Webhook** to verify that your collector instance is reachable from Mailgun's servers.

You may want to set the snowplow `aid=` parameter in your URL query string to the company for which this webhook has been configured; this is the `app_id` parameter taken from the [Snowplow Tracker Protocol](/docs/events/), however this is optional. The company is also contained in the received messages.

If you want, you can also manually override the event's `platform` parameter by appending a query string to the end of the URL, in combination or separately with aid above. Here is what the final URL would look like for a configured aid and platform:

```markup
http://<collector host>/com.mailgun/v1?aid=<company>&p=<platform code>
```

Supported platform codes can again be found in the [Snowplow Tracker Protocol](/docs/events/); if not set, then the value for `platform` will default to `srv` for a server-side application.

---

# Mandrill webhook
> Track Mandrill email events including sent messages, bounces, opens, clicks, spam reports, rejections, and unsubscribes.
> Source: https://docs.snowplow.io/docs/sources/webhooks/mandrill/

This webhook integration lets you track a variety of events logged by [Mandrill](https://mandrill.com/).

Available events are:

- Message sent
- Message hard bounced
- Message soft bounced
- Message opened
- Message marked as spam
- Message rejected
- Message delayed
- Message clicked
- Recipient unsubscribed

## Compatibility

- [Snowplow 0.9.14](https://github.com/snowplow/snowplow/releases/tag/0.9.14)+ (`POST`-capable collectors only)
- [Mandrill webhook API](http://help.mandrill.com/entries/21738186-Introduction-to-Webhooks)

## Setup

Integrating Mandrill's webhooks into Snowplow is a two-stage process:

1. Configure Mandrill to send events to Snowplow
2. (Optional) Create the Mandrill events tables into Amazon Redshift

## Mandrill

First login to Mandrill and click on the **Settings** button which will be on the left hand side of the screen.

Once in the Settings menu click on **Webhooks** and then click the **Add a Webhook** button.

Now we can start to setup the webhook by selecting what events we want our Mandrill Webhook to trigger!

- Please note: 'Rejection Whitelist Changes' & 'Rejection Blacklist Changes' are not supported by the Snowplow MandrillAdapter.

![](/assets/images/mandrill-1-cd468779fceea6dfa60e1b0368accad1.png)

Once we have selected what events we want to record we need to fill in the **Post To URL** field:

- For the this field you will need to provide the URI to your Snowplow Collector. We use a special path to tell Snowplow that these events are generated by Mandrill:

```markup
http://<collector host>/com.mandrill/v1?aid=<company code>
```

The `aid=` name-value pair in your URI's querystring is optional; this is the `app_id` parameter taken from the [Snowplow Tracker Protocol](/docs/events/). You can use it to specify which company in Mandrill these call complete events belong to. Putting it all together, our setup screen now looks like this:

![](/assets/images/mandrill-2-723fe55962c02e12fae65adb12b1eeab.png)

If you want, you can also manually override the event's `platform` parameter like so:

```markup
http://<collector host>/com.mandrill/v1?aid=<company code>&p=<platform code>
```

Supported platform codes can again be found in the [Snowplow Tracker Protocol](/docs/events/); if not set, then the value for `platform` will default to `srv` for a server-side application.

Once you click the **Create Webhook** button it will attempt to authenticate that the Collector actually exists and is ready to receive events. If everything is setup correctly it will return to the previous page and you will now see your new Webhook listed!

That's it - with this table deployed, your Mandrill events should automatically flow through into Redshift.

---

# SendGrid webhook
> Track SendGrid email notification events including processed, delivered, bounced, opened, clicked, unsubscribed, and spam report events.
> Source: https://docs.snowplow.io/docs/sources/webhooks/sendgrid/

This webhook integration lets you track a variety of events logged by [SendGrid](http://sendgrid.com/).

Available events are:

- Processed
- Dropped
- Delivered
- Deferred
- Bounce
- Open
- Click
- Spam Report
- Unsubscribe
- Group Unsubscribe
- Group Resubscribe

## Compatibility

- The support of the latest version of Sendgrid webhook has been introduced in [Snowplow R114 Polonnaruwa](https://github.com/snowplow/snowplow/releases/tag/r114-polonnaruwa)
- [SendGrid webhook API](https://docs.sendgrid.com/for-developers/tracking-events/event)

## Setup

Integrating SendGrid's webhooks into Snowplow is a two-stage process:

1. Configure SendGrid to send events to Snowplow
2. (Optional) Create the SendGrid events tables for Amazon Redshift

## Configure SendGrid

First login to SendGrid. Select **Settings** from the menu panel along the left-hand side of the screen. You should then navigate in the expanded list to the **Mail Settings** page.

Select **Event Notification** from the list by clicking the row. Ensure it's switched **ON** in order to send events to Snowplow.

Click **edit** on the top right-hand side of the **Event Notification** dropdown.

For the **HTTP POST URL** field you will need to provide the URI to your Snowplow Collector. We use a special path to tell Snowplow that these events are generated by SendGrid:

```markup
http://<collector host>/com.sendgrid/v3
```

Our Webhooks setup page should look like this after we have added our **HTTP POST URL**:

![](/assets/images/sendgrid-388161440c37b156c61747fc25dda42b.png)

If you want, you can also manually override the event's `platform` parameter by appending a query string to the end of the URL so:

```markup
http://<collector host>/com.sendgrid/v3?p=<platform code>
```

Supported platform codes can again be found in the [Snowplow Tracker Protocol](/docs/events/); if not set, then the value for `platform` will default to `srv` for a server-side application.

The other values you can set up manually in the similar fashion are `nuid`, `aid`, `cv`, `eid`, `ttm`, and `url`.

Before we save our SendGrid webhook we can configure what types of events SendGrid will send to our webhook and what channels will trigger these events. Simply select the boxes that are applicable to you and SendGrid will send these events to our webhook.

---

# Zendesk webhook
> Track Zendesk ticket events and user context data including ticket creation, updates, and associated requester, assignee, and submitter information.
> Source: https://docs.snowplow.io/docs/sources/webhooks/zendesk/

You can configure Zendesk to automatically send `POST` requests to a (Clojure or Scala) collector. The first step is to set up a Zendesk "extension" pointing at the collector.

Log in to Zendesk. Click the cogwheel-shaped "Admin" icon located at the bottom-left corner of the Dashboard page to take you to the _Admin_ page.

In the "SETTINGS" menu, click on "Extensions":

![](/assets/images/extensions-button-371bebc45257a1de8edc107adb4d5bef.png)

Click "add target":

![](/assets/images/add-extension-d0f1cadc80c515fba9ba2f8780682330.png)

Choose "HTTP target" from the list of target types to add:

![](/assets/images/http-target-ea24a5ba3fc11a7065b41456521f8496.png)

Name the new extension something like "Snowplow Collector - Iglu POST". The "Iglu POST" here represents the fact we will be sending Zendesk events and contexts to [Iglu webhook adapter](/docs/sources/webhooks/iglu-webhook/) via `POST` request.

In the **URL** field, enter `https://{{collector_domain}}/com.snowplowanalytics.iglu/v1?aid=zendesk`, replacing `{{collector_domain}}` with your collector domain.

You can optionally have `?aid={{my_zendesk_namespace}}` added to this URL, where `{{my_zendesk_namespace}}` is a label for the application (here: "zendesk"). This label will be attached to all events fired by the extension, so you can later check where a given event came from (useful if you have more than one Zendesk account).

Set the **Method** field to "POST" and the **Content type** to "JSON" from the drop-down lists.

Select "Create Target" and click the _**Submit**_ button.

![](/assets/images/extension-form-4fc613af29cbde8a2876f6a6e249e8df.png)

We have set up our collector as a Zendesk extension. We can now add a trigger which sends `POST` requests to the collector whenever certain events occur.

## 2. Setting up a trigger for Zendesk event

### Setting up trigger conditions

From the _Admin_ page, select "Triggers" from the "BUSINESS RULES" menu and click "add trigger":

![](/assets/images/add-trigger-button-2b2cf08ed2b193b48b6cddd69ec05662.png)

Name the trigger something like "Ticket created or updated" to reflect Zendesk data will be send on ticket creation and update events.

Under "Meet ANY of the following conditions" header click _**Add condition**_ button to add 2 "Ticket: Is..." conditions and set them to "Created" and "Updated" respectively.

![](/assets/images/trigger-conditions-291f41055b3389ade6c909f47ed2bfba.png)

### Setting up body for ticket event

In the "Actions" section, click on _**Add action**_ button and select "Notify target" and "Snowplow Collector - Iglu POST".

In the _**JSON body**_ box, paste the following:

```json
{
  "schema": "iglu:com.zendesk.snowplow/ticket_updated/jsonschema/1-0-0",
  "data": {
    "via": "{{ticket.via}}",
    "ticketType": {% if ticket.ticket_type.size > 0 %}"{{ticket.ticket_type}}"{% else %}null{% endif %},
    "updatedAt": "{{ticket.updated_at_with_timestamp}}",
    "ticketId": {{ticket.id}},
    "ticketTitle": "{{ticket.title}}",
    "priority": {% if ticket.priority.size > 0 %}"{{ticket.priority}}"{% else %}null{% endif %},
    "inBusinessHours": {{ticket.in_business_hours}},
    "createdAt": "{{ticket.created_at_with_timestamp}}",
    "account": "{{ticket.account}}",
    "brand": "{{ticket.brand.name}}",
    "url": "{{ticket.link}}",
    "externalId": {% if ticket.external_id.size > 0 %}"{{ticket.external_id}}"{% else %}null{% endif %},
    "organizationName": "{{ticket.organization.name}}",
    "organizationId": {% if ticket.requester.organization.id %}{{ticket.requester.organization.id}}{% else %}null{% endif %},
    "status": "{{ticket.status}}",
    "dueDate": {% if ticket.due_date_with_timestamp.size > 0 %}"{{ticket.due_date_with_timestamp}}"{% else %}null{% endif %},
    "tags": {% if ticket.tags.size > 0 %}"{{ticket.tags}}"{% else %}null{% endif %},
    "ccNames": "{{ticket.cc_names}}",
    "groupAssigned": "{{ticket.group.name}}",
    "latestCommentAuthorName": "{{ticket.latest_comment.author.name}}",
    "latestComment": "{{ticket.latest_comment.value}}",
    "latestCommentIsPublic": {{ticket.latest_comment.is_public}}
  }
}
```

![](/assets/images/json-body-e06e6da2d1ecf2abfa96172d7e17cbe5.png)

_NOTE:_ Ignore the warning on the left-hand side of the _**JSON body**_ textbox. It is due to usage of [Liquid markup](https://shopify.github.io/liquid/) in JSON.

### Setting up user contexts

#### Setting up body for ticket requester

In the "Actions" section, select the 2nd "Notify target" and "Snowplow Collector - Iglu POST" extension.

In the _**JSON body**_ box, paste the following:

```json
{
  "schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",
  "data": {
      "ticketId": {{ticket.id}},
      "updatedAt": "{{ticket.updated_at_with_timestamp}}",
      "type": "requester",
      "firstName": {% if ticket.requester.first_name.size > 0 %}"{{ticket.requester.first_name}}"{% else %}null{% endif %},
      "lastName": {% if ticket.requester.last_name.size > 0 %}"{{ticket.requester.last_name}}"{% else %}null{% endif %},
      "language": {% if ticket.requester.language.size > 0 %}"{{ticket.requester.language}}"{% else %}null{% endif %},
      "tags": {% if ticket.requester.tags.size > 0 %}"{{ticket.requester.tags}}"{% else %}null{% endif %},
      "locale": {% if ticket.requester.locale.size > 0 %}"{{ticket.requester.locale}}"{% else %}null{% endif %},
      "notes": {% if ticket.requester.notes.size > 0 %}"{{ticket.requester.notes}}"{% else %}null{% endif %},
      "timeZone": {% if ticket.requester.time_zone.size > 0 %}"{{ticket.requester.time_zone}}"{% else %}null{% endif %},
      "userId": {% if ticket.requester.id %}{{ticket.requester.id}}{% else %}null{% endif %},
      "phone": {% if ticket.requester.phone.size > 0 %}"{{ticket.requester.phone}}"{% else %}null{% endif %},
      "extendedRole": {% if ticket.requester.extended_role.size > 0 %}"{{ticket.requester.extended_role}}"{% else %}null{% endif %},
      "role": {% if ticket.requester.role.size > 0 %}"{{ticket.requester.role}}"{% else %}null{% endif %},
      "details": {% if ticket.requester.details.size > 0 %}"{{ticket.requester.details}}"{% else %}null{% endif %},
      "signature": {% if ticket.requester.signature.size > 0 %}"{{ticket.requester.signature}}"{% else %}null{% endif %},
      "organization": {% if ticket.requester.organization.size > 0 %}"{{ticket.requester.organization}}"{% else %}null{% endif %},
      "externalId": {% if ticket.requester.external_id.size > 0 %}"{{ticket.requester.external_id}}"{% else %}null{% endif %},
      "email": {% if ticket.requester.email.size > 0 %}"{{ticket.requester.email}}"{% else %}null{% endif %}
  }
}
```

#### Setting up body for ticket assignee

In the "Actions" section, select the 3nd "Notify target" and "Snowplow Collector - Iglu POST" extension.

In the _**JSON body**_ box, paste the following:

```json
{
  "schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",
  "data": {
      "ticketId": {{ticket.id}},
      "updatedAt": "{{ticket.updated_at_with_timestamp}}",
      "type": "assignee",
      "firstName": {% if ticket.assignee.first_name.size > 0 %}"{{ticket.assignee.first_name}}"{% else %}null{% endif %},
      "lastName": {% if ticket.assignee.last_name.size > 0 %}"{{ticket.assignee.last_name}}"{% else %}null{% endif %},
      "language": {% if ticket.assignee.language.size > 0 %}"{{ticket.assignee.language}}"{% else %}null{% endif %},
      "tags": {% if ticket.assignee.tags.size > 0 %}"{{ticket.assignee.tags}}"{% else %}null{% endif %},
      "locale": {% if ticket.assignee.locale.size > 0 %}"{{ticket.assignee.locale}}"{% else %}null{% endif %},
      "notes": {% if ticket.assignee.notes.size > 0 %}"{{ticket.assignee.notes}}"{% else %}null{% endif %},
      "timeZone": {% if ticket.assignee.time_zone.size > 0 %}"{{ticket.assignee.time_zone}}"{% else %}null{% endif %},
      "userId": {% if ticket.assignee.id %}{{ticket.assignee.id}}{% else %}null{% endif %},
      "phone": {% if ticket.assignee.phone.size > 0 %}"{{ticket.assignee.phone}}"{% else %}null{% endif %},
      "extendedRole": {% if ticket.assignee.extended_role.size > 0 %}"{{ticket.assignee.extended_role}}"{% else %}null{% endif %},
      "role": {% if ticket.assignee.role.size > 0 %}"{{ticket.assignee.role}}"{% else %}null{% endif %},
      "details": {% if ticket.assignee.details.size > 0 %}"{{ticket.assignee.details}}"{% else %}null{% endif %},
      "signature": {% if ticket.assignee.signature.size > 0 %}"{{ticket.assignee.signature}}"{% else %}null{% endif %},
      "organization": {% if ticket.assignee.organization.size > 0 %}"{{ticket.assignee.organization}}"{% else %}null{% endif %},
      "externalId": {% if ticket.assignee.external_id.size > 0 %}"{{ticket.assignee.external_id}}"{% else %}null{% endif %},
      "email": {% if ticket.assignee.email.size > 0 %}"{{ticket.assignee.email}}"{% else %}null{% endif %}
  }
}
```

#### Setting up body for ticket submitter

In the "Actions" section, select the 4th "Notify target" and "Snowplow Collector - Iglu POST" extension.

In the _**JSON body**_ box, paste the following:

```json
{
  "schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",
  "data": {
      "ticketId": {{ticket.id}},
      "updatedAt": "{{ticket.updated_at_with_timestamp}}",
      "type": "submitter",
      "firstName": {% if ticket.submitter.first_name.size > 0 %}"{{ticket.submitter.first_name}}"{% else %}null{% endif %},
      "lastName": {% if ticket.submitter.last_name.size > 0 %}"{{ticket.submitter.last_name}}"{% else %}null{% endif %},
      "language": {% if ticket.submitter.language.size > 0 %}"{{ticket.submitter.language}}"{% else %}null{% endif %},
      "tags": {% if ticket.submitter.tags.size > 0 %}"{{ticket.submitter.tags}}"{% else %}null{% endif %},
      "locale": {% if ticket.submitter.locale.size > 0 %}"{{ticket.submitter.locale}}"{% else %}null{% endif %},
      "notes": {% if ticket.submitter.notes.size > 0 %}"{{ticket.submitter.notes}}"{% else %}null{% endif %},
      "timeZone": {% if ticket.submitter.time_zone.size > 0 %}"{{ticket.submitter.time_zone}}"{% else %}null{% endif %},
      "userId": {% if ticket.submitter.id %}{{ticket.submitter.id}}{% else %}null{% endif %},
      "phone": {% if ticket.submitter.phone.size > 0 %}"{{ticket.submitter.phone}}"{% else %}null{% endif %},
      "extendedRole": {% if ticket.submitter.extended_role.size > 0 %}"{{ticket.submitter.extended_role}}"{% else %}null{% endif %},
      "role": {% if ticket.submitter.role.size > 0 %}"{{ticket.submitter.role}}"{% else %}null{% endif %},
      "details": {% if ticket.submitter.details.size > 0 %}"{{ticket.submitter.details}}"{% else %}null{% endif %},
      "signature": {% if ticket.submitter.signature.size > 0 %}"{{ticket.submitter.signature}}"{% else %}null{% endif %},
      "organization": {% if ticket.submitter.organization.size > 0 %}"{{ticket.submitter.organization}}"{% else %}null{% endif %},
      "externalId": {% if ticket.submitter.external_id.size > 0 %}"{{ticket.submitter.external_id}}"{% else %}null{% endif %},
      "email": {% if ticket.submitter.email.size > 0 %}"{{ticket.submitter.email}}"{% else %}null{% endif %}
  }
}
```

#### Setting up body for current user

In the "Actions" section, select the 5th (final) "Notify target" and "Snowplow Collector - Iglu POST" extention.

In the _**JSON body**_ box, paste the following:

```json
{
  "schema": "iglu:com.zendesk.snowplow/user/jsonschema/1-0-0",
  "data": {
      "ticketId": {{ticket.id}},
      "updatedAt": "{{ticket.updated_at_with_timestamp}}",
      "type": "current_user",
      "firstName": {% if current_user.first_name.size > 0 %}"{{current_user.first_name}}"{% else %}null{% endif %},
      "lastName": {% if current_user.last_name.size > 0 %}"{{current_user.last_name}}"{% else %}null{% endif %},
      "language": {% if current_user.language.size > 0 %}"{{current_user.language}}"{% else %}null{% endif %},
      "tags": {% if current_user.tags.size > 0 %}"{{current_user.tags}}"{% else %}null{% endif %},
      "locale": {% if current_user.locale.size > 0 %}"{{current_user.locale}}"{% else %}null{% endif %},
      "notes": {% if current_user.notes.size > 0 %}"{{current_user.notes}}"{% else %}null{% endif %},
      "timeZone": {% if current_user.time_zone.size > 0 %}"{{current_user.time_zone}}"{% else %}null{% endif %},
      "userId": {% if current_user.id %}{{current_user.id}}{% else %}null{% endif %},
      "phone": {% if current_user.phone.size > 0 %}"{{current_user.phone}}"{% else %}null{% endif %},
      "extendedRole": {% if current_user.extended_role.size > 0 %}"{{current_user.extended_role}}"{% else %}null{% endif %},
      "role": {% if current_user.role.size > 0 %}"{{current_user.role}}"{% else %}null{% endif %},
      "details": {% if current_user.details.size > 0 %}"{{current_user.details}}"{% else %}null{% endif %},
      "signature": {% if current_user.signature.size > 0 %}"{{current_user.signature}}"{% else %}null{% endif %},
      "organization": {% if current_user.organization.size > 0 %}"{{current_user.organization}}"{% else %}null{% endif %},
      "externalId": {% if current_user.external_id.size > 0 %}"{{current_user.external_id}}"{% else %}null{% endif %},
      "email": {% if current_user.email.size > 0 %}"{{current_user.email}}"{% else %}null{% endif %}
  }
}
```

Submit the new trigger by clicking _**Create**_ button. It should look something like this:

![](/assets/images/submit-target-b44ec8e7ca6f1bf57767b3b2c853e35e.png)

---

# WebView tracker (legacy)
> Track events from web views in mobile hybrid apps by forwarding events to native iOS, Android, or React Native trackers with shared session data.
> Source: https://docs.snowplow.io/docs/sources/webview-tracker/

The [Snowplow WebView Tracker](https://github.com/snowplow-incubator/snowplow-webview-tracker) allows you to track Snowplow events from web views in **mobile hybrid apps**. The current tracker version is 0.3.0.

Hybrid apps are mobile apps that in addition to a native interface, provide part of the UI through an embedded web view. Snowplow events are tracked from both the native code (e.g. written in Swift or Kotlin) as well as the web view (in JavaScript). The goal is to have events that are tracked from the native code and web view share the same session, and appear as tracked with the same tracker.

> **Tip:** We recommend using the [Snowplow web tracker](/docs/sources/web-trackers/) with [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/) (which uses the WebView tracker as a dependency), rather than using this tracker directly.
>
> The WebView plugin automatically forwards all tracked events to the mobile tracker. Events must be manually tracked when using the WebView tracker by itself.

The WebView tracker forwards tracked events to the native app code to be tracked by the Snowplow mobile trackers ([iOS, Android tracker](/docs/sources/mobile-trackers/hybrid-apps/), or [React Native](/docs/sources/react-native-tracker/hybrid-apps/)). The mobile trackers must be subscribed to the WebView messages.

The mobile trackers will continue processing the events as if they were tracked from the native app code. For example, adding any configured context entities, and timestamps.

The diagram below shows the interaction of the WebView and mobile trackers in hybrid apps.

```mermaid
flowchart TB

subgraph hybridApp[Hybrid Mobile App]

    subgraph webView[Web View]
        webViewCode[App logic]
        webViewTracker[Snowplow WebView tracker]

        webViewCode -- "Tracks events" --> webViewTracker
    end

    subgraph nativeCode[Native Code]
        nativeAppCode[App logic]
        nativeTracker[Snowplow iOS/Android/React Native tracker]

        nativeAppCode -- "Tracks events" --> nativeTracker
    end

    webViewTracker -- "Forwards events" --> nativeTracker
end

subgraph cloud[Cloud]
    collector[Snowplow Collector]
end

nativeTracker -- "Sends tracked events" --> collector
```

## Installation

You may choose to install the tracker as an npm package or by loading it through an HTML script tag.

**Using npm:**

To install the WebView tracker in your JavaScript or TypeScript app, add the npm package:

```bash
npm install --save @snowplow/webview-tracker
```

You will then be able to use the functions provided by the WebView tracker as follows:

```typescript
import { trackSelfDescribingEvent } from '@snowplow/webview-tracker';
```

**Using Snowplow tag:**

You may download the `sp.js` file from the [Releases section on GitHub](https://github.com/snowplow-incubator/snowplow-webview-tracker/releases), self-host it, and load to your page using the following tag:

```html
<script type="text/javascript" async=1>
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[]; p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments) };p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1; n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","{{URL to sp.js}}","snowplow"));
</script>
```

> **Note:** Replace the `{{URL to sp.js}}` with the URL to the `sp.js` file in the snippet.

***

In addition, you will need to implement the [iOS, Android](/docs/sources/mobile-trackers/hybrid-apps/), or [React Native](/docs/sources/react-native-tracker/hybrid-apps/) tracker and subscribe to WebView messages.

## Tracking events

To track events, simply call their corresponding functions given the event data. The events will be processed based on the equivalent mobile event types after forwarding.

The following functions are available:

| Method                     | Event type tracked                                                                                                      |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `trackSelfDescribingEvent` | Track a custom event based on "self-describing" JSON schema                                                             |
| `trackStructEvent`         | Track a semi-custom structured event                                                                                    |
| `trackScreenView`          | Track a view of a screen in the app                                                                                     |
| `trackPageView`            | Track a Web page visit                                                                                                  |
| `trackWebViewEvent`        | Track any Snowplow event (used internally by the [WebView plugin](/docs/sources/web-trackers/tracking-events/webview/)) |

All the methods share common features and parameters. Every type of event can have optional entities added.

### Self-describing (custom)

Use the `trackSelfDescribingEvent` function to track a [fully custom event](/docs/fundamentals/events/).

| Argument  | Description                                                | Required? |
| --------- | ---------------------------------------------------------- | --------- |
| `event`   | Self-describing event, with `schema` and `data` properties | Yes       |
| `context` | List of context entities as self-describing JSON           | No        |

**Installed using npm:**

```javascript
trackSelfDescribingEvent({
    event: {
        schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
        data: {
            targetUrl: 'http://a-target-url.com'
        }
    }
});
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackSelfDescribingEvent', {
    event: {
        schema: 'iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1',
        data: {
            targetUrl: 'http://a-target-url.com'
        }
    }
});
```

***

### Screen views

Use `trackScreenView` to track a user viewing a screen (or similar) within your app. This is the page view equivalent for apps that are not web pages.

| Argument         | Description                                                | Required? |
| ---------------- | ---------------------------------------------------------- | --------- |
| `name`           | The human-readable name of the screen viewed               | Yes       |
| `id`             | The id (UUID v4) of screen that was viewed                 | Yes       |
| `type`           | The type of screen that was viewed                         | No        |
| `previousName`   | The name of the previous screen that was viewed            | No        |
| `previousType`   | The type of screen that was viewed                         | No        |
| `previousId`     | The id (UUID v4) of the previous screen that was viewed    | No        |
| `transitionType` | The type of transition that led to the screen being viewed | No        |
| `context`        | List of context entities as self-describing JSON           | No        |

**Installed using npm:**

```javascript
trackScreenView({
    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
    name: 'home',
    type: 'full',
    transitionType: 'none'
});
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackScreenView', {
    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
    name: 'home',
    type: 'full',
    transitionType: 'none'
});
```

***

### Page views

The `PageViewEvent` may be used to track page views on the web. The event is designed to track web page views and automatically captures page title, referrer, and URL.

| Argument  | Description                                      | Required? |
| --------- | ------------------------------------------------ | --------- |
| `title`   | Override the page title                          | No        |
| `context` | List of context entities as self-describing JSON | No        |

**Installed using npm:**

```javascript
trackPageView();
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackPageView', {});
```

***

### Structured

Track a semi-custom Structured event.

| Argument   | Description                                                    | Required? |
| ---------- | -------------------------------------------------------------- | --------- |
| `category` | The grouping of structured events which this action belongs to | Yes       |
| `action`   | Defines the type of user interaction which this event involves | Yes       |
| `label`    | Often used to refer to the "object" the action is performed on | No        |
| `property` | Describing the "object", or the action performed on it         | No        |
| `value`    | Provides numerical data about the event                        | No        |
| `context`  | List of context entities as self-describing JSON               | No        |

**Installed using npm:**

```javascript
trackStructEvent({
    category: 'shop',
    action: 'add-to-basket',
    label: 'Add To Basket',
    property: 'pcs',
    value: 2.00,
});
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackStructEvent', {
    category: 'shop',
    action: 'add-to-basket',
    label: 'Add To Basket',
    property: 'pcs',
    value: 2.00,
});
```

***

### Web view

> **Info:** This method is available since v0.3.0.

This method is used internally by the WebView plugin. We recommend implementing the web tracker with WebView plugin rather than using this directly.

Use this method to track any kind of Snowplow event e.g. a page ping. You will need to define the event name yourself, e.g. "pp" for page ping. It also allows you to set a tracker version, to help distinguish between native and WebView events (e.g. "webview-0.3.0" while the native tracker version might be something like "ios-6.1.0").

| Argument     | Description                                                                               | Required? |
| ------------ | ----------------------------------------------------------------------------------------- | --------- |
| `properties` | Event properties that are ["baked-in"](/docs/fundamentals/canonical-event/#common-fields) | Yes       |
| `event`      | An optional self-describing event, with `schema` and `data` properties                    | No        |
| `context`    | List of context entities as self-describing JSON                                          | No        |

| Event type                 | `eventName`    |
| -------------------------- | -------------- |
| Page view                  | `pv`           |
| Page ping                  | `pp`           |
| Structured                 | `se`           |
| Ecommerce transaction      | `tr`           |
| Ecommerce transaction item | `ti`           |
| All other events           | `ue` (default) |

**Installed using npm:**

```javascript
trackWebViewEvent({
      properties: {
        eventName: 'pp',
        trackerVersion: 'webview-0.3.0',
        url: 'https://test.com/test',
        referrer: 'https://test.com',
      },
    });
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackWebViewEvent', {
      properties: {
        eventName: 'pp',
        trackerVersion: 'webview-0.3.0',
        url: 'https://test.com/test',
        referrer: 'https://test.com',
      },
    });
```

***

## Adding context entities

You can add a list of entities to any event. In this example, two entities are added.

**Installed using npm:**

```javascript
trackScreenView({
      id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
      context: [
        {
          schema: 'iglu:com.my_company/movie_poster/jsonschema/1-0-0',
          data: {
            movie_name: 'Solaris',
            poster_country: 'JP',
            poster_date: '1978-01-01',
          },
        },
        {
          schema: 'iglu:com.my_company/customer/jsonschema/1-0-0',
          data: {
            p_buy: 0.23,
            segment: 'young_adult',
          },
        },
      ],
    });
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackScreenView', {
    id: '2c295365-eae9-4243-a3ee-5c4b7baccc8f',
    context: [
        {
          schema: 'iglu:com.my_company/movie_poster/jsonschema/1-0-0',
          data: {
            movie_name: 'Solaris',
            poster_country: 'JP',
            poster_date: '1978-01-01',
          },
        },
        {
          schema: 'iglu:com.my_company/customer/jsonschema/1-0-0',
          data: {
            p_buy: 0.23,
            segment: 'young_adult',
          },
        },
      ],
});
```

***

## Specifying a tracker namespace

You can specify tracker namespaces for the event. If not specified, the default tracker will be used.

All of the `trackXYZ()` methods accept two arguments:

| Argument   | Description                                                                                 | Required? |
| ---------- | ------------------------------------------------------------------------------------------- | --------- |
| `event`    | Event body, depends on the event being tracked                                              | Yes       |
| `trackers` | Optional list of tracker namespaces to track the event with (undefined for default tracker) | No        |

For instance, the following tracks a page view event using a tracker initialized with the namespace `ns1`:

**Installed using npm:**

```javascript
trackPageView({}, ['ns1']);
```

**Installed using Snowplow tag:**

```javascript
window.snowplow('trackPageView', {}, ['ns1']);
```

***

> **Warning:** If you specify namespaces in the configuration, and no mobile trackers actually exist with those namespaces, the event will be lost.

---

# Verify schema dependencies with the Data Structures CI tool
> Integrate Data Structures CI into your deployment pipelines to verify schema dependencies are deployed before code deployment. Use as a GitHub Action or with other CI/CD tools.
> Source: https://docs.snowplow.io/docs/testing/data-structures-ci-tool/

The Data Structures CI is a command-line tool which integrates Data Structures API into your CI/CD pipelines and currently has one task which verifies that all schema dependencies for a project are already deployed into a specified environment (e.g. "DEV", "PROD").

This is available as a [Github Action](#setting-up-as-a-github-action) and as a [universal install for other deployment pipelines](#setting-up-for-other-deployment-pipelines) e.g. Travis CI, CircleCI, Gitlab, Azure Pipelines, Jenkins…

## Authorization

In order to be able to perform tasks with the tool, you will need to supply both your Organization ID and an API key.

You can find your Organization ID [on the _Manage organization_ page](https://console.snowplowanalytics.com/settings) in Console.

An API Key can be created [here](https://console.snowplowanalytics.com/credentials).

## Create your manifest file

This command allows you to verify that all schema dependencies for a project (declared in a specific "manifest") are already deployed into an environment (e.g. "DEV", "PROD").

In your application project, create a JSON file for your manifest that will store references to the schema dependencies you have for your project. During a CI build this file will be parsed, validated and used by Data Structures CI to check that each schema is correctly deployed to the appropriate environment before the code for the application gets deployed, effectively guarding against the 'Schema not found' type of [failed events](/docs/fundamentals/failed-events/).

Here is an example manifest file where our application has dependencies on three schemas:

- `checkout_process` version `1-0-7`
- `user` version `1-0-1`
- `product` version `2-0-0`

```json
{
  "schema": "iglu:com.snowplowanalytics.insights/data_structures_dependencies/jsonschema/1-0-0",
  "data": {
    "schemas": [
      {
        "vendor": "com.acme.marketing",
        "name": "checkout_process",
        "format": "jsonschema",
        "version": "1-0-7"
      },
      {
        "vendor": "com.acme",
        "name": "user",
        "format": "jsonschema",
        "version": "1-0-1"
      },
      {
        "vendor": "com.acme",
        "name": "product",
        "format": "jsonschema",
        "version": "2-0-0"
      }
    ]
  }
}
```

The manifest must adhere to this [self-describing JSON Schema](http://iglucentral.com/schemas/com.snowplowanalytics.insights/data_structures_dependencies/jsonschema/1-0-0).

## Setting up as a Github Action

To use the Github Action simply add this snippet as a step on your existing GitHub Actions pipeline, replacing the relevant variables:

```yaml
name: Example workflow using Snowplow's Data Structures CI
on: push
jobs:
  data-structures-check:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master
    - name: Run Snowplow's Data Structures CI
      uses: snowplow-product/msc-schema-ci-action/check@v1
      with:
        organization-id: ${{ secrets.SNOWPLOW_ORG_ID }}
        api-key: ${{ secrets.SNOWPLOW_API_KEY }}
        manifest-path: 'snowplow-schemas.json'
        environment: ${{ env.ENVIRONMENT }}
```

View the [Github Action repository](https://github.com/snowplow-product/msc-schema-ci-action).

## Setting up for other deployment pipelines

### Prerequisites

- JRE 8 or above

### Download the CI tool

You can download Data Structures CI from our Bintray repository, using the following command:

```bash
$ curl -L https://github.com/snowplow-product/msc-schema-ci-tool/releases/download/1.0.0/data_structures_ci_1.0.0.zip | jar xv && chmod +x ./data-structures-ci
```

### Run the task

You can run the task using the following syntax:

```bash
$ export ORGANIZATION_ID=<organization-id>
$ export API_KEY=<api-key>
$ ./data-structures-ci check \
    --manifestPath /path/to/snowplow-schemas.json \
    --environment DEV
```

View the repository for [integration examples](https://github.com/snowplow-product/msc-schema-api-examples/).

---

# Test your Snowplow tracking and pipeline implementation
> Test and validate your Snowplow tracking implementation using Snowplow Inspector or Micro. Run automated tests and QA your pipeline before deploying to production environments.
> Source: https://docs.snowplow.io/docs/testing/

There are a number of ways you can test your tracking implementation and QA your pipeline to follow good data practices.

When implementing new tracking, or when making changes to your schemas or enrichments, we recommend you run testing by sending events to a test environment before deploying your changes to production environments. You can use the [Snowplow Inspector](/docs/testing/snowplow-inspector/) or [Snowplow Micro](/docs/testing/snowplow-micro/) to do this.

## Test web tracking using the browser extension

Use the [Snowplow Inspector](/docs/testing/snowplow-inspector/) browser extension to validate your web tracking implementation.

## Test schema and enrichment changes with Snowplow Micro

[Snowplow Micro](/docs/testing/snowplow-micro/) is a lightweight version of a Snowplow pipeline that you can use to validate that your schemas, tracking code and enrichments work as expected.

Micro can be deployed [through Console](/docs/testing/snowplow-micro/console/) or [locally](/docs/testing/snowplow-micro/local/) using Docker or Java.

Micro can also be used for [automated testing](/docs/testing/snowplow-micro/automated-testing/) in a CI/CD environment.

---

# Add custom schemas to Snowplow Inspector
> Configure Snowplow Inspector to access custom schemas from local registries, Data Structures API, Iglu Server, or static repositories. Validate events against private schemas by logging into Console or manually adding registries.
> Source: https://docs.snowplow.io/docs/testing/snowplow-inspector/adding-schemas/

By default, the extension is able to see [Iglu schemas](/docs/fundamentals/schemas/) for events and entities generated by the standard Snowplow [tracker](/docs/sources/) and plugin APIs, as they're publicly available on the public [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/) registry. The Iglu Central registry is a default configuration in all Snowplow Pipelines as a common base, so events validated in the Inspector against these schemas should also validate correctly in your pipeline.

For custom schemas that aren't publicly available in Iglu Central, you need to configure the extension to access your registries so it can access the schemas to perform validation.

If you don't log in or configure registries, the extension won't recognize or be able to validate any custom events/entities, though they will still display.

![The Snowplow Inspector extension displaying a Self Describing Event called \&quot;example\_event\&quot;. It contains a property \&quot;example\_field\_1\&quot; with the value \&quot;abc\&quot;. The event\&#39;s schema displays as Unrecognized in any registries.](/assets/images/unrecognized-event-41abdf23415d71f1454e56092ae757df.png)

Snowplow customers with access to the Snowplow Console can log in using the button in the bottom-left of the extension, which will automatically discover schemas defined for any [data structures](/docs/event-studio/data-structures/) set up in your development or production Iglu Server environments.

When installed, the extension also creates a local Iglu registry, which lets you iterate on schemas without having to deploy them to a external registry. You can create any number of additional local registries to help organize your work.

You can also configure the extension to access other private schema repositories (such as [Iglu Servers](/docs/api-reference/iglu/iglu-repositories/iglu-server/) or [static repositories](/docs/api-reference/iglu/iglu-repositories/static-repo/)).

## Manage schemas and registries

To access the full list of registries configured and find which schemas are accessible, choose **...** > **Manage Schemas**.

![Screenshot of the Snowplow Inspector\&#39;s \&quot;More Options\&quot; menu open and the \&quot;Manage Schemas\&quot; option selected.](/assets/images/access-manage-schemas-4bb3ef87b3c56471b369b0e932587600.png)

The extension will connect to your registries and request a listing of all available schemas.

![Screenshot of the Snowplow Inspector\&#39;s \&quot;Manage Schemas\&quot; page with the default Local and Iglu Central registries configured.](/assets/images/manage-schemas-5898f2f1952b3c1c724c23de0ae11093.png)

At the top is a search for filtering the shown schemas, controls for managing registries and local schemas, and then a list of your configured registries.

Underneath is a tree listing of all schemas configured, grouped by vendor, name, format, and version. Each version has a list of the registries that contained it.

Clicking a registry will toggle selecting the registry. The schema list below will filter to only show schemas within selected registries. If there are no registries selected, schemas from all registries will display. Any search terms will also only search schemas from the selected registries.

Using the **Registries...** button you can:

- Add more custom registries
- Import a list of registries from an existing [Iglu Resolver configuration](/docs/api-reference/iglu/iglu-resolver/)

If you have some registries selected, this button will also allow you to:

- Edit existing registries you've previously configured
- Remove configured registries so they aren't accessible to the extension any more

Selecting a local registry will activate the **Schemas...** button and allow you to manage the schemas it contains.

You can clear the search and registry selections to see all schemas again using the **Clear Filters** button.

## Add custom registries

You can add or edit different types of registry:

- Local
- Data structures API
- Iglu Server
- Static

Configuring a registry requires a name and one of these types. You may also optionally include a priority, and list of vendor prefixes. Each type of registry may require additional configuration like API keys to function.

Once configured, schemas from the registry load in the **Manage Schemas** view, and events using those schemas get validated against them.

Once the extension can query your private Iglu repositories it will be able to tell you in real time whether your events and entities are failing validation from entirely within the browser.

Here's an example of an event passing validation:

![The Snowplow Inspector extension displaying a Self Describing Event called \&quot;example\_event\&quot;. The event payload contains a \&quot;example\_field\_1\&quot; property with the value \&quot;abc\&quot;. The extension says the event is valid against a schema from the Local Registry, signalled by a checkmark.](/assets/images/event-passing-validation-9f31ef8956d51292f512f251bb9965c3.png)

Here's an example of an event failing validation. The failure reason is that the schema defines `example_field_1` as "string" type, meaning it can't be null.

![The Snowplow Inspector extension displaying a Self Describing Event called \&quot;example\_event\&quot;. The event payload contains a \&quot;example\_field\_1\&quot; property with a null value. The extension says the event is invalid against a schema from the Local Registry, signalled by a red cross.](/assets/images/event-validation-failed-8495885c08522e78ffee0b9457d09938.png)

Hovering over the registry displays text describing which aspect of the payload failed the validation, which is also available in the **Errors** tab. Find the **Errors** tab next to the **Data**, **JSON**, and **Schema** tabs within the event details pane. Clicking the registry copies this error to your clipboard.

### Local

Local registries are often used for testing changes of schemas still in development, or quickly including a schema when you don't have the full access required to configure a registry. The extension UI lets you manage the schemas, which it stores on your local machine.

The only required configuration is a name, and the schemas themselves.

You manage the schemas by selecting the registry and using the **Schemas...** button. You can manually edit the JSON in the extension, or bulk import JSON files from a directory on your machine.

### Data structures API

Data structures API registries use the [data structures API](/docs/event-studio/programmatic-management/data-structures-api/) included in Console.

Most users can just log in to [Console](https://console.snowplowanalytics.com) via the extension and it will import registries for any Snowplow Console organizations your account has access to.

If you need to provide access for someone that doesn't have a Console account, they can manually configure one using a Console API key.

In order to function, the extension requires:

- Organization ID: find this under **Settings** > **Manage organization** in Console
- API Key ID: go to **Settings** > **Manage organization** in Console to manage your API keys
- API Key: go to **Settings** > **Manage organization** in Console to manage your API keys

### Iglu Server

[Iglu Server](/docs/api-reference/iglu/iglu-repositories/iglu-server/) is a more full-featured dedicated service for hosting server that's more flexible than static registries.

You may need to add one of these registries if you are accessing the registry built into [Snowplow Mini](/docs/api-reference/snowplow-mini/).

To authenticate with your server, the extension will require:

- Iglu API endpoint: this is the base URL the extension will use when contacting the API. If you include a path component, the API request will be `api/*` relative to this path; you may need to add or remove trailing slashes if the API isn't hosted at the root.
- Iglu API key: see [API keys and the authentication service](/docs/api-reference/iglu/iglu-repositories/iglu-server/#5-api-keys-and-the-authentication-service-apiauth) for instructions on generating an Iglu API key.

When logged into Console, these details should be available in **Manage organization** > **API keys for utilities**.

### Static

Most other registries, such as those hosted as websites or via [S3](https://aws.amazon.com/s3/) or [GCS](https://cloud.google.com/products/storage/) buckets will use this [static](/docs/api-reference/iglu/iglu-repositories/static-repo/) type.

If you can access the schemas directly via a browser, this is probably the right choice.

Static registries managed with [`igluctl`](/docs/api-reference/iglu/igluctl-2/) version 8 or newer will include a manifest file listing all the schemas in a registry, e.g. [the Iglu Central manifest](https://iglucentral.com/schemas).

If the registry doesn't include a manifest listing the contained schemas, the extension will attempt to request all referenced schemas from static registries to see if it's included. In this case the registries will also not have their schemas listed in **Manage Schemas** - unless they're referenced by tracked events since you last opened the extension.

To connect to your static registry, the extension will need a **base URI** to use when requesting schemas. The extension will request schemas relative to this base, expecting a `schemas/{vendor}/{name}/{format/{version}` format.

Optionally you can also provide a **manifest URI** to use for the schema manifest file that lists the schemas contained in the registry. If not provided, defaults to `schemas`, to match the `igluctl` default. The manifest file is also known as a "schema list", as described in the [release notes](https://github.com/snowplow/igluctl/releases/tag/0.8.0).

---

# Import events into Snowplow Inspector from multiple sources
> Import events into Inspector from HAR files, failed events, ElasticSearch, ngrok tunnels, or remote debugging sessions. View and debug events from mobile devices and other sources as if generated locally.
> Source: https://docs.snowplow.io/docs/testing/snowplow-inspector/importing-events/

Within the Snowplow Inspector, the main **Events** view allows you to import events from other devices that you can view, as if your own browser generated them.

Import events via the **Import Events** icon button at the top of the events list. The extension supports several file types.

## Importing from HAR files

[HAR](https://en.wikipedia.org/wiki/HAR_\(file_format\)) files are JSON representations of HTTP sessions. They include a list of requests and responses made to servers by a single client, including all the metadata such as headers.

You can export the current contents of the **Network** panel in your DevTools to a HAR file, and then later load them into the extension as a record that events fired and successfully validated, or failed to validate. This can be useful for troubleshooting, or as a record of the QA process.

Aside from the browser, other tools like [Charles Proxy](https://www.charlesproxy.com/) or [Fiddler](https://www.telerik.com/fiddler), commonly used as proxies for verifying analytics from mobile applications, can also export to this format.

## Importing failed events

You can import historical events that have already failed the enrichment process into the extension to allow you to easily find the errors with the events.

The extension will accept an uncompressed file, which you can paste a selection from, or in total straight from your clipboard.

## Importing events from ElasticSearch / OpenSearch

If you use [ElasticSearch](https://www.elastic.co/) / [OpenSearch](https://opensearch.org/) as a destination for your events (or as used in [Snowplow Mini](/docs/api-reference/snowplow-mini/)) you can specify a query to use and the extension will load events as they're indexed.

This can be useful for testing many devices at once, e.g. multiple mobile devices that are all sending events to your Snowplow Mini instance.

## Importing events from an ngrok tunnel

[ngrok](https://ngrok.com/) is a service for creating ad-hoc network endpoints that can accept / tunnel requests and offers an API for other services to introspect and act on the requests that it received. Using ngrok, you can create an endpoint, use that endpoint as a Collector destination for your tracking, and then examine any events sent to it via the extension.

When you attempt to import from ngrok, the extension will attempt to connect to the ngrok tool running on your local machine on port 4040 (`localhost:4040`) to access the [ngrok Agent API](https://ngrok.com/docs/ngrok-agent/api/). If successful, any Snowplow events sent to the corresponding tunnel endpoint will appear in the extension.

## Remote debugging with Chrome DevTools

You can [remotely inspect](https://developer.chrome.com/docs/devtools/remote-debugging/) some non-web applications using the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). This includes some mobile browsers, some mobile applications, WebViews on Android, [Chromium Embedded Framework](https://github.com/chromiumembedded/cef) applications, and more. You may need to access the remote DevTools via the URL `chrome://inspect/#devices` to access DevTools with Snowplow Inspector and other browser extensions active. Network requests made on the remote device should appear in the extension as usual.

---

# Debug web tracking with the Snowplow Inspector browser extension
> Validate and debug Snowplow web tracking using the Inspector browser extension. View events, attributes, and interventions in real-time with schema validation support for Chrome, Edge, and Firefox.
> Source: https://docs.snowplow.io/docs/testing/snowplow-inspector/

Snowplow recommends using the Snowplow Inspector browser extension for validating your tracking code. To install in Chrome, Edge and other Chromium-based browsers, find [Snowplow Inspector on the chrome web store](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm).

> **Note:** Firefox users can install the extension manually using an `.xpi` file from the [GitHub Releases page](https://github.com/snowplow/chrome-snowplow-inspector/releases).
>
> The Snowplow Inspector is not supported on Safari at this time.

The code for the Snowplow Inspector is available on [GitHub](https://github.com/snowplow/chrome-snowplow-inspector).

Once you add the extension to your browser, you can view it by [opening Developer Tools](https://developer.chrome.com/docs/devtools/open/), where it has its own tab named **Snowplow** (look for the tab adjacent to **Console**, **Network**, etc. It may be necessary to expand the list of tabs using the `»` button).

> **Tip:** Common keyboard shortcuts to open Developer Tools include:
>
> - PC: `Ctrl`+`Shift`+`I`
> - Mac: `Cmd`+`Option`+`I`

## How it works

As you browse a webpage you can perform different actions that you can track as Snowplow events. These events get sent as HTTP requests to the Snowplow Collector that's configured for the tracker on each webpage. The Snowplow Inspector extension observes and parses these HTTP requests for display in a more easily readable format. This allows anyone with the extension to more easily see which actions on a webpage trigger Snowplow events to send to a Snowplow pipeline.

You can debug standard Snowplow events, and [Signals attributes and interventions](/docs/testing/snowplow-inspector/signals-integration/) using the Inspector. Choose which data type you want to look at using the vertical tabs on the left-hand side of the extension: **Events**, **Attributes**, and **Interventions**.

The **Events** tab shows the usual event debugging functionality. The inspector shows a list of all the events received in the order they were fired, with more recent events at the top. Click on each event to see its details.

In the example below, events fired as HTTP POST requests from the Snowplow homepage:

- An automatic [self-describing](/docs/fundamentals/events/#self-describing-events) [`web_vitals`](/docs/events/ootb-data/app-performance/#web-vitals) event.
- A [`page_view`](/docs/fundamentals/canonical-event/#page-views) event.
- A second `page_view` event from an embedded iframe on the page. It has a blue dot because its App ID and Collector combination are distinct from the other events, which have red dots because they share an App ID/Collector.
- A self-describing [`link_click`](/docs/sources/web-trackers/tracking-events/link-click/) event.
- Two [`page_ping`](/docs/events/ootb-data/page-activity-tracking/#page-engagement) events.

![A screenshot of the Snowplow Inspector extension listing several events observed on the current pageview. A selected page-view event displays, detailing the properties collected as a part of that event.](/assets/images/using-poplin-chrome-extension-d7c9abdb27e54813634c7b5985a54cf1.png)

Within the parsed HTTP request bodies that contain event payloads are a variety of [Snowplow canonical event](/docs/fundamentals/canonical-event/) fields, such as unique `event_id`, timestamps, user and session identifiers, as well as any custom [event](/docs/fundamentals/events/#self-describing-events) or [entity](/docs/fundamentals/entities/) fields.

## Try it out

Once you've installed the extension, you can try it out immediately by visiting a webpage with Snowplow tracking. For example, the [Snowplow website](https://snowplow.io/), or this documentation site. You don't need a Snowplow account to use the extension.

## Use in event QA

The Snowplow Inspector extension is useful for debugging any web tracking implementation. For questions such as "why is X event not appearing in the data warehouse", the extension allows you to see if the event is actually firing.

Additionally, you can configure the extension to show whether or not an event has passed validation according to any event validation rules codified in the corresponding [schema](/docs/fundamentals/schemas/).

For events that failed validation in production historically that you are unable to replicate in your own browser, see the guides on [how to query failed events](/docs/monitoring/exploring-failed-events/file-storage/). These failed events have a [specific format](/docs/fundamentals/failed-events/) that includes an array of helpful, detailed error messages that explain the exact reasons why the event failed validation. You can also [import bad events](/docs/testing/snowplow-inspector/importing-events/#importing-failed-events) into the extension to view as if your browser had generated them itself.

---

# Debug Signals attributes and interventions in Snowplow Inspector
> Connect Snowplow Inspector to Signals to validate attribute and intervention definitions. View real-time attribute values and monitor triggered interventions using the browser extension.
> Source: https://docs.snowplow.io/docs/testing/snowplow-inspector/signals-integration/

The Snowplow Inspector optionally integrates with [Snowplow Signals](/docs/signals/introduction/) to help validate your [attribute](/docs/signals/attributes/attributes/) and [intervention](/docs/signals/concepts/#interventions) definitions.

There are two ways to connect Inspector to Signals:

- Automatic: log into [Snowplow Console](https://console.snowplowanalytics.com) via the extension. The extension will detect Signals instances for any organizations you have access to. For full functionality you will need to add API credentials for each organization in the extension options.
- Signals Sandbox: if you're trialing Signals using the [Signals Sandbox](/tutorials/signals-interventions/start/#signals-sandbox), you can enter your Profiles API and credentials information in the extension options.

To access the extension options, use your browser toolbar to find the extensions menu, and access the **Options** for the Snowplow Inspector extension.

![Screenshot of the Chromium browser with the extension menu open. The installed Snowplow Inspector has its Options menu item selected.](/assets/images/extension-options-28e909a5bd99292f37454eb2683846fe.png)

After connecting to Signals, the Inspector will use the Signals API to discover any defined [attribute keys](/docs/signals/concepts/#attribute-keys), and the definitions for your attribute groups. It uses these definitions with any events it has observed in the debugger to build a set of attribute keys.

By default Signals includes `domain_userid` as an attribute key. Once the extension is aware of this, it will look at any events it observes, and build a list of unique `domain_userid` values it finds. It then uses this list of attribute keys for requesting [attributes](#attributes) and subscribing to [interventions](#interventions).

## Signals functionality

You can access Signals features by switching the vertical tabs down the left-hand side of the extension. The three tabs are: **Events**, **Attributes**, and **Interventions**.

**Events** is the usual event debugging functionality, and the other two interact with the Signals APIs.

### Attributes

When you switch to the **Attributes** page in the extension, it will combine the attribute group definitions from the Signals API with the observed attribute keys, and then request the current profile values for each attribute key.

![Screenshot of the Snowplow Inspector viewing attribute values from a \&quot;demo\_agentic\&quot; attribute group. The values are for a specific \&quot;domain\_userid\&quot; attribute key. The tabs show 20 events counted as well as 1 intervention.](/assets/images/signals-attributes-450c951be5e6fbab765dc2f12c16d495.png)

While you stay on the **Attributes** page, any new events that occur in the background (and increase the event counter on the **Events** tab) will automatically trigger a refresh. If there are no new events after 5 seconds, it will also do a final refresh, in case there was any latency in processing events and updating the values. Switching to the **Events** or **Interventions** tab and then back to the **Attributes** tab will initially show the most recently fetched values, while requesting any new updates in the background.

If the attribute values have still not updated or look out of date, you can use the refresh button at the top of the screen to manually trigger a refresh.

> **Note:** Since Signals Sandbox requests are free, attribute values get requested much more frequently than described here, to update as quickly as possible.

All attribute groups from all Signals instances get fetched if their configured attribute key has any values extracted from events. If you have access to many organizations, this may slow down the retrieval process. You can filter by environment, organization, or attribute source type to limit what's requested and displayed.

The controls at the top allow you to filter or search to filter what's displayed. Each attribute group listing includes its organization, environment, version, and source in the top-right for reference.

> **Note:** If you only have access to one organization, organization filters don't display. Similarly, if there is only a single version of an attribute group, there will be no option to change it.
>
> If configured, Signals Sandbox counts as its own organization.

By default, the Inspector only fetches the highest version of each attribute group. If an attribute group has more than one version available, you can switch to an older version you're interested in to request that instead.

You can click a displayed attribute row to show the JSON definition for that attribute. Click the same row again to hide the definition.

### Interventions

As the extension builds its list of attribute keys based on observed events, it will automatically start intervention subscriptions for each attribute key found.

If an intervention triggers on a subscription, the interventions counter will increase and you can switch to this tab to view the intervention content of each observed intervention.

Clicking an intervention from the list of received interventions on the will display the contents, including the targeted attribute key and any associated attributes at the time of the intervention.

For rule-based interventions, you can also expand the definition of the intervention to see the logic used to trigger it.

![Screenshot of the Snowplow Inspector viewing interventions. The most recent of two triggered interventions is displayed.](/assets/images/signals-attributes-450c951be5e6fbab765dc2f12c16d495.png)

---

# Set up automated testing with Snowplow Micro
> Integrate Snowplow Micro with automated testing frameworks like Nightwatch and Cypress. Build end-to-end GitHub Actions workflows to validate tracking implementations with custom commands and assertions.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/automated-testing/

The basic approach for using Snowplow Micro in automated testing is this:

- Run it alongside your tests, either via Docker or Java
- In each test, send some events then validate the results using the [Micro REST API](/docs/api-reference/snowplow-micro/api/)

> **Tip:** The instructions for running and configuring Micro in CI/CD are identical to the ones for [running locally](/docs/testing/snowplow-micro/local/).

The [snowplow-micro-examples](https://github.com/snowplow-incubator/snowplow-micro-examples) repository shows how to set up automated tests using Nightwatch and Cypress as examples of test frameworks and how to build end-to-end GitHub Actions testing workflows.

## Local setup

### Prerequisites

We recommend setting up the following tools before starting:

- Git
- Docker and Docker-compose
- Npm

### Clone the repo, start Snowplow Micro and serve the app

```bash
git clone https://github.com/snowplow-incubator/snowplow-micro-examples.git

cd snowplow-micro-examples

docker-compose up
```

This will:

1. Start serving the app on `localhost:8000`

2. Launch Snowplow Micro, mounting the `micro` directory and setting the port 9090 for accessing Micro's [REST API](/docs/api-reference/snowplow-micro/api/) endpoints. Inside the `micro` directory are:

   1. The [configuration for Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/micro.conf).
   2. The [configuration for Iglu resolvers](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/iglu.json).
   3. The `iglu-client-embedded` directory containing the custom schemas used for tracking.

### Install npm dependencies

```bash
npm install
```

This step will install Nightwatch and Cypress.

### Run the tests

Our demo web app uses snowplow to track user activity. We will be testing that tracking is properly configured on our site.

To run all tests:

```bash
npm test
```

For running only [Nightwatch](https://nightwatchjs.org/) tests:

```bash
npm run test:nightwatch
```

For running only [Cypress](https://www.cypress.io/) tests:

```bash
# From the command line
npm run cypress:run

# To launch the Test Runner
npm run cypress:open
```

## Github Actions

Inside the `.github/workflows/` directory you can find the `.yml` files we use to test this exaple app with Micro and Nightwatch/Cypress. A general workflow file would definitely use the [Snowplow Micro](https://github.com/snowplow/snowplow-micro) step, which for our example is:

```yaml
- name: Start Micro
    run: docker-compose up -d
    working-directory: snowplow-micro-examples
```

In order to use it, just make sure that:

1. Your `working-directory` contains the `micro/` directory that Micro will mount, containing a `micro.conf` and a `iglu.json` configuration for Micro and Iglu respectively.
2. Your `docker-compose.yml` file is adjusted accordingly

If you wanted to use `docker run` instead of `docker-compose`, the same step would be:

```yaml
- name: Start Micro
  run: docker run --mount type=bind,source=$(pwd)/micro,destination=/config -p 9090:9090 snowplow/snowplow-micro:4.1.1 --collector-config /config/micro.conf --iglu /config/iglu.json & \
  working-directory: snowplow-micro-examples
```

## Tracking design

Our demo web app uses Snowplow to track user activity. Then, using Snowplow Micro we will be testing that tracking is properly configured on our site using, as examples, two popular web test tools, Nightwatch and Cypress.

### Overview of the demo app

The example application is a simple ecommerce app consisting of just 3 pages:

1. A start-up page with a sample login form (no authentication involved - see also the **Note** below)
2. The shop page including a same-page cart
3. The purchase-confirmation page

> **Note:** The form of the login-page is only for demonstrating the tracking of form events (which can also show that password fields are not being tracked). There is no authentication involved. While you can type anything and continue to the shop-page, we recommend that you do not use any personal data.

Each page serves the purpose of demonstrating possible event-tracking, which will then be tested using Snowplow Micro and a test tool.

### Event dictionary

Using the [JavaScript Tracker](/docs/sources/web-trackers/):

```html
<!-- Snowplow starts plowing -->
<script type="text/javascript">
;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[];
p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments)
};p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1;
n.src=w;g.parentNode.insertBefore(n,g)}}(window, document, "script", "{% static 'ecommerce/js/sp.js' %}", "snowplow"));
```

The tracking implemented consists of:

- Pageview tracking

  - This event happens when a user visits a page.
  - It is a predefined Snowplow event, that automatically captures the URL, referrer and page title.

```javascript
window.snowplow('trackPageView');
```

- Activity Tracking

  - This event happens when a user engages with a page (e.g. user scrolls).
  - It is a predefined Snowplow event, that automatically records the maximum scroll left-right, up-down in the last ping period.
  - Method call - before the trackPageView method call:

```javascript
// login-page
window.snowplow('enableActivityTracking', {
    minimumVisitLength: 20,
    heartbeatDelay: 20
});

// shop-page:
window.snowplow('enableActivityTracking', {
    minimumVisitLength: 10,
    heartbeatDelay: 10
});
```

- Form Tracking

  - This set of events happens when a user interacts with a web form (e.g. focus on an form element, change a value of input-textarea-select element form, submit a form)

  - These are predefined Snowplow events that are of unstructured eventType and can be customized. Data captured:

    - **focus\_form**: id, classes of the form and name, type, value of the form element that received focus.
    - **change\_form**: name, type, new value of the element, id of the parent form
    - **submit\_form**: id, classes of the form and name, type, value of all form elements

  - **Note**: _By default, Form Tracking does not track password fields. We used the `options` to demonstrate how you can ensure the Non-tracking of sensitive fields. With Snowplow Micro we can also later test that any denylisting of forms is implemented correctly and that no sensitive fields are tracked._

```javascript
var options = {
    forms: {
        denylist: []
    },
    fields: {
        denylist: ['user_password']
    }
};

window.snowplow('enableFormTracking', { options: options });
```

- Tracking custom self-describing(unstructured) events

  1. cart-events ([schema](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/iglu-client-embedded/schemas/test.example.iglu/cart_action_event/jsonschema/1-0-0))

     - These events happen when a user interacts with the cart, adding or removing items, using the Add-to-cart or Remove buttons.
     - This is a self-describing event that captures the type of cart interaction: "add" versus "remove".
     - We also want to add as [custom context](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-custom-entity) the product involved in the cart-event, which is described by the product entity ([schema](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/iglu-client-embedded/schemas/test.example.iglu/product_entity/jsonschema/1-0-0), see more below)
     - Implemented in the shop-page (see file [shoppage.js](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/app/static/ecommerce/js/shoppage.js)):

```javascript
// TRACK cart_action_event (add)
window.snowplow('trackSelfDescribingEvent', {
    event: {
        schema: 'iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0',
        data: {
            type: 'add',
        },
    },
    context: [
        {
            schema: 'iglu:test.example.iglu/product_entity/jsonschema/1-0-0',
            data: {
                sku: sku,
                name: title,
                price: parseFloat(price),
                quantity: parseInt(quantity)
            }
        }
    ]
});

// TRACK cart_action_event (remove)
window.snowplow('trackSelfDescribingEvent', {
    event: {
        schema: 'iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0',
        data: {
            type: 'remove',
        },
    },
    context: [
        {
            schema: 'iglu:test.example.iglu/product_entity/jsonschema/1-0-0',
            data: {
                sku: sku,
                name: title,
                price: parseFloat(price),
                quantity: parseInt(quantity)
            }
        }
    ]
});
```

- purchase-event ([schema](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/iglu-client-embedded/schemas/test.example.iglu/purchase_event/jsonschema/1-0-0))

  1. - These events happen when a user completes the purchase of the products in their cart by clicking the Purchase button.
     - This is a self-describing event that captures the total amount of the transaction.
     - We also want to add as custom contexts the products involved, each of which is described by the product entity ([schema](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/micro/iglu-client-embedded/schemas/test.example.iglu/product_entity/jsonschema/1-0-0))
     - Implemented in the shop-page as well

```javascript
// create the contexts array
let productsContext = [];
userCart.forEach(function(elt) {
    productsContext.push({
        schema: 'iglu:test.example.iglu/product_entity/jsonschema/1-0-0',
        data: {
            sku: elt.itemSku,
            name: elt.itemTitle,
            price: parseFloat(elt.itemPrice),
            quantity: parseInt(elt.itemQuant)
        }
    });
});

// TRACK purchase_event
window.snowplow('trackSelfDescribingEvent', {
    event: {
        schema: 'iglu:test.example.iglu/purchase_event/jsonschema/1-0-0',
        data: {
            total: parseFloat(total),
        },
    },
    context: productsContext
});
```

- `webPage` [predefined Context](/docs/sources/web-trackers/tracker-setup/initialization-options/). Note: The webPage predefined context is enabled by default in JavaScript Tracker v3, as it is also a prerequisite for the official [Snowplow web data model](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/legacy/dbt-web-data-model/).

## Testing with Snowplow Micro

The purpose of this repo is to show how Snowplow Micro can be used to validate that the tracking implemented in a demo app is operating in the way we expect.

Simulate particular situations, and check that:

- There are no bad events.
- The data sent is as expected.
- The right event data is sent with each event.
- The right entities / contexts are attached to each event.
- The right values are sent with each event.

In this repo you can also find information on how to set up trackers, how to make customised (unstructured) events, and, mainly, how to configure and add tests in your test tool of choice, using as examples the popular web test tools Nightwatch and Cypress, in order to demonstate the capabilities of Snowplow Micro.

### The tests implemented

1. No bad events
2. Number of good events = number of expected good events
3. Ensuring that total number of events is right - expected number of good events + bad events (noBadEvents = True)
4. Checking proper values of structured and unstructured events are sent to Micro
5. Event With Property test - context, event type and schema
   - user puts in an event and we match by all 3 conditions (contexts, properties and schema - this determines whether or not the fake test event is equal to the one on micro - for both structured and unstructured)
6. Race condition test - to ensure that event x is always sent to Micro before event y (in our case we wanted to ensure cart action occurred before purchase)
7. Form tracking test to ensure blacklisted form fields are not tracked, and to ensure the right properties are sent for each field

## Snowplow Micro and Nightwatch

Powered by Node.js, [Nightwatch.js](https://nightwatchjs.org/) is an open-source automated testing framework that aims at providing complete E2E (end to end) solutions to automate testing with Selenium Javascript for web-based applications, browser applications, and websites.

This framework relies on Selenium and provides several commands and assertions within the framework to perform operations on the DOM elements.

### How does Nightwatch JS work?

Nightwatch communicates over a restful API protocol that is defined by the W3C WebDriver API. It needs a restful HTTP API with a Selenium JavaScript WebDriver server. In order to perform any operation i.e. either a command or assertion, Nightwatch usually requires sending a minimum of two requests.

It works as follows:

- The first request locates the required element with the given XPath expression or CSS selector.
- The second request takes the element and performs the actual operation of command or assertion.

### Getting Started

If you want to isolate Nightwatch JS testing, follow these steps below.

Install nightwatch and a chrome driver which enables nightwatch to interact with the Chrome browser:

```bash
npm install nightwatch --save-dev
npm install chromedriver --save-dev
```

1. Create default json package
2. Create node modules folder and add nightwatch to dev dependencies
3. Add nightwatch.conf.js file (must contain basic configuration file)
4. Add a chrome driver binary for nightwatch configs - adds it into node modules - allows us to send commands to the chrome driver
5. Add `test:nightwatch` to the scripts section of the package.json
6. Write your own tests

Running nightwatch:

```bash
npm run test:nightwatch
```

### Simulating a User with Nightwatch

When using Nightwatch, one testing strategy is to simluate a user interaction with your app. This is useful so that we can fire the exact events that a user would fire when using the app in the field.

In Nightwatch a test involves 3 phases:

1. Preparing a state:

   1. Reset Micro command - deleting the cache in Micro so that each test has independent events from other tests
   2. Configure Nightwatch to interact with your app

2. Taking an action: Nightwatch interacts with the app creating events

3. Making an assertion on the resulting state: Nighwatch requests the state of Micro and based on a test performs the corresponding assertions

### Organisation of tests

#### Commands

##### resetMicro

The resetMicro command can be added before each test as the beforeEach hook: [Nightwatch Test Hooks](https://github.com/nightwatchjs/nightwatch-docs/blob/643e9a63a94deba6bd84bf2dd78cb27693620e20/guide/using-nightwatch/using-test-hooks.md)

_Example call_:

```javascript
module.exports = {
    beforeEach: function(browser) {
        browser
            .resetMicro();
    },
    "test1" :function (browser0 {
        // code here
    }
}
```

_Arguments_: None

This command utilises the ability to reset Micro through the `/micro/reset` endpoint.

This is the general structure of how to create a command which aims to request information from Snowplow Micro using Nightwatch:

```javascript
this.command = (callback) => {
    const request = require('request');

    request(
        {
            url: 'http://localhost:9090/micro/all',
            json: true
        },
        (err, res, body) => {
            if (err) {
                console.warn(error);
                return false;
            }
            callback(body);
        }
    );
};
```

#### Assertions

##### .noBadEvents

_Example call_:

```javascript
browser.assert.noBadEvents();
```

_Arguments_: None

This is arguably the most important assertion, if you only implement one this is a great place to start. It ensures that all of your data is sent to Micro and ends up in good events, so all of your data is in your warehouse and you can interpret it as expected.

##### .noOfGoodEvents

_Example call_:

```javascript
browser.assert.noOfGoodEvents(2);
```

_Arguments_: Number of expected good events to be sent to Micro

An extension of noBadEvents, asserts that you sent the correct number of good events to Micro for a given event. For example, you might expect the onClick() action to send 2 good events, then you can make sure 2 are sent to good events and call noBadEvents on the same assertion as total number of events = number of good events + number of bad events.

##### .noOfTotalEvents

_Example call_:

```javascript
browser.assert.noOfTotalEvents(2);
```

_Arguments_: Number of expected total events sent to Micro

A further extension on noOfGoodEvents, this assertion ensures that both the correct number events are sent to Micro, and that no bad events are sent. Using all 3 of these assertions consecutively provides the best assurance that every event you send is sent to your warehouse properly.

##### .orderOfEvents

_Example call_:

````javascript
browser.assert.orderOfEvents(events_list);

```javascript

_Arguments_: events_list

Checks for when one event must be before the other so that your application works as expected. This works by retrieving the derived timestamps for each event, and asserting that events fired in the specified order. In our case we use this to test that a cart action occurs before purchasing an item. If our application didn't get this order of events correct, then the application does not act as expected. This can also be considered a race condition test.

##### .successfulEvent

_Example call_:

```javascript
browser.assert.successfulEvent(
    {
        "eventType": "unstruct",
        "schema": "iglu:test.example.iglu/cart_action_event/jsonschema/1-0-0",
        "values": {
            "type": "remove"
        },
        "contexts": [
            {
                "schema": "iglu:test.example.iglu/product_entity/jsonschema/1-0-0",
                "data": {
                    "name": "One-size summer hat",
                    "price": 15.5,
                    "quantity": 1
                }
            }
        ]
    }
);
````

_Arguments_: test\_event (with schema, eventType, values and context), number\_of\_occurences

Ensures that when an event is sent to Micro the correct parameters are sent as expected. In our case we check that the schema, properties and contexts are correct:

- We match what the user expects to be sent to what is received on Micro.
- In the end, the number of matched events is retrieved and asserted to the expected number of occurrences.
- By doing that, we can also specify which events we don't expect on Micro by setting the argument number\_of\_occurences=0.

## Snowplow Micro and Cypress

[Cypress](https://www.cypress.io/) is an open [source](https://github.com/cypress-io/cypress) JavaScript End-to-End testing framework with extensive [documentation](https://docs.cypress.io/). In this section we note few things that are specifically related to using this test tool with Snowplow Micro, describe the rationale behind the tests' organization used in this example and document the commands used.

### Introduction

Generally, a test involves 3 phases:

1. Prepare a state
2. Take an action
3. Assert on the resulting state

While Cypress considers as state the application's state, it is still a fact that an app is rarely an isolated system without side effects.

This is especially true when a tracking strategy is implemented, which means that any action can fire [events](/docs/events/) through the [trackers](/docs/sources/). There is an increasing number of reports highlighting the importance of upstream data quality and Data Ops, which means that testing your Data Collection and trackers' implementation besides your product's features is of highest priority. Tracking is as important as your shipping, and that is why it is highly recommended that you include its testing in your E2E tests.

Cypress, even though it considers as [best practice](https://docs.cypress.io/guides/references/best-practices.html#Visiting-external-sites) to avoid requiring or testing 3rd party services, it still offers the ability to "talk" to 3rd party API's via [cy.request()](https://docs.cypress.io/api/commands/request.html), that:

- Entirely bypasses CORS
- Expects the server to exist and provide a response
- Does not retry its assertions (as that could affect external state) , which makes it great to use for querying Snowplow Micro's endpoints. For example:

```javascript
cy.request({
    url: 'http://localhost:9090/micro/all',
    json:true
});
```

So, following on the 3 test's phases:

1. Preparing state:

   1. Reset Micro
   2. Configure Cypress to visit your app

2. Actions: Cypress interacts with the app creating events

3. Assertions: Cypress sends requests to Micro, and attempts assertions on the responses

### Tests' organization

Another Cypress' recommendation for best [practices](https://docs.cypress.io/guides/references/best-practices.html#Having-tests-rely-on-the-state-of-previous-tests) is the decoupling of tests, which, for the case of testing with Snowplow Micro, would mean to run both the state-changing and the micro-requests in the same spec file. However, there were some issues in doing so. More specifically, those issues had only to do with cases where links (or submit buttons) were clicked, in other words in cases where a window [unload event](https://developer.mozilla.org/en-US/docs/Web/API/Window/unload_event) was fired.

To describe the issue, we first describe what normally happens upon an unload event: When a user clicks, for example, a link, on one hand the browser wants to navigate to the link, and on the other hand, the tracker (in our case the [Javascript Tracker](/docs/sources/web-trackers/)) tries to send the [link click](/docs/sources/web-trackers/tracking-events/link-click/) or the [submit form](/docs/sources/web-trackers/tracking-events/form-tracking/) events, while also storing them in local storage, just in case the events don't get sent before the page unloads. While it is normal for browsers to cancel all requests, a cancelled request does not necessarily mean that the request did not reach the server, but that the client sending it, does not wait for an answer anymore. So, there is no way to know from client side whether the request (be it POST or GET) succeeded.

That problem was especially apparent when Micro was being queried in the same spec file with the app's actions. For example, POST requests appeared as cancelled in Cypress' test runner, but the events may have reached Micro. Taking advantage of the fact that Cypress also clears browser cache when it changes spec file, we decided to move the testing part of Micro into separate spec files.

The consequences of this decision are:

1. You need to ensure a naming strategy, and the reason for this is the fact that Cypress decides the order of execution for test files based on alphabetical order. In this example, we use this naming strategy:

- `xx_app_spec.js` for the spec files that visit the app and create events
- `xx_micro_spec.js` for the spec files that query Micro and make the assertions on those events , where `xx` stands for numbering (but it could be anything as long as it matches uniquely).

```bash
$ tree testing/cypress/integration

testing/cypress/integration/
├── 01_app_spec.js
├── 01_micro_spec.js
├── 02_app_spec.js
├── 02_micro_spec.js
├── 03_app_spec.js
├── 03_micro_spec.js
```

2. You need to consider how and when you reset Micro. We chose to use a Before [hook](https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests.html#Hooks) in the start of every `xx_app_spec.js` file, since the naming strategy ensures that the tests will run in `app`-`micro` pairs.
3. If you just want to run only a particular app test file (and not all of them), you will also need its corresponding Micro test file. Since this is a usual case, for example, when a particular spec file fails, we added another npm script `cy-micro:pair-run`, which will match a naming pattern and run the corresponding micro-spec file after the matching app spec. The script can be seen in [package.json](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/package.json).

Example usage:

1. To run all spec files in your `cypress/integration` directory (in alphabetical order)

```bash
npm run cypress:run
```

2. To run an app-micro pair of spec files given a name pattern

```bash
PAIR_PATTERN=03 npm run cy-micro:pair-run  # will run first the 03_app_spec.js and then the 03_micro_spec.js
```

Just make sure that this name pattern is unique for this pair.

This kind of organization also has the benefit, that you can keep having the tests you normally had for your app (just adding the before-hook to reset Micro), and just add a corresponding micro-spec file, to test the events emitted from the original run of that app test. That way, you can test your app's features in the `app_spec` files and your tracking implementation in the corresponding `micro_spec` files.

### Commands

Since Cypress allows to define your own [custom commands](https://docs.cypress.io/api/cypress-api/custom-commands.html), in this repo you can find commands specifically for use with Snowplow Micro and assertions of events. You can see them all in [commands.js](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/testing/cypress/support/commands.js).

#### cy.noBadEvents

_Example call_:

```javascript
cy.noBadEvents();
```

Even if this is the only thing that you check in your tests, you are already brilliant. It is going to ensure that your app is not sending any bad events, in other words you ensure that all your events end up in your warehouse. There are no more gaps in your data or in your analytics and no recovery jobs to get those bad events back, jobs that are not going to be trivial, especially if you are dealing with high volume of events.

#### cy.numGoodEvents

_Example call_:

```javascript
cy.numGoodEvents( 19 );
```

This command ensures that all the events you want to track, actually get tracked. Because there may be not bad events, but maybe your tracker is not implemented correctly into the app's logic.

#### cy.eventsWithEventType

_Example call_:

```javascript
cy.eventsWithEventType( "page_view", 8 );
cy.eventsWithEventType( "struct", 45 );
```

This command is useful when you want to ensure that a particular type of events got tracked as many times as it should.

#### cy.eventsWithParams

_Example call_:

```javascript
cy.eventsWithParams(
    {
        "event": "struct",
        "se_category": "Media",
        "se_action": "Play video",
        "se_label": "Surfing"
    },
    3
);
```

This command accepts as first argument an object with the expected event's field-value pairs. You can read about all the fields in Snowplow docs [here](/docs/fundamentals/canonical-event/). This command is particularly useful when checking on [structured events](/docs/sources/web-trackers/custom-tracking-using-schemas/#track-a-structured-event).

#### cy.eventsWithSchema

_Example call_:

```javascript
cy.eventsWithSchema( "iglu:com.snowplowanalytics.snowplow/submit_form/jsonschema/1-0-0", 5 );
```

With this command you can look specifically for [self-describing events](/docs/fundamentals/events/#self-describing-events), which include both custom self-describing events and all other out-of-the-box Snowplow events that are of "unstruct" eventType (link-click, submit-form, ad-impression etc.)

#### cy.eventsWithContexts

_Example call_:

```javascript
cy.eventsWithContexts( [ { "schema": "iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0" } ], 10 );

cy.eventsWithContexts(
    [
        {
            "schema": "iglu:com.example.eg/article_context/jsonschema/1-0-0",
            "data": {
                "writer": "John Doe",
                "category": "Sports",
                "title": "The match of the year"
            }
        },
        {
            "schema": "iglu:com.example.eg/writer_entity/jsonschema/1-0-0",
            "data": {
                "name": "John Doe",
                "age": 34,
                "numOfArticles": 50,
                "categories": ["sports", "history", "food"]
            }
        }
    ],
    2
);
```

With this command you can check whether the predefined(e.g. webpage, geolocation) or custom contexts/entities got properly attached to events. You can not only check by the schema of the entities but also by their data. Note that the first argument to this command should be an array of objects, like the contexts' array that can be attached to any Snowplow event. The keys of these objects can be either "schema" or "data". For "schema" the value should be a string (the schema). For "data" the value should be an object of key-value pairs, depending on the context.

#### cy.eventsWithProperties

_Example call_:

```javascript
cy.eventsWithProperties(
    {
        "parameters": {
        "event": "unstruct",
        "v_tracker": "js-3.1.3"
        },
        "schema": "iglu:com.example.eg/custom_cart_event/jsonschema/1-0-0",
        "values": {
            "type": "add",
            "productSku": "12345",
            "quantity": 1
        },
        "contexts": [
            {
                "schema": "iglu:com.example.eg/custom_product_context/jsonschema/1-0-0",
                "data": {
                    "sku": "12345",
                    "name": "Laptop",
                    "onOffer": false,
                },
            }
        ]
    },
    1
);
```

This is a command that combines some of the above (eventsWithSchema, eventsWithParams, eventsWithContexts), and also adds the ability to look into the data of unstructured events. The object that gets passed as the first argument, can have as keys:

- "schema" : matches by schema
- "values" : matches by the data of an unstructured event
- "contexts" : matches by contexts
- "parameters" matches by the event's fields It will return the events that have all those properties. As shown in the examples above, you do not have to use all the properties, and the command works accordingly.

#### cy.eventsWithOrder

_Example call_:

```javascript
cy.eventsWithOrder([
    {
        "schema": "iglu:com.snowplowanalytics.snowplow/focus_form/jsonschema/1-0-0",
        "values": {"elementId": "user_email"}
    },
    {
        "schema": "iglu:com.snowplowanalytics.snowplow/change_form/jsonschema/1-0-0",
        "values": {"elementId": "user_email"}
    },
    {
        "schema": "iglu:com.snowplowanalytics.snowplow/submit_form/jsonschema/1-0-0"
    }
]);
```

With this command you can assert that events happened in a specified (ascending) order. For example, in the call above, we can assert that the focus\_form event happened before the corresponding change\_form event, which in turn happened before the submit\_form event. The argument to this command is an array of at least 2 event "descriptions", which are exactly like the properties' object argument of `eventsWithProperties`. Those event descriptions need to uniquely identify exactly one Snowplow event. Internally, this command compares events' `derived_tstamp`.

### Some further notes

```bash
$ tree testing/cypress/

testing/cypress/
├── integration
│   ├── 01_app_spec.js
│   ├── 01_micro_spec.js
│   ├── 02_app_spec.js
│   ├── 02_micro_spec.js
│   ├── 03_app_spec.js
│   ├── 03_micro_spec.js
│   └── helpers_spec.js
├── plugins
│   └── index.js
└── support
    ├── commands.js
    └── index.js
```

1. Helpers

   - In the `commands.js` file, the commands are defined based on the `Micro` [helper module](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/testing/jsm/helpers.js)
   - The `integration/helpers_spec.js` [file](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/testing/cypress/integration/helpers_spec.js) tests the helper functions that define the matching logic. For a different custom matching logic, you can tweak the helper functions, and then test them.

2. Environment variables.
   - Cypress allows for [many ways](https://docs.cypress.io/guides/guides/environment-variables.html) to set environment variables. In this example we set them in the `plugins/index.js` [file](https://github.com/snowplow-incubator/snowplow-micro-examples/blob/main/testing/cypress/plugins/index.js).

---

# Run Snowplow Micro through Console
> Run Snowplow Micro through Snowplow Console to validate and debug tracking implementations. Send events to the collector endpoint and view results through the dashboard.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/console/

Snowplow Micro is fully integrated in Snowplow Console:

- You can deploy one or more Micro-based **development environments**
- Each environment is automatically connected to your development [data structures](/docs/event-studio/data-structures/)
- You can manage [enrichments](/docs/pipeline/enrichments/available-enrichments/) the same way as for regular pipelines, and copy enrichment configuration from a development environment to a pipeline

> **Tip:** Using a development environment is a great way to test your changes before applying them to a real pipeline.

## Setup

To create a development environment, navigate to **Settings > Workspaces**, select your workspace, scroll to the **Development environments** section and click **Create environment**.

> **Note:** You will need _Edit environments_ permissions to do this.

You can add multiple environments, or create and delete them as you see necessary. For example, it might be convenient to create separate environments for different teams testing different code in parallel.

Under the hood, an instance of Snowplow Micro will be deployed in your cloud account along with a database to store events. (This account is managed by you, in the case of Private Managed Cloud, or by Snowplow otherwise.)

## Usage

Once your development environment is ready, you can access it from the **Pipelines** section in the Console sidebar.

Select your environment and you will see the Collector endpoint URL you can use in your tracking code to send events to this environment.

Events will be stored for a (rolling) 7 day period.

To view events in the [Micro dashboard](/docs/testing/snowplow-micro/ui/), select your environment and then click **Open dashboard**.

> **Warning:** Do not send production data to development environments. Anyone with the _View environments_ permission can access the dashboard and see the events. Also, development environments are not configured to withstand high volumes of events.

To enable or disable enrichments and edit their configurations, select the **Enrichments** tab. The process is the same as for regular pipelines. We highly recommend testing enrichments in a development environment before deploying them to production.

---

# Test and debug tracking with Snowplow Micro
> Snowplow Micro is a lightweight test pipeline for debugging and automated testing. It receives, validates, and enriches events with a UI and API for inspection.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/

[Snowplow Micro](https://github.com/snowplow/snowplow-micro) is a lightweight version of the Snowplow pipeline. It's great for:

- Getting familiar with Snowplow
- Debugging and testing, including [automated testing](/docs/testing/snowplow-micro/automated-testing/)

![Snowplow Micro dashboard](/assets/images/overview-1728998b503b68a664fe3b2d8c0e36b8.png)

Just like a real Snowplow pipeline, Micro receives, validates and enriches events sent by your [tracking code](/docs/sources/).

> **Warning:** Micro is not designed for production traffic.

To get started in minutes, [deploy Micro through Console](/docs/testing/snowplow-micro/console/) or [run it locally](/docs/testing/snowplow-micro/local/).

---

# Configure additional Snowplow Micro settings
> Enable HTTPS with SSL/TLS certificates, add custom Iglu resolver configurations, and modify collector settings. Advanced configuration options for Snowplow Micro using bind mounts and environment variables.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/local/advanced-usage/

This page describes additional configuration options for Snowplow Micro.

## Enabling HTTPS

While in most cases HTTP is sufficient, you may want to enable HTTPS in Micro (for an example of when that’s useful, see [Locally resolving an existing domain name to Micro](/docs/testing/snowplow-micro/local/remote-usage/#locally-resolving-an-existing-domain-name-to-micro)).

You will need an SSL/TLS certificate in [PKCS 12](https://en.wikipedia.org/wiki/PKCS_12) format (`.p12`). Pass your certificate file and its password to the container (using a [bind mount](https://docs.docker.com/storage/bind-mounts/) and an [environment variable](https://docs.docker.com/compose/environment-variables/)). Don’t forget to expose the HTTPS port (by default, 9543):

```bash
docker run -p 9090:9090 -p 9543:9543 \
--mount type=bind,source=$(pwd)/my-certificate.p12,destination=/config/ssl-certificate.p12 \
-e MICRO_SSL_CERT_PASSWORD=... \
snowplow/snowplow-micro:4.1.1
```

> **Note:** For the certificate, the path inside the container must be exactly `/config/ssl-certificate.p12`.

You should see a message like this in the logs:

```text
[INFO] com.snowplowanalytics.snowplow.micro.Main$ - HTTPS REST interface bound to /0.0.0.0:9543
```

As usual, you can change the ports to your liking (see [Running Micro](/docs/testing/snowplow-micro/local/)).

## Adding custom Iglu resolver configuration

If you’d like to tweak the Iglu registries Micro uses, the priority between them, the cache sizes, etc, you can provide your own [Iglu resolver configuration](/docs/api-reference/iglu/iglu-resolver/) (`iglu.json`).

> **Tip:** If you are just looking to add custom schemas or connect to your private Iglu registry, check out [Adding custom schemas](/docs/testing/snowplow-micro/local/schemas/) for simpler ways to achieve that.

Pass your configuration file to the container (using a [bind mount](https://docs.docker.com/storage/bind-mounts/)) and instruct Micro to use it:

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/iglu.json,destination=/config/iglu.json \
snowplow/snowplow-micro:4.1.1 \
--iglu /config/iglu.json
```

That’s it. You can use the [the API](/docs/api-reference/snowplow-micro/api/#microiglu) to check if Micro is able to reach your schemas (replace `com.example` and `my-schema` as appropriate).

```bash
curl localhost:9090/micro/iglu/com.example/my-schema/jsonschema/1-0-0
```

## Adding custom collector configuration

If you’d like to tweak the [collector configuration](/docs/api-reference/stream-collector/configure/) inside Micro, the simplest approach is to override individual settings. For example, to change the cookie name:

```bash
docker run -p 9090:9090 \
snowplow/snowplow-micro:4.1.1 \
-Dcollector.cookie.name=sp
```

For more extensive changes, you can also bring your own configuration file (`micro.conf`).

**Example**

```hcl
loading...
```

[View on GitHub](https://github.com/snowplow/snowplow-micro/blob/master/example/micro.conf)

Pass your configuration file to the container (using a [bind mount](https://docs.docker.com/storage/bind-mounts/)) and instruct Micro to use it:

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/micro.conf,destination=/config/micro.conf \
snowplow/snowplow-micro:4.1.1 \
--collector-config /config/micro.conf
```

## Persisting events across restarts

By default, Micro only stores events in memory. Since version 4.0.0, you can connect it to a PostgreSQL database to persist events across restarts.

To enable this, first create a storage configuration file:

```hcl
host = "localhost"
port = 5432
database = "micro_test"
user = "test_user"

// How long to keep the events for (minimum: 5m)
ttl = "7d"

// How often to clean up expired events (minimum: 1m)
cleanupInterval = "1h"
```

Next, place the database password into an [environment variable](https://en.wikipedia.org/wiki/Environment_variable) named `MICRO_POSTGRESQL_PASSWORD`.

Now you can run Micro and pass the configuration, as well as the password:

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/storage.conf,destination=/config/storage.conf \
-e MICRO_POSTGRESQL_PASSWORD \
snowplow/snowplow-micro:4.1.1 \
--storage /config/storage.conf
```

---

# Enable enrichments in Snowplow Micro
> Add enrichments to Snowplow Micro including YAUAA, IP Lookup, and JavaScript enrichment. Mount enrichment configuration files to test pipeline behavior locally.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/local/enrichments/

By default, when running locally, Micro does not have any [enrichments](/docs/pipeline/enrichments/available-enrichments/) enabled. This page shows how you can enable them.

> **Tip:** When [running Micro through Console](/docs/testing/snowplow-micro/console/), you can use the UI to configure enrichments.

## YAUAA (Yet Another User Agent Analyzer)

One of the most useful enrichments, [YAUAA](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/), requires no configuration, and can be turned on simply with the `--yauaa` flag (since version 3.0.1):

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --yauaa
```

> **Tip:** We recommend not enabling this enrichment in a CI setting unless necessary, because it consumes a few extra hundred MB of RAM.

## Other enrichments

You can enable any enrichments you like by passing corresponding configuration files to Micro.

**Limitations for enrichments that rely on data files**

Some enrichments require data files (e.g. a database of IPs).

The Enrich application in a full Snowplow pipeline will automatically download and periodically update these files. However, Micro will only download them once. You can always restart Micro to get a fresher copy of the files.

You might also face access limitations where the database files are hosted on S3, GCS, or ADLS and are authenticated via the cloud environment roles and permissions.

For example, let’s say that you want to configure the [IP Lookup enrichment](/docs/pipeline/enrichments/available-enrichments/ip-lookup-enrichment/). The default configuration file looks like this:

```json
loading...
```

[View on GitHub](https://github.com/snowplow/enrich/blob/master/config/enrichments/ip_lookups.json)

Put this file somewhere on the machine where you are running Micro, let’s say `my-enrichments/ip_lookups.json`. (Feel free to add any other configurations to `my-enrichments`).

Now you will need to pass this directory to the Docker container (using a [bind mount](https://docs.docker.com/storage/bind-mounts/)):

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/my-enrichments,destination=/config/enrichments \
snowplow/snowplow-micro:4.1.1
```

> **Note:** The directory _inside_ the container (what goes after `destination=`) must be exactly `/config/enrichments`.

Alternatively, if you are running Micro as a Java application, put your enrichment configurations in `some-directory/enrichments` on your machine (`enrichments` must be called exactly that) and use the following command:

```bash
java -cp micro-4.1.1.jar:some-directory com.snowplowanalytics.snowplow.micro.Main
```

Once Micro starts, you should see messages like these:

```text
[INFO] com.snowplowanalytics.snowplow.micro.Run - Downloading http://snowplow-hosted-assets.s3.amazonaws.com/third-party/maxmind/GeoLite2-City.mmdb...
[INFO] com.snowplowanalytics.snowplow.micro.Run - Enabled enrichments: IpLookupsEnrichment
```

> **Tip:** Micro is especially great for [testing the JavaScript enrichment](/docs/pipeline/enrichments/available-enrichments/custom-javascript-enrichment/testing/).

---

# Run Snowplow Micro locally
> Run Snowplow Micro locally using Docker to validate and debug tracking implementations. Send events to port 9090 and view results through the dashboard, API, or exported TSV/JSON format.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/local/

> **Tip:** For general use, we recommend [running Micro through Snowplow Console](/docs/testing/snowplow-micro/console/). However, running locally can be useful if you don’t have access to Console or need an advanced configuration.

The easiest way to run Micro locally is through [Docker](https://www.docker.com/).

Run the following command:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1
```

You should see output like this:

```text
[INFO] com.snowplowanalytics.snowplow.micro.Run - No enrichments enabled
[INFO] com.snowplowanalytics.snowplow.micro.MicroHttpServer - Building blaze server
[INFO] org.http4s.blaze.channel.nio1.NIO1SocketServerGroup - Service bound to address /[0:0:0:0:0:0:0:0]:9090
```

**A note on ports...**

The command above will route port `9090` on your machine to Micro. If that port is already taken, you will want to change it, like so:

```bash
docker run -p 5000:9090 snowplow/snowplow-micro:4.1.1
            ↑↑↑↑
```

Note that Micro will still log `REST interface bound to /0.0.0.0:9090` — `9090` here refers to the port _inside_ the container.

We will use `9090` in the examples below, but remember to substitute the port of your choosing.

> **Tip:** We also provide a _distroless_ image of Micro. Because it only includes the bare minimum, it’s smaller and more secure. However, the downside of using the distroless image is that basic utilities (such as a shell) are not available.
>
> To use this image, add `-distroless` to the tag:
>
> ```bash
> docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1-distroless
> ```

If Docker is not available on your system, you can also run Micro as a Java application (since version 2.3.1). The `jar` files are availble via [GitHub releases](https://github.com/snowplow/snowplow-micro/releases).

```bash
java -jar micro-4.1.1.jar
```

Most of the examples on this and the following pages will show the Docker option, but feel free to adapt them accordingly.

## Sending events to Micro

Follow the documentation for one of [our trackers](/docs/sources/) to implement some tracking code on your website or application.

You can then point it to `localhost:9090` where Micro is listening. For example, using the [Browser tracker](/docs/sources/web-trackers/):

```js
import { newTracker, trackPageView, enableActivityTracking } from '@snowplow/browser-tracker';

newTracker('snowplow', 'localhost:9090', {
  appId: 'my-app-id',
  plugins: [],
});

trackPageView();
enableActivityTracking({
  heartbeatDelay: 10,
  minimumVisitLength: 10,
});
```

## Checking the results

Once you have the tracking code and the events are flowing in, you should see something like this in the Micro logs:

```text
[INFO] EventLog - GOOD id:4bfd5b32-d02a-4f83-a731-4339898437e1 app_id:test type:page_view (iglu:com.snowplowanalytics.snowplow/page_view/jsonschema/1-0-0)
[INFO] EventLog - GOOD id:e7a5c64d-d0f7-48d9-9a50-be044db7d2f6 app_id:test type:page_ping (iglu:com.snowplowanalytics.snowplow/page_ping/jsonschema/1-0-0)
[INFO] EventLog - GOOD id:1608ca85-f5f9-4948-898a-728aa8f1131b app_id:test type:page_ping (iglu:com.snowplowanalytics.snowplow/page_ping/jsonschema/1-0-0)
```

This means your tracking is set up correctly and your events are valid (`GOOD`).

Would you rather see the events visually? Open <http://localhost:9090/micro/ui> in your browser. You might want to check [a few tips on how to use the dashboard](/docs/testing/snowplow-micro/ui/).

![Micro dashboard overview](/assets/images/overview-1728998b503b68a664fe3b2d8c0e36b8.png)

Alternatively, you can inspect the events via [the API](/docs/api-reference/snowplow-micro/api/). For example, try:

```bash
curl localhost:9090/micro/good
```

## Exporting events

Snowplow pipelines output data in the [_enriched TSV format_](/docs/pipeline/enriched-tsv-format/). Typically, this is picked up by one of our [loaders](/docs/destinations/warehouses-lakes/) or by tools such as [Snowbridge](/docs/api-reference/snowbridge/).

With Micro, you can see what your data would look like in this format — useful if you want to test any logic that is parsing this data.

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv
```

Since version 2.4.0, you can alternatively output the data in JSON format (the same as provided in [Snowplow Analytics SDKs](/docs/api-reference/analytics-sdk/)):

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-json
```

Also since version 2.4.0 you can send the data in either format to an HTTP endpoint, rather then standard output:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv \
--destination http://some-url.com
```

**Output vs logs**

The TSV or JSON data will be printed to the [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_\(stdout\)). As you saw above, Micro also prints logs, which go into the [standard error stream](https://en.wikipedia.org/wiki/Standard_streams#Standard_error_\(stderr\)).

Depending on how you are running Micro, you might find the logs distracting. If so, you can turn off event logs with an extra option:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv \
-Dorg.slf4j.simpleLogger.log.EventLog=off
```

Or just discard the standard error stream entirely using the syntax appropriate for your shell:

```bash
# for bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv \
2>/dev/null
```

Finally, you can save the output to a file:

```bash
docker run -p 9090:9090 snowplow/snowplow-micro:4.1.1 --output-tsv > output.tsv
```

---

# Connect remote sites and apps to Snowplow Micro
> Test remote websites and apps with Micro using ngrok or localtunnel for public access, or use local domain resolution to receive first-party cookies. Configure HTTPS and match collector settings.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/local/remote-usage/

If you are not running your website or app locally but would still like to use Micro for local testing and debugging, there are two options:

|                                         | Option 1 [Exposing Micro via a public domain name](#exposing-micro-via-a-public-domain-name) | Option 2 [Locally resolving an existing domain name to Micro](#locally-resolving-an-existing-domain-name-to-micro) |
| --------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Requires control over the tracking code | Yes                                                                                          | No                                                                                                                 |
| Accessible from multiple devices        | Yes                                                                                          | No                                                                                                                 |
| Micro receives first-party cookies      | No                                                                                           | Yes                                                                                                                |
| Micro receives external IP addresses    | Yes                                                                                          | No                                                                                                                 |

## Exposing Micro via a public domain name

> **Tip:** This method is for running Micro locally. You can also [run Micro through Console](/docs/testing/snowplow-micro/console/), which automatically gives you a public endpoint.

This method allows you to get a publicly accessible URL for your Micro, which you can point your tracking code to. You can also send this URL to colleagues for them to inspect the results in the [dashboard](/docs/testing/snowplow-micro/ui/) (since version 2.0.0) or via the [API](/docs/api-reference/snowplow-micro/api/).

```mermaid
flowchart LR
  website(Website or app)
  tracking(Tracking code)
  url(URL like <code>https://22d8-....ngrok.app</code>)
  tunnel{{"<i>Tunneling magic</i> 🪄"}}
  micro(Micro on your machine)

  website --> tracking --> url --> tunnel --> micro
```

The easiest way to achieve this is with a tool like [ngrok](https://ngrok.com/) or [localtunnel](https://theboroer.github.io/localtunnel-www/).

> **Note:** Either tool will take care of HTTPS and the required certificates. However, the resulting URL will not be under your domain, so you will not be able to set or receive first-party cookies this way.
>
> You will, however, receive external IP addresses (not something like `192.168.0.42`) when you or others browse the website or app pointing to Micro. This might be useful for testing certain enrichments.

After running Micro as usual, you just need to expose the port (by default, `9090`):

**ngrok:**

[Sign up](https://dashboard.ngrok.com/signup), [download](https://ngrok.com/download) ngrok and follow the instructions to authenticate your client. Then run this command:

```bash
ngrok http 9090
```

You will see the publicly available URL in the output.

**localtunnel:**

Install:

```bash
npm install -g localtunnel
```

Then run this command:

```bash
lt --port 9090
```

You will see the publicly available URL in the output. Before use, visit this URL in your web browser and click “Continue”.

***

## Locally resolving an existing domain name to Micro

This method only works on your machine, but it allows you to connect any website or app to your Micro, even if you don’t control the tracking code.

```mermaid
flowchart LR
  website(Website or app)
  tracking(Tracking code)
  url(Existing collector URL like <code>c.example.com</code>)
  collector(Actual collector)
  micro(Micro on your machine)

  url -.-x collector
  website --> tracking --> url -- "<i>Custom local domain<br/>resolution rule</i>" --> micro
```

Let’s say you have a website `example.com` with Snowplow tracking that points to a Collector hosted at `c.example.com`.

> **Note:** With this approach, because Micro is “pretending” to be behind the actual Collector domain (`c.example.com`), it will receive first-party cookies set for that domain.
>
> However, you will only get local IP addresses (like `192.168.0.42`) in your data, because the traffic never leaves your machine. This might be relevant for testing some enrichments.
>
> If you are using a web browser to test your site or app, you can spoof a specific IP address using a browser plugin that sets an `X-Forwarded-For` header. For example, here are plugins for [Chrome](https://chrome.google.com/webstore/detail/x-forwarded-for-header/hkghghbnihliadkabmlcmcgmffllglin/) and [Firefox](https://addons.mozilla.org/en-US/firefox/addon/x-forwarded-for-injector/). Install the plugin and set the IP address to your liking.

### Generate a local SSL/TLS certificate

Chances are that the website in question uses HTTPS. If so, you will need to configure Micro to [enable HTTPS](/docs/testing/snowplow-micro/local/advanced-usage/#enabling-https). Otherwise, feel free to skip this step.

First, install [`mkcert`](https://github.com/FiloSottile/mkcert). This tool allows you to easily generate SSL/TLS certificates that your machine trusts.

Now, run the following commands in a terminal (make sure to substitute your Collector domain for `c.example.com`):

```bash
# one-time setup
mkcert -install

# generate a certificate
mkcert -pkcs12 c.example.com
```

You should now have a local file called `c.example.com.p12` with the default password `changeit`.

### Match the Collector configuration

To make sure your Micro behaves the same way as the Collector it’s “pretending” to be, copy the relevant parts of your Collector configuration and [pass them to Micro](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-collector-configuration).

The two most important settings are the cookie name (you can set it with `-Dcollector.cookie.name` as shown on the page linked above) and any custom paths (via a configuration file).

### Run Micro

You will need to run Micro as usual, with a few changes:

- [Enable HTTPS](/docs/testing/snowplow-micro/local/advanced-usage/#enabling-https), if needed
- Use port `80` instead of `9090`
- If HTTPS is enabled, also use port `443` instead of `9543`

If you don’t need HTTPS:

```bash
docker run -p 80:9090 snowplow/snowplow-micro:4.1.1
```

If you need HTTPS (substitute your Collector domain for `c.example.com`):

```bash
docker run -p 80:9090 -p 443:9543 \
--mount type=bind,source=$(pwd)/c.example.com.p12,destination=/config/ssl-certificate.p12 \
-e MICRO_SSL_CERT_PASSWORD=changeit \
snowplow/snowplow-micro:4.1.1
```

### Modify the hosts file

To conclude the setup, you will have to modify your [hosts file](https://en.wikipedia.org/wiki/Hosts_\(file\)) (`/etc/hosts` on Linux/macOS, `C:\Windows\System32\drivers\etc\hosts` on Windows) to point `c.example.com` — the Collector domain — to your local machine. Add these lines:

```text
127.0.0.1 c.example.com
```

_With this change, whenever your system tries to send data to `c.example.com`, it will send it to your Micro instead._ Done!

> **Tip:** Don’t forget to revert the hosts file change once no longer necessary.

---

# Add custom schemas to Snowplow Micro
> Connect Snowplow Micro to Iglu registries or add schemas directly. Point Micro to Console registries, Iglu Server, or mount local schema files for custom event validation.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/local/schemas/

One of the benefits of using Snowplow is that you can design your own schemas for your events.

> **Tip:** See [our explanation](/docs/fundamentals/schemas/) on what schemas are for and what they look like.

To track an event with a custom schema, you would need code like this (using the [Browser tracker](/docs/sources/web-trackers/custom-tracking-using-schemas/) as an example):

```js
import { trackSelfDescribingEvent } from '@snowplow/browser-tracker';

trackSelfDescribingEvent({
  event: {
    schema: 'iglu:com.example/my-schema/jsonschema/1-0-0',
    data: {
        ...
    }
  }
});
```

For Micro to understand this event, it will need to know about `com.example/my-schema/jsonschema/1-0-0` or any other relevant schemas. There are two ways you can achieve this:

- **Point Micro to an Iglu registry that contains your schemas.** This is a good option if you use Snowplow [Console](/docs/event-studio/data-structures/) UI or [API](/docs/event-studio/programmatic-management/data-structures-api/) to create schemas, or if you have deployed your own Iglu registry.
- **Add schemas to Micro directly.** This can be handy for quickly testing a schema.

Whichever approach you choose, you can use the [the API](/docs/api-reference/snowplow-micro/api/#microiglu) to check if Micro is able to reach your schemas (replace `com.example` and `my-schema` as appropriate).

```bash
curl localhost:9090/micro/iglu/com.example/my-schema/jsonschema/1-0-0
```

> **Tip:** If you are [running Micro through Console](/docs/testing/snowplow-micro/console/), it automatically integrates with your development Iglu Server, so you don’t need to follow these steps.

## Pointing Micro to an Iglu registry

Place your Iglu registry URL and API key (if any) into two [environment variables](https://en.wikipedia.org/wiki/Environment_variable): `MICRO_IGLU_REGISTRY_URL` and `MICRO_IGLU_API_KEY`.

Make sure to fully spell out the URL, including the protocol (`http://` or `https://`). For most Iglu registries, including those provided by Snowplow CDI, the URL will end with `/api` — make sure to include that part too, for example: `https://com-example.iglu.snplow.net/api`. [Static registries](/docs/api-reference/iglu/iglu-repositories/static-repo/), such as `https://iglucentral.com`, are an exception — you don’t need to append `/api` to the URL.

> **Tip:** You can find your Iglu registry URLs and generate API keys [via the console](https://console.snowplowanalytics.com/iglu-keys).

The following Docker command will pick up the environment variables and pass them to Micro:

```bash
 export MICRO_IGLU_REGISTRY_URL=https://com-example.iglu.snplow.net/api
export MICRO_IGLU_API_KEY=abcdef123456


docker run -p 9090:9090 \
-e MICRO_IGLU_REGISTRY_URL \
-e MICRO_IGLU_API_KEY \
snowplow/snowplow-micro:4.1.1
```

This will ensure Micro uses your Iglu registry, in addition to [Iglu Central](/docs/api-reference/iglu/iglu-repositories/iglu-central/).

For more flexibility, see [Advanced usage](/docs/testing/snowplow-micro/local/advanced-usage/#adding-custom-iglu-resolver-configuration).

## Adding schemas directly to Micro

> **Note:** Currently, this method does not work for [marking schemas as superseded](/docs/fundamentals/schemas/versioning/#mark-a-schema-as-superseded).

Structure your schema file or files like so:

```text
schemas
└── com.example
    └── my-schema
        └── jsonschema
            ├── 1-0-0
            └── 1-0-1
```

> **Note:** This folder structure is significant. Also note that the schema files must be named `1-0-0`, `1-0-1`, and so on, **not** `1-0-0.json` or `1-0-1.json`.

Next, you will need to place the schemas in `/config/iglu-client-embedded/` inside the container.

```bash
docker run -p 9090:9090 \
--mount type=bind,source=$(pwd)/schemas,destination=/config/iglu-client-embedded/schemas \
snowplow/snowplow-micro:4.1.1
```

> **Tip:** You can read more about bind mounts in the [Docker documentation](https://docs.docker.com/storage/bind-mounts/).

Alternatively, if you are running Micro as a Java application, place your schemas — using the same structure as above — in `some-directory/iglu-client-embedded` on your machine (`iglu-client-embedded` must be called exactly that) and use the following command:

```bash
java -cp micro-4.1.1.jar:some-directory com.snowplowanalytics.snowplow.micro.Main
```

---

# View and filter events in the Snowplow Micro dashboard
> Access the Micro dashboard to view, filter, and investigate good and bad events. Share the dashboard with colleagues through public domain names.
> Source: https://docs.snowplow.io/docs/testing/snowplow-micro/ui/

Micro provides a user interface that helps you explore events sent to it, including [failed events](/docs/fundamentals/failed-events/).

If you are [running Micro through Console](/docs/testing/snowplow-micro/console/), navigate to the **Pipelines** side menu and select your Micro. Then click **Open dashboard**. You will need the _View environments_ permission to access the dashboard.

If you are [running Micro locally](/docs/testing/snowplow-micro/local/), head to <http://localhost:9090/micro/ui> in your web browser.

Here’s an overview of the dashboard functionality:

---

# Set up an abandoned browse campaign in Braze
> Learn how to create and configure an abandoned browse email campaign in Braze using product view data synced from Census. Set up triggers, templates, and test your campaign to re-engage users who viewed products but didn't purchase.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/braze-campaign/

This guide will walk you through setting up and testing an abandoned browse campaign in Braze using the product view data synced from Census.

## Setting up the campaign

1. Log into your Braze dashboard
2. Navigate to **Campaigns** and click **Create campaign**
3. Select **Email** as the channel
4. Name your campaign (e.g. "Abandoned browse - product reminder")

![Create Campaign](/assets/images/retl-braze-create-campaign-b03594aa1c9aa2f02db4bdecb1243c07.png)

## Configure campaign trigger

1. In the **Delivery** section, select **Action-Based**
2. Set up the trigger criteria

## Create email template

![Braze Campaign Builder](/assets/images/retl-braze-preview-8b56ef2f1fd9bb457740995c290770a8.png)

1. Click **Edit Campaign** in the campaign builder and click **Edit Email Body**

2. Design your email using Braze's visual editor:

   ```html
   Subject: Don't miss out on {{custom_attribute.${product}}}!

   Hey {{${first_name}}} we saw you were interested in {{custom_attribute.${PRODUCT}}}

   <a href="{{custom_attribute.${product_url}}}?abandonedEmail=true">
     You spent {{custom_attribute.${TIME_ENGAGED_IN_S}}} seconds checking it out.... Why not check it out once more before it sells out?!!
   </a>
   ```

3. Add personalization:

   - product name using `{{custom_attribute.${product}}}`
   - product URL using `{{custom_attribute.${product_url}}}`
   - add dynamic product images if available

> **Tip:*** The `abandonedEmail=true` parameter in the URL helps track when users click through from abandoned browse emails
>
> * You can use this parameter to:
>
>   - track email campaign success
>   - remove users from the campaign audience once they've engaged. See image below for how to configure this in your Reverse ETL audience.

![Filter Winback](/assets/images/retl-winback-filtered-2091ae0f18dc04133a5875f4e77163c4.png)

## Campaign settings

1. Configure timing:

   - set delay after trigger e.g. 1 hour
   - set quiet hours e.g. 9 PM - 9 AM local time
   - set frequency capping e.g. max 1 email per user per 24 hours

2. Set conversion tracking:

   - primary conversion: "Purchase"
   - secondary conversion: "Add to Cart"

## Testing the campaign

1. Return to **Edit Campaign** and click **Preview** at the bottom of the page

   - select **Search User** under preview message as user
   - if the user can't be found, ensure Census has synced product view data for this user
   - verify custom attributes are populated correctly in the email
   - click **Send Test** to send the test email

2. Verify email content:

   - check all personalization renders correctly
   - verify product links work
   - test on multiple email clients

3. Verify winback success:

   - click on the link you receive in the test email with the `abandonedEmail=true` parameter
   - in Snowflake or Census, check the `winback_successful` column for the user has been set to true

***

![Braze Test Email](/assets/images/retl-email-0888d05d2f92d2d747eca5c1bf5a1083.png)

***

## Best practices

- keep email content focused on one product
- include social proof (reviews, ratings)
- add sense of urgency (limited time offer)
- ensure mobile responsiveness
- include clear unsubscribe option

## Congratulations

Thanks for your effort! You have now set up an abandoned browse campaign in Braze, sent an email, and verified winback success. Let's review what we have achieved in the [conclusion](/tutorials/abandoned-browse-ccdp/conclusion/).

---

# Conclusions and next steps from the abandoned browse composable CDP accelerator
> Review what you've accomplished in this abandoned browse tutorial and explore recommended next steps including advanced segmentation, A/B testing, and personalization strategies to improve conversion rates.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/conclusion/

Congratulations on completing the tutorial on setting up an abandoned browse tracking and re-engagement system using Snowplow, Snowflake, Braze, and Census. Here's a summary of what you've learned and achieved.

## Summary of achievements

1. **Tracking setup**: You have successfully implemented product view and engagement tracking using Snowplow's JavaScript tracker. This includes setting up page view, time spent, and add-to-cart tracking.

2. **Data modeling**: You have learned how to model your data to identify users who have shown interest in products but have not completed a purchase. This involved writing SQL queries to aggregate and analyze user behavior data, including tracking successful winback campaigns through URL parameters.

3. **Reverse ETL**: You have set up a reverse ETL workflow using Census to sync your modeled data to a marketing automation platform like Braze. This enables you to create targeted re-engagement campaigns.

4. **Campaign creation**: You have created and configured an abandoned browse email campaign in Braze, utilizing personalized content and tracking parameters to re-engage users, drive conversions, and measure campaign success.

## Recommended next steps

- **Expand tracking**: extend your tracking setup to include additional user interactions, checkout steps, and campaign success metrics to create a more comprehensive view of user behavior and marketing effectiveness.

- **Advanced segmentation**: use the data collected to create more advanced audience segments, such as users with high engagement but no purchase, or users who frequently view specific product categories.

- **A/B testing**: implement A/B testing for different campaign elements, such as subject lines, send times, and content layouts, to identify what resonates best with your audience.

- **Personalization**: enhance your campaigns with more personalized content, such as product recommendations based on browsing history or dynamic pricing offers.

- **Integration with other tools**: consider integrating additional tools and platforms, such as CRM systems, to enrich your data with the customer's name and other data to improve campaign effectiveness.

By following these steps, you can continue to refine and expand your abandoned browse tracking and re-engagement system, ultimately driving higher conversion rates and improving customer engagement.

Thank you for following this tutorial. We hope it has been informative and helpful.

---

# Model the data to identify unfinished purchases
> Write SQL queries in Snowflake to identify users who viewed products but didn't complete purchases. Aggregate product view data and calculate engagement metrics for abandoned browse campaigns.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/data-modeling/

After implementing the tracking setup, send some events and test that they are arriving in your data warehouse. If you don't have a data warehouse, you can sign up for a [free trial of Snowflake](https://www.snowflake.com/).

Once you have sent data to Snowflake, you can run the following query to verify the events are coming through. The [Snowflake Streaming loader](http://localhost:3000/docs/api-reference/loaders-storage-targets/snowflake-streaming-loader/) has a latency of several seconds so you should see results immediately.

```sql
SELECT
    domain_userid,
    user_id,
    page_urlpath AS product_id,
    event_name,
    page_title,
    unstruct_event_com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1,
    contexts_com_snowplowanalytics_snowplow_ecommerce_product_1,
    contexts_com_snowplowanalytics_snowplow_web_page_1
FROM DATABASE.ATOMIC.EVENTS
WHERE
    DATE(load_tstamp) = CURRENT_DATE()
ORDER BY load_tstamp DESC;
```

The output of this query should be similar to below: ![Atomic Events](/assets/images/retl-atomic-events-215e6baee1010e1b127281382d11c76d.png)

## Identifying most viewed but not added-to-cart products

Once events are confirmed, use the following query to check that we can aggregate the data correctly. This query will be used in the next step to build an abandoned browse audience in Census.

The SQL query identifies users who have viewed product pages today but have not completed a purchase. It aggregates data from Snowplow events to calculate the total time a user has engaged with product pages, checks if they have added items to their cart, and determines if they have previously received a win-back email. The second part of the query then selects the most engaged product for each user and ensures that only users who are logged in are included.

Snowplow loads to most data warehouses / lakes in real-time so this query will include users that have been active within the last couple seconds!

```sql
WITH productsViewedToday AS (
    SELECT
        domain_userid,
        page_urlpath AS product_id,
        MAX(user_id) AS email,
        MAX(product.value:name::STRING) AS product,
        5 * SUM(CASE WHEN event_name = 'page_ping' THEN 1 ELSE 0 END) AS time_engaged_in_s,
        MAX(
            CASE
                WHEN ecom_action.value:type = 'add_to_cart'
                THEN TRUE
                ELSE FALSE
            END
        ) AS add_to_cart,
        MAX(
            CASE
                WHEN page_urlquery = 'abandonedEmail=true'
                THEN TRUE
                ELSE FALSE
            END
        ) AS winback_successful,
        MAX(page_url) AS product_url
    FROM
        SNOWPLOW_SALES_AWS_PROD1_DB.ATOMIC.EVENTS,
        LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_ecommerce_product_1) product,
        LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_web_page_1) page,
        LATERAL FLATTEN(input => unstruct_event_com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1) ecom_action
    WHERE
        DATE(load_tstamp) = CURRENT_DATE()
        AND page_urlpath LIKE '/product%'
    GROUP BY
        1, 2
    ORDER BY
        time_engaged_in_s DESC
)

SELECT
    a.*
FROM
    productsViewedToday a
LEFT JOIN
    productsViewedToday b
    ON a.email = b.email
    AND a.time_engaged_in_s < b.time_engaged_in_s
WHERE
    b.time_engaged_in_s IS NULL
    AND a.email IS NOT NULL;
```

The output of this query should be similar to below: ![Aggregated Query](/assets/images/retl-aggregated-query-1878273d3c24faa254ac5c89f49f66bd.png)

## Next step

Proceed to the [reverse ETL setup](/tutorials/abandoned-browse-ccdp/reverse-etl/) to sync this data to your marketing platform.

---

# Learn how to implement an abandoned browse system with a composable CDP
> Build an abandoned browse tracking and re-engagement system using Snowplow, Snowflake, Census, and Braze. Learn how a composable CDP approach helps recover lost ecommerce conversions through personalized campaigns.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/introduction/

Abandoned browse is a common ecommerce problem where users show interest in products but don't complete a purchase. It is also referred to as "shopping cart abandonment," "abandoned basket," or "abandoned cart." Despite how common it is, it is still a challenge to implement a successful re-engagement campaign when using traditional marketing tools because they lack all the context needed to create a compelling personalized message. At Snowplow, we have found that a composable CDP approach is the best way to solve this problem. This tutorial has been written to show that it is straightforward to get started.

***

![Abandoned Browse](/assets/images/retl-email-0888d05d2f92d2d747eca5c1bf5a1083.png)

***

This tutorial demonstrates how to implement an abandoned browse tracking and re-engagement system using [Snowplow](https://snowplow.io/), [Snowflake](https://www.snowflake.com/), and [Census](https://www.getcensus.com/). This solution helps ecommerce businesses identify and re-engage users who have shown interest in a product (e.g., viewed something for 10+ seconds) but haven't proceeded further.

***

![Composable CDP](/assets/images/retl-snowplow-composable-cdp-20c8ca5ec4dcbc667e2ad13607fbb80d.png)

***

## Prerequisites

- An ecommerce website with a product catalog to track events from

- **Snowplow instance**:

  - [Localstack](https://github.com/snowplow-incubator/snowplow-local) (recommended)
  - [Community edition](/docs/get-started/self-hosted/)
  - Snowplow CDI if you're already a customer

- **Access to a data warehouse**: e.g., [Snowflake](https://www.snowflake.com)

- **Reverse ETL**: [Census Reverse ETL](https://www.getcensus.com) or Snowplow Reverse ETL

- **Marketing automation platform**: e.g., [Braze](https://www.braze.com)

## What you'll learn

- How to implement product view tracking using Snowplow's JavaScript tracker
- Setting up time-on-page tracking to measure user engagement
- Creating SQL queries to identify abandoned browse behavior
- Implementing Reverse ETL workflows to sync data to marketing platforms
- Building automated re-engagement campaigns

## Business outcomes

- Identify users showing genuine interest in products
- Measure product engagement through view time and interaction
- Create targeted re-engagement campaigns
- Increase conversion rates through personalized messaging
- Track campaign effectiveness and ROI

## Similar use cases

This solution can be adapted for:

1. **Abandoned cart recovery**: extend the tracking to include cart additions and checkout steps
2. **Product recommendations**: use viewing patterns to suggest related items
3. **Category affinity analysis**: understand user preferences across product categories
4. **Price drop alerts**: notify users when viewed items go on sale
5. **Inventory alerts**: alert users when viewed out-of-stock items become available

By following this tutorial, you'll establish a complete abandoned browse tracking and re-engagement system that can be expanded to support various ecommerce marketing initiatives.

## Next step

Proceed to the [tracking setup](/tutorials/abandoned-browse-ccdp/tracking-setup/) step to implement the tracking setup.

---

# Integrate with with Census using reverse ETL
> Set up Census reverse ETL to sync abandoned browse audiences from Snowflake to Braze. Configure data sources, create audience segments, and automate marketing campaign triggers.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/reverse-etl/

Next we will set up a Census sync to build an audience using our query from Snowflake, filter the audience to focus on users who have shown interest in products but haven't purchased, and sync the data to Braze.

## Connect Census to Snowflake

1. Log into your Census account ([sign up if needed](https://www.getcensus.com/))
2. Go to **Sources**
3. Click **New Source**
4. Select **Snowflake**
5. Enter your Snowflake connection details. Refer to the [documentation](https://docs.getcensus.com/sources/available-sources/snowflake) if you have any questions
6. Test the connection and save

## Create your abandoned browse audience

Use the query from the [data modeling](/tutorials/abandoned-browse-ccdp/data-modeling/#identifying-most-viewed-but-not-added-to-cart-products) step to identify users who have shown interest in products but haven't purchased. Here it is again:

```sql
WITH productsViewedToday AS (
    SELECT
        domain_userid,
        page_urlpath AS product_id,
        MAX(user_id) AS email,
        MAX(product.value:name::STRING) AS product,
        5 * SUM(CASE WHEN event_name = 'page_ping' THEN 1 ELSE 0 END) AS time_engaged_in_s,
        MAX(
            CASE
                WHEN ecom_action.value:type = 'add_to_cart'
                THEN TRUE
                ELSE FALSE
            END
        ) AS add_to_cart,
        MAX(
            CASE
                WHEN page_urlquery = 'abandonedEmail=true'
                THEN TRUE
                ELSE FALSE
            END
        ) AS winback_successful,
        MAX(page_url) AS product_url
    FROM
        SNOWPLOW_SALES_AWS_PROD1_DB.ATOMIC.EVENTS,
        LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_ecommerce_product_1) product,
        LATERAL FLATTEN(input => contexts_com_snowplowanalytics_snowplow_web_page_1) page,
        LATERAL FLATTEN(input => unstruct_event_com_snowplowanalytics_snowplow_ecommerce_snowplow_ecommerce_action_1) ecom_action
    WHERE
        DATE(load_tstamp) = CURRENT_DATE()
        AND page_urlpath LIKE '/product%'
    GROUP BY
        1, 2
    ORDER BY
        time_engaged_in_s DESC
)

SELECT
    a.*
FROM
    productsViewedToday a
LEFT JOIN
    productsViewedToday b
    ON a.email = b.email
    AND a.time_engaged_in_s < b.time_engaged_in_s
WHERE
    b.time_engaged_in_s IS NULL
    AND a.email IS NOT NULL;
```

## Set up the Census sync

1. In Census, go to **Syncs** and click **Create New Sync**

2. For the source:

   - select your Snowflake connection
   - configure the authentication method

3. Configure the dataset:

   - select **Datasets** from the menu
   - select the Snowflake source
   - enter the SQL query above
   - click **Preview** and ensure it returns data

![Census Dataset](/assets/images/retl-datasets-4d90c1ead6c352f1f50790443978093a.png)

4. Configure the audience segment:

   - select **Segments** from the menu
   - click **New Segment**
   - under **Segment Of**, select the dataset from the previous step
   - add filters to the segment based on the image below
   - click preview to confirm that the correct data is being returned

![Census Sync](/assets/images/retl-census-7819e962fba8de71543eb2613cbdf98b.png)

5. Configure the sync settings to send data from our audience to Braze:

   - select **Syncs** from the menu
   - click **New Sync**
   - under **Source**, select the audience from the previous step
   - under **Destination**, select Braze
   - configure the mapping as shown in the image below
   - click **Run Now**

![Census Mapping](/assets/images/retl-census-mapping-8c59b71665b4069e31c3d8ab3f5100c5.png)

## Monitoring and optimization

- Monitor sync status in Census dashboard
- Track campaign performance metrics
- A/B test different message content and timing
- Adjust view time thresholds based on results

## Best practices

- Review view time thresholds (10+ seconds) to ensure genuine interest
- Include product images and details in your messages
- Add urgency elements (e.g., limited time offers)
- Test different message sequences and timings
- Monitor unsubscribe rates to avoid message fatigue

By following these steps, you'll have an automated system that identifies users showing genuine interest in products and triggers relevant abandoned browse campaigns to encourage purchases.

## Next step

Proceed to the [creating campaigns in Braze](/tutorials/abandoned-browse-ccdp/braze-campaign/) step to set up your marketing automation campaign.

---

# Capture abandoned browse behavior with the JavaScript tracker
> Implement Snowplow JavaScript tracker to capture product views, user engagement time, and add-to-cart events on your ecommerce site. Track abandoned browse behavior with the ecommerce accelerator plugin.
> Source: https://docs.snowplow.io/tutorials/abandoned-browse-ccdp/tracking-setup/

To begin, we will set up Snowplow tracking on your ecommerce website. In this section we will capture what products a customer views, how long they view them for and if they add them to their cart. We assume that you already have a Snowplow pipeline. If you do not yet have a pipeline running, please return to the [Introduction](/tutorials/abandoned-browse-ccdp/introduction/) information on the different deployment options.

![website](/assets/images/retl-shopfront-6dc061d7fe6da3fc3370ad6992a6d3a1.png)

First, [initialize the Snowplow JavaScript tracker](/docs/sources/web-trackers/quick-start-guide/). Below is an example of how to set up the tracker:

```javascript
; (function (p, l, o, w, i, n, g) {
  if (!p[i]) {
    p.GlobalSnowplowNamespace = p.GlobalSnowplowNamespace || [];
    p.GlobalSnowplowNamespace.push(i); p[i] = function () {
      (p[i].q = p[i].q || []).push(arguments)
    }; p[i].q = p[i].q || []; n = l.createElement(o); g = l.getElementsByTagName(o)[0]; n.async = 1;
    n.src = w; g.parentNode.insertBefore(n, g)
  }
}(window, document, "script", "https://cdn.jsdelivr.net/npm/@snowplow/javascript-tracker@latest/dist/sp.lite.js", "snowplow"));

// Replace with your collector URL
var collector = "yourcollector.site.com";

window.snowplow('newTracker', 'trackerName', collector, {
  encodeBase64: false,
  appId: 'ecommerceDemo',
  platform: 'web',
  contexts: {
    webPage: true,
    performanceTiming: true
  }
});

// Add ecommerce accelerator plugin
window.snowplow(
  'addPlugin',
  'https://cdn.jsdelivr.net/npm/@snowplow/browser-plugin-snowplow-ecommerce@latest/dist/index.umd.min.js',
  ['snowplowEcommerceAccelerator', 'SnowplowEcommercePlugin']
);
```

## Track page views and user engagement time

We want to understand how long a product is viewed for in order to determine the which product each customer is paying the most attention to.

Use the `enableActivityTracking` function to calculate the time the user is actively engaged on the page:

- **minimumVisitLength**: the minimum time (in seconds) a user must stay on the page before tracking starts
- **heartbeatDelay**: the interval (in seconds) at which activity pings are sent

If you change these parameters, make sure to update the values in the [data modeling](/tutorials/abandoned-browse-ccdp/data-modeling/#identifying-most-viewed-but-not-added-to-cart-products) step.

This event must be called before the `trackPageView` function.

```javascript
snowplow('enableActivityTracking', {
  minimumVisitLength: 5,
  heartbeatDelay: 5
});

snowplow('trackPageView');
```

## Test your tracking

To verify your tracking implementation, use the [Snowplow Chrome extension](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm). This extension allows you to inspect Snowplow events in real-time as they are sent from your website. Navigate to your product pages and add items to cart while monitoring the extension to ensure events are firing correctly with all expected parameters. The extension will show you the full event payload including all entities and properties, making it easy to debug your implementation.

![Chrome Extension](/assets/images/retl-chrome-extension-5fae75e18e8df7e43a5dfb1359e19fa8.png)

## Track product views

Implement the product view tracking when a product is viewed. This will create a column in the warehouse dedicated to storing information on viewed products. Please ensure you use the correct data type for each variable. Refer to the [ecommerce documentation](/docs/sources/web-trackers/tracking-events/ecommerce/) for further information.

```javascript
  snowplow('trackProductView', {
    id: "12345",
    name: "Baseball T",
    brand: "Snowplow",
    category: "apparel",
    price: 200,
    currency: "USD",
});
```

## Track "add to cart" events

Track "add to cart" events so users that perform this action can be filtered out or be placed in a different cohort. Below is an example implementation, showing a single product being added to the cart.

```javascript
window.snowplow("trackAddToCart", {
  products: [
    {
      id: "P125",
      name: "Baseball T",
      brand: "Snowplow",
      category: "Mens/Apparel",
      price: 200,
      currency: "USD",
    },
  ],
  total_value: 200,
  currency: "USD",
});
```

Once add to cart events are tracked, the event should look like this in your Snowplow Chrome extension.

![Add to cart](/assets/images/retl-add-to-cart-320ff009a887e9e8af27245a1726b116.png)

## Explanation of parameters

- **`id`**: the unique identifier for the product
- **`name`**: the product's name
- **`price`**: the product's price as a floating-point number
- **`brand`**: the brand associated with the product
- **`currency`**: the currency code (e.g., USD, EUR)
- **`category`**: the product's category or taxonomy
- **`total_value`**: the updated total cart value after adding the product

---

# Examine your Android screen view events with Snowplow Micro
> Verify and analyze Android screen view events using Snowplow Micro. View session data, screen engagement metrics, and automatically generated screen end events with foreground and background time tracking.
> Source: https://docs.snowplow.io/tutorials/android-event-tracking/examining-events/

After implementing tracking, you'll want to verify and analyze the events being sent. You can use [Snowplow Micro](/docs/testing/snowplow-micro/) for testing and development — either [through Console](/docs/testing/snowplow-micro/console/) or [locally](/docs/testing/snowplow-micro/local/). Here's how to examine the events:

1. Set up a Snowplow Micro pipeline and configure your tracker to send events to its endpoint, by updating the string in the `Analytics` object.

2. Run the app and move between screens.

3. Access the [Micro dashboard](/docs/testing/snowplow-micro/ui/) to view incoming events.

4. Each event will contain several context entities by default, providing data about the:

   - Session
   - App
   - Device
   - Current screen view
   - Whether the app was visible when the event was tracked (foreground or background)

5. For screen view events, you'll see:

   - Screen name
   - Previous screen name
   - Screen view ID
   - Previous screen view ID

6. The tracker also automatically generates screen end events, which include a screen summary entity with engagement data:

   - Time spent on the screen
   - Foreground vs background time

For more detailed information about all available Android events and their structures, refer to the comprehensive documentation provided by Snowplow Analytics.

This concludes the basic tutorial for getting started with Snowplow Analytics' Android tracker. In future tutorials, we'll explore how to track custom events and utilize more advanced features of the tracker.

---

# Learn how to track events on Android
> Set up the Snowplow Android tracker in your Jetpack Compose app using Kotlin. Install dependencies, initialize the tracker, and begin capturing mobile analytics events.
> Source: https://docs.snowplow.io/tutorials/android-event-tracking/installation/

## Requirements

- Snowplow pipeline running
- Android Studio installed

## Installation and Setup

To begin tracking events using Snowplow Analytics' Android tracker, you'll need to set up your project and initialize the tracker. We’ll be following the modern JetPack Compose framework using Kotlin as our langauge.

1. Clone the official Android JetSurvey example app here, or import it from Android Studio as described [here](https://developer.android.com/develop/ui/compose/setup#sample).

   ```bash
   git clone https://github.com/android/compose-samples/tree/main
   cd jetsurvey
   ```

2. Add the Snowplow Android tracker dependency to your project-level `libs.versions.toml` file (Replace `x.x.x` with the latest version of the tracker.)

   ```text
   snowplow = "x.x.x"
   ```

3. Add the dependency to your `build.gradle.kts` file

   ```kotlin
   implementation(libs.snowplow.analytics)
   ```

4. Create a new package in your project for analytics-related code, and create a `Analytics.kt` file inside it.

5. Now we’ll create the `Analytics` object in that `Analytics.kt` file by adding the following code. Replace `"YOUR_NAMESPACE"` with a unique identifier for your tracker and `"YOUR_COLLECTOR_URL"` with your Snowplow collector endpoint.

   ```kotlin
   package com.example.compose.jetsurvey.analytics

   import android.content.Context

   object Analytics {
    fun start(context: Context) {
     Snowplow.createTracker(
      context,
      namespace: "YOUR_NAMESPACE",
      endpoint: "YOUR_COLLECTOR_URL"
     )
    }
   }
   ```

6. In the entrypoint for your app, you’ll now want to initialize the tracker. In this case the first composable function called is the `JetsurveyNavHost()`, inside the `Navigation.kt` file. Add this line to the start of the function, before the `NavHost` instantiation:

---

# Track screen views using the Android tracker
> Implement automatic screen view tracking in Jetpack Compose apps using navigation listeners. Track screen engagement, time spent, and foreground/background status with the Snowplow Android tracker.
> Source: https://docs.snowplow.io/tutorials/android-event-tracking/screen-views/

In a Jetpack Compose app, screen views aren't tracked automatically. Screen view tracking can be manually added to every screen, but it’s more efficient to add a navigation listener. Here's how to track screen views using the navigation component:

1. Within the `Analytics` object, create a listener function:

   ```kotlin
   fun addScreenViewNavListener(navController: NavController){
    navController.addOnDestinationChangedListener { _, destination, ->
     Snowplow.defaultTracker?.track(ScreenView(destination.route ?: "unknown"))
   ```

2. Call this function by passing in your navigation, in your main composable:

   ```kotlin

   @Composable
   fun JetSurveyNavHost(
     navController = rememberNavController()
   ) {
     Analytics.start(LocalContext.current)
     Analytics.addScreenViewNavListener(navController)

     ...
   }
   ```

This setup will track a screen view event every time the navigation destination changes. The screen name will be set to the route of the destination.

Additionally, the Snowplow Android tracker uses the screen view events to automatically track screen engagement, including how long a user spends on each screen and whether the app is in the foreground or background.

---

# Optional Attribution dbt package configuration variables
> Review optional configuration variables for the Attribution dbt package. Customize conversion filters, path transforms, conversion windows, and reporting options to fit your marketing attribution needs.
> Source: https://docs.snowplow.io/tutorials/attribution/before-you-start/

Marketing Attribution Analysis can be done in many ways depending on your business and needs. Our aim has always been to make it flexible for users to make changes from selecting a different data source to reshaping how the customer journey to conversion will be adjusted for the analysis. We suggest taking the time to read the documentation before running the model and also readjust the configurations later on to fit your needs.

In the subsequent setup steps, we will only include the absolute minimum that's needed to be able to run it for the first time, but there are many other variables that should also be considered to fine-tune for the best outcome as the default values might not be the best for your specific analysis. Below we list a few optional variables that would dictate the modeled outcome to give you some examples:

- `snowplow__conversion_hosts`: url\_hosts to process, if left empty it will include all
- `snowplow__conversion_clause`: a user defined sql script to filter on specific conversions if needed. Defaulted to 'cv\_value > 0'
- `snowplow__path_transforms`: a dictionary of path transforms and their arguments (see [Transform Paths](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/#transform-paths) section which includes other relevant variable changes including changing the `snowplow_path_lookback_days` and `snowplow_path_lookback_steps` variables.)
- `snowplow__conversion_window_days`: by default the package processes the last complete n number of days (calculated from the last processed path in the path source) to dynamically define the conversion window for each incremental run
- `snowplow__enable_attribution_overview`. By default, the package creates a model called snowplow\_attribution\_overview which can be used directly for BI reporting. If you are using the Attribution Modeling Data Model Pack, this is not required to be enabled as the Data Model Pack will take care of querying this data for you

> **Info:** For an in-depth guide, follow our [package documentation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/). For a full list of variables you can use and their definitions, check out the [Configurations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/) page.

---

# Enable conversions in the Unified Digital dbt package
> Configure the conversions module in Snowplow Unified dbt package for attribution modeling. Define conversion events, enable the conversions table, and set up referral field fallbacks for marketing attribution.
> Source: https://docs.snowplow.io/tutorials/attribution/enabling-unified-conversions/

Snowplow’s Attribution package requires the optional conversions module to be configured within the Unified package, because the `snowplow_unified_conversions` table is used as a source by default. This is an additional incremental table that will be updated for each run of the Unified package.

Enabling Conversions will also add 6 extra columns into the sessions table. See more details about that on [this page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/).

If you already have conversion events and the conversion module is enabled, you can skip this step.

> **Info:** Please note: All the actions within this step are to be completed within your Snowplow **Unified Package** dbt project.

### Configuring Conversions in Unified

1. Add `snowplow__conversion_events: []` variable to dbt\_project.yml file inside your project where you the run Snowplow Unified package.
2. Add a conversion event into the array as shown below to the dbt\_project.yml file in the vars section.

You can see further details on configuring the conversion event on [this page of the docs](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/)

> **Info:** Please Note: The total value of conversions must be greater than 0 for the Attribution Package as it is required to calculate the proportion of value brought in by a campaign or channel.

```yml
vars:
  snowplow_unified:
    snowplow__conversion_events: [
      {
        "name": "purchase", # Required, name of the conversion, string (must be valid SQL column name)
        "condition": " event_name == 'purchase' ", # Required, valid SQL condition that returns true or false
        "value": " 1 ", # Optional but recommended, can be field name or SQL
        "default_value": "0", # Optional, can be field name or SQL
        "list_events": "true" # Optional, boolean
      }
    ]
```

1. Enable the Conversions Module to create the `snowplow_unified_conversions` table by setting the following variable in your Unified dbt\_project.yml file.

```yml
vars:
  snowplow_unified:
    snowplow__enable_conversions: true
```

> **Warning:** If you have not previously run the Unified package with the conversions module enabled by default it will cause the Snowplow package to backfill from the `snowplow__start_date` (with `snowplow__backfill_limit_days` increments for each run) until the `snowplow_unified_conversions` table becomes up-to-date with the rest of the incremental tables. During backfilling any existing derived incremental tables (e.g. sessions table) are not going to get updated.
>
> Alternatively, you can adjust the run configurations the first time you run the package with the enabled conversions module to only backfill the `snowplow_unified_conversions` table from a certain more recent date only. [Read more details here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/backfilling/).

### (Optional) Enable referral fields if marketing fields are null

It can be useful to use the referral fields source and medium if the marketing fields are null, by default the marketing fields are set from UTM parameters. Having referral fields as a fallback can be enabled with a variable from within the Unified Package.

1. Set the following variable to true within your `dbt_project.yml` file

```yml
vars:
  snowplow_unified:
    snowplow__use_refr_if_mkt_null: true
```

### Run the Unified Package

1. Run the Snowplow Unified package to create the `snowplow_unified_conversions` table and include your conversion events in your `snowplow_unified_sessions` table. By default, your `snowplow_unified_conversions` table will be in the same schema/dataset as your other unified model derived tables.

```bash
dbt run --select snowplow_unified
```

---

# Learn how to set up the Attribution dbt package
> Set up Snowplow's Attribution dbt package to analyze marketing attribution with multi-touch models. Prerequisites include Unified package, campaign enrichment, and conversion tracking.
> Source: https://docs.snowplow.io/tutorials/attribution/intro/

This tutorial walks you through how to set up Snowplow’s Attribution dbt package. It assumes you have already followed our Snowplow Unified Package [tutorial](/tutorials/unified-digital/intro/), or previously configured it, as the Attribution package requires the model outputs (e.g. unified views and conversions tables).

## Prerequisites

- [dbt](https://github.com/dbt-labs/dbt) installed (and dbt profile configured for Data Warehouse connection)

- have the [Campaign attribution enrichment](/docs/pipeline/enrichments/available-enrichments/campaign-attribution-enrichment/) enabled to generate the marketing campaign fields such as mkt\_medium, mkt\_source, mkt\_term, mkt\_content, and mkt\_campaign which would be needed for channel and campaign classification to ultimately calculate the appropriate attribution

- have the [Referer parser enrichment](/docs/pipeline/enrichments/available-enrichments/referrer-parser-enrichment/) enabled to extract attribution data from referer URLs

- Snowplow Unified Digital [dbt Package](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/) configured and run (v0.4.0 or above) it will generate the following source tables for you:

  - `snowplow_unified_views` table as a path (touch points) source
  - `snowplow_unified_conversions` table as a conversions source including the revenue (generated by the optional conversions module)
  - `snowplow_unified_user_mapping` table as the user mapping source (needed for user stitching)

- (Optional) Marketing Spend Source table. [See more information here](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-attribution-data-model/#3-channel-spend-information-optional-but-recommended). It will contain your marketing spend data by channel and or campaign with a timestamp field which denotes the period. This is needed for the ROAS calculation within the `snowplow_attribution_overview` model.

---

# Optional: Integrate marketing spend for attribution ROAS
> Integrate marketing spend data with the Attribution package to calculate ROAS (return on ad spend). Configure spend sources by channel and campaign for comprehensive attribution analysis.
> Source: https://docs.snowplow.io/tutorials/attribution/optional-adding-marketing-spend-source/

If you have marketing spend source data in your data warehouse then the Snowplow Attribution model package provides a configuration option that allows you to integrate your spend data with the `attribution_overview` view.

1. Create a view (or regularly updated table) based on your marketing spend data that has the columns below. Your channel and spend data will be summed separately in the `attribution_overview`, so you would not need to pre-aggregate this, it is fine to have channel twice for the same period even, but make sure that the data does not include duplicates as it might lead to unexpected outcomes.

| **Column Name** | **Data Type** | **Required**                   |
| --------------- | ------------- | ------------------------------ |
| spend\_tstamp   | TIMESTAMP     | Required                       |
| channel         | STRING        | Optional (Channel or Campaign) |
| campaign        | STRING        | Optional (Channel or Campaign) |
| spend           | NUMERIC       | Required                       |

2. Update your `dbt_project.yml` file to configure the variable to the location of your spend table.

```yaml
vars:
	snowplow_attribution:
		snowplow__spend_source: 'your_schema.marketing_spend_source'
```

3. Run the dbt package

```yaml
dbt run
```

4. Congratulations, your `attribution_overview` view within your `_derived` schema should now reference your marketing spend source to provide a ROAS calculation!

> **Info:** Please Note: By default in the `attribution_overview` view created by the package spend is allocated associated with a channel/campaign for the 90 days prior to the conversion. You can configure this by overriding the macro within dbt.

---

# Optional: Configure custom channel group classifications
> Customize channel grouping for attribution analysis by overriding dbt macros in the Unified or Attribution package. Define custom channel classifications to match your marketing taxonomy.
> Source: https://docs.snowplow.io/tutorials/attribution/optional-configuring-channel-group-classification/

By default, the Attribution Package uses the channel grouping set within the Unified Package. It is recommended to set the channel grouping within the Unified Package, so you have matching channel groups across both packages.

Optionally, you can choose to have differing channel groupings within the Attribution Package by customizing the macro only within the Attribution Package.

> **Info:** You only need to follow **one** of the configuration guides below.

### Configuring within the Unified Package

1. Create a new file called `channel_group_query.sql` in your root dbt projects macro folder

![](/assets/images/Screenshot00-6644eabdb84ee5aa899f2ff1d594a1de.png)

2. Open [this link](https://github.com/snowplow/dbt-snowplow-unified/blob/main/macros/field_definitions/channel_group_query.sql) to find the GitHub link to the channel\_group\_query.sql macro within Unified Package. Within the file there are three macros, find the one relevant to you:

   1. bigquery\_\_channel\_group\_query - BigQuery
   2. redshift\_\_channel\_group\_query - Redshift
   3. default\_\_channel\_group\_query - Any other Data Warehouse or Query Engine

3. Copy the relevant macro into the file you created within step 1. Ensure you copy the full macro.

```sql
{% macro default__channel_group_query() %}
...
{% endmacro %}
```

4. Customize the `CASE` statement to capture your requirements. For example, you may want to set `Unassigned` to be called `Direct`.

```sql
CASE
...
	else 'Direct'
END
```

5. Next time you run the Unified Package the macro you created will be prioritized over the built-in package macro.

### Configuring within the Attribution Package

1. Create a new file called `channel_classification.sql` in your root dbt projects macro folder

![](/assets/images/Screenshot31-f4804a0bf764df025ef3e2e429b37a14.png)

2. Paste the following into the file you created within step 1.

```sql
{% macro default__channel_classification() %}

  default_channel_group

{% endmacro %}
```

3. Customize the macro to capture your requirements. See the example below on how you may want to set traffic to the channel `Unassigned` to be called `Direct`.

   The `default_channel_group` field is from the Unified Package.

```sql
{% macro default__channel_classification() %}

  CASE
	  WHEN default_channel_group = 'Unassigned' THEN 'Direct'
	  ELSE default_channel_group
	END

{% endmacro %}
```

4. Next time you run the Attribution Package the macro you created will be prioritized over the built-in package macro.

---

# Set up the Attribution dbt package locally
> Install and configure the Snowplow Attribution dbt package locally. Set up conversion sources, attribution models, and generate channel and campaign attribution tables for marketing analysis.
> Source: https://docs.snowplow.io/tutorials/attribution/setting-up-locally/

1. Create a new dbt project in a new directory

> **Info:** Please Note: Skip this step if you want to add the Attribution package to the same project as Unified.

```bash
dbt init
```

2. Create a `packages.yml` file in the same directory as the newly created `dbt_project.yml`

> **Info:** Please Note: Skip this step if you want to add the Attribution package to the same project as Unified.

3. Add the following code to your `packages.yml` file

```yml
packages:
  - package: snowplow/snowplow_attribution
    version: 0.4.0
```

4. Run the following command to install the package

```bash
dbt deps
```

5. Now open your `dbt_project.yml` where we will configure the package. A full list of configuration options can be seen on [this page](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/attribution/).

6. Add the following to your `dbt_project.yml` file and configure it accordingly for your use-case.

```yml
vars:
  snowplow_attribution:
    # Attribution Package configuration
    snowplow__attribution_start_date: 'YYYY-MM-DD' # Required, the start date for conversions to be processed from.
    snowplow__attribution_list: ['first_touch', 'last_touch', 'linear', 'position_based'] # Optional, by default all are calculated

    # Configure input sources
    snowplow__conversion_path_source: 'my_schema_derived.snowplow_unified_views' # Location of your snowplow_unified_views table
    snowplow__conversions_source: 'my_schema_derived.snowplow_unified_conversions' # Location of your snowplow_unified_conversions table
```

> **Info:** Please Note: If you are running Unified in the same project as Attribution and intend to run them in the same run, set the conversion\_path\_source and conversions\_source to the following so that dbt can run the models in the correct order:

```yml
vars:
  snowplow_attribution:
    snowplow__conversion_path_source: "{{ ref('snowplow_unified_views') }}"
    snowplow__conversions_source: "{{ ref('snowplow_unified_conversions') }}"
    snowplow__user_mapping_source: "{{ ref('snowplow_unified_user_mapping') }}"
```

7. Run the dbt package

```yml
dbt run --select snowplow_attribution
```

> **Info:** Please Note: If you are running Unified in the same project as Attribution and intend to run them in the same run, you can execute dbt run instead, but better handle it through a custom selectors.yml file

8. After running the package you should see the following tables in your `my_schema_derived` schema:

- `SNOWPLOW_ATTRIBUTION_PATHS_TO_CONVERSION` - Row per conversion that shows the channel and campaign path, both the raw path and the transformed path.
- `SNOWPLOW_ATTRIBUTION_CHANNEL_ATTRIBUTIONS` - Row for each conversion and each channel step to show how much revenue is attributed to that channel step for each attribution methodology.
- `SNOWPLOW_ATTRIBUTION_CAMPAIGN_ATTRIBUTIONS` - Row for each conversion and each campaign step to show how much revenue is attributed to that campaign step for each attribution methodology.

1. Additionally, you will have a `attribution_overview` view in your `my_schema_derived` that is useful to ingest directly into a BI tool to provide a summary of attributed conversions and revenue by attribution method and touchpoint (by default it shows the last 30 days, however this can be configured). See the optional guide below on how to integrate marketing spend data to calculate ROAS within the attribution overview view.

---

# Run the Attribution dbt package using Snowplow Console
> Configure the Attribution data model in Snowplow Console with a fully managed service. Set attribution start dates, path lookback windows, and schedule automated model runs.
> Source: https://docs.snowplow.io/tutorials/attribution/setting-up-via-console/

Once you are happy with how your local configuration is working, you will want to schedule it to run regularly. Snowplow provides a fully managed service for running data models in securely.

To get started, follow these steps:

1. Commit the local configuration you built previously to a Git repository.

2. Create a Git connection to your repository. In Console, navigate to **Destinations > Connections > Set up connection > Git connection**

3. Create a warehouse connection. In Console, navigate to **Destinations > Connections > Set up connection > Data modeling connection**

4. Create a model project. In Console, select **Modeling > Model projects > Create model project**

5. Add a run configuration with a schedule to your model project. Click **Add run configuration** and **Add schedule**. Pick your preferred schedule, e.g. daily at 00:00 AM.

For a full reference on running data models via Snowplow Console, see [Run data models from Console](/docs/modeling-your-data/running-data-models-via-console/).

---

# Add event specifications to the todo tracking plan
> Create event specifications with implementation instructions, entity cardinality rules, and trigger details. Define precise expectations for custom events and make them available through Snowtype code generation.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/add-event-specs/

Now you can create the Event Specification representing the todo addition. Click `Create new event specification`, fill the event information modal with the following inputs and click `Save and continue`.

![](/assets/images/start-add-todo-1ed91f0215ecfcc6cabad85de9f6a9c1.png)

The next step is to add the event Data Structure `todo_action` you created previously to represent the todo addition interaction. Select the Event Specification and on the `Event data structure` panel click `Add existing data structure` and select the `todo_action` custom Data Structure.

![](/assets/images/todo-action-77d7ebfbdbadf4fcf7f1b68a1f7f33a2.png)

To make sure your intention on this Event Specification is clear and also instructions for implementation are as precise as possible, you should go ahead and add instructions for this event. Click the `Add instructions` button and click `Edit` on the `value` attribute. Now you can fill the instructions with the following inputs, indicating exactly what is expected for this event.

![](/assets/images/implementation-instructions-667f4e45a2a336cfa426f1cb360675bb.png)

> **Tip:** Information for implementation instructions, cardinality rules and trigger details will be available for the implementation engineers directly through Snowtype [instructions feature](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#generating-event-specification-instructions).

Now on the Entity data structures section, click `Add existing data structure`, find and select the todo entity created earlier. On the next modal step you can define the expected cardinality, of this entity on the event specification. For the todo entity, you want to have exactly 1 instance.

![](/assets/images/entity-cardinality-59314c191b771cbe52ac8f39bec15bd3.png)

Finally you should add a trigger to represent exactly the conditions when this event should be fired. For this case, you expect the event to be fired when a user hits the Enter key after adding a title for the todo.

![](/assets/images/trigger-70bafef9a5e2cc969a93262e2fda6f91.png)

**A similar process can be taken to create both the completion and removal Event Specifications.**

---

# Create a base tracking plan from the web template in Console
> How to create a base tracking plan using the Base Web template in Snowplow Console and connect it to a source application.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/create-base-dp/

After creating a Source Application, the recommended way to keep track of what we consider base events for a tracking setup is through the Base tracking plan templates. For this application you can use the [Base Web](/docs/event-studio/tracking-plans/templates/#base-web).

To create a Base Web tracking plan for your application, navigate to the Tracking Plans section and after clicking the `Templates` button, select the Base Web template. ![](/assets/images/create-dp-1b8065bc8869fba27caad81fb0108b31.png)

By default, a Base tracking plan is not connected to a Source Application and will show all the Base Event application IDs, so for this you need to edit the tracking plan and set the Source Application to `Todo Web Application`. ![](/assets/images/create-base-dp-inputs-b5e696d2fd1ff18edf58f0457cce57d6.png)

The Base Web (_or Mobile_) tracking plan will monitor and count base events as they are sent for the selected Source Application app IDs, as we will see later on. There is no need for additional implementation.

---

# Create a custom tracking plan for todo behavior
> How to create a custom tracking plan in Snowplow Console with event specifications for tracking todo interactions.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/create-custom-dp/

As in every application, there are interactions that are important to the business and are measured in a custom way. For this application the interactions you need to measure are:

- Adding a todo
- Completing a todo
- Removing a todo

We are going to show how you can create the tracking plan for the goal interactions and the Event Specification for a todo addition.

### Create the custom tracking plan

To start off, navigate to the Tracking Plans section and click through the `Create` button and `Start` to create a new custom tracking plan.

On the basic information screen you can add the inputs as shown below and also select the `Todo Web Application` as the connected Source Application. You can name the tracking plan something similar to `Todo - Goals`.

![](/assets/images/create-todo-goals-84c7ef88e9a43363010c42af0bcbf963.png)

Finally click `Create and continue`.

> **Tip:** When connecting a Source Application to a tracking plan there are a few benefits you get automatically:
>
> - All Event Specifications inherit the expected app IDs from the Source Application. _That way you can be sure that an Event Specification is fired in all environments it needs to._
> - All Event Specifications automatically reference the Application Context Data Structures that are expected to be available with each event. _This prevents duplicate information and also understanding which context is going to be available in addition to your Event Specification entities._

---

# Create a source application for tracking plans
> How to create a source application in Snowplow Console, including setting application IDs and application context entities.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/create-source-application/

The basis for every tracking setup is the Source Application. It represents the tracking estate, in a specific platform (_web, Android, iOS, etc._), for an application which in our case would be named _'Todo Web Application'_.

Navigate to the Source Applications section and click the `Add a source application` button. ![](/assets/images/add-sap-992a308efd2fc2718eb7bed71f09d74f.png)

The inputs on the creation screen are:

- **Name**: a name used for this application which is fairly understood across the organization.
- **Description**: a few words on what this Source Application represents and/or any notes.
- **Primary owner**: owner email address.
- **Application IDs**: the [Application IDs](https://docs.snowplow.io/docs/understanding-tracking-design/organize-data-sources-with-source-applications/#application-ids) expected to be used for this application. _These will automatically flow down to the tracking plans and Event specifications you define relating to this Source Application._
- **Application entities**: here is where you will set the [Application Contexts](https://docs.snowplow.io/docs/understanding-tracking-design/organize-data-sources-with-source-applications/#application-context) you will implement and expect to be available with every event hit coming from this application.

An example of inputs can be the following which you can adjust to your case: ![](/assets/images/filled-sap-721f9176201822acaba16fe46bb84fde.png)

---

# Generate tracking code from tracking plans with Snowtype
> Generate type-safe tracking code and implementation instructions with Snowtype. Use generated APIs to track event specifications and verify events with Snowplow Inspector and Console volume counts.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/generate-tracking-code/

On your terminal now run `npx @snowplow/snowtype@latest generate --instructions`. If all is as expected, after a few seconds, you will have the following files generated:

![](/assets/images/fs-93f506090adab7da1c8deb07e7f41fb8.png)

The `src/tracking/instructions.md` file includes detailed instructions and information about the Event Specifications to be implemented while `src/tracking/snowplow.ts` contains all the required code to be used to track the Event Specifications.

> **Info:** In some editors like [Visual Studio Code](https://code.visualstudio.com/), the APIs that are available in a project are shown to the developer as they type. For Snowtype exposed APIs to track Event Specifications or event Data Structures start with `track` and then the name of the Data Structure or Event Specification. For Event Specification APIs, there is also the suffix of `Spec` or `spec` depending on the language. E.g. for our custom tracking plan, we have available the `trackAddTodoSpec`, `trackCompleteTodoSpec` and `trackRemoveTodoSpec` methods.

## Tracking interactions

To track interactions such as adding a new todo, you can add the following piece of code at `src/pagesTodo/components/Header.tsx`

```diff
import { useState } from "react";
import { v4 } from "uuid";
import { Todo } from "../../../types";
+import { createTodo, trackAddTodoSpec } from "../../../tracking/snowplow";

// Rest of the code...

  function addItem(e: React.KeyboardEvent<HTMLInputElement>) {
    if (e.key === "Enter" && value) {
      addTodo((preItems) => {
        return [
          {
            id: v4(),
            value,
            completed: false,
          },
          ...preItems,
        ];
      });
+     trackAddTodoSpec({
+       action: "add",
+       context: [createTodo({ title: value })],
+     });
      setValue("");
    }
  }
```

Finally, you can go to the application, add a new todo and observe on the Snowplow Inspector extension as the addition Event Specification is being sent. The event contexts should include the `todo`, `event_specification` and any other extra context you might add.

![](/assets/images/inspector-spec-db0b4a3b905faf7badb73b525180fe05.png)

After a while the Event Specification volume counts for each event will be available at the tracking plans screen.

![](/assets/images/custom-dp-volumes-0a0736408362506989e7342543aff991.png)

You can checkout the completed code at the ['implemented' branch](https://github.com/snowplow-incubator/data-products-basic-tracking-recipe/tree/implemented).

---

# Track page views, activity, and link clicks with the Browser tracker
> Install and configure the Snowplow Browser tracker to capture page views, activity tracking with page pings, and automatic link click tracking using the link click plugin.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/setup-basic-tracking/

For this example application, we will use the [Browser tracker](/docs/sources/web-trackers/quick-start-guide/) which is distributed through npm.

Switch to the project root directory and then install it by running `npm install @snowplow/browser-tracker`.

Next, add the following piece of code at `src/main.tsx`.

```diff
import Todo from "./pages/Todo";
import "./styles.css";

+import {
+  enableActivityTracking,
+  newTracker,
+  trackPageView,
+} from "@snowplow/browser-tracker";

+newTracker("t1", "{{COLLECTOR_URL}}", {
+  appId: "todo-web-dev",
+});

+enableActivityTracking({
+  minimumVisitLength: 30,
+  heartbeatDelay: 10,
+});

+trackPageView();

createRoot(document.getElementById("root")!).render(
```

_For this showcase, placing the initialization code at the main file is enough._

What this code does is:

1. Initializes the tracker with the app ID representing the Todo web application in the development environment.
2. Enables activity tracking which will send periodic page pings.
3. Sends a page view when the main application component is first rendered.

You can validate this step being implemented properly using the [Snowplow Inspector](/docs/testing/snowplow-inspector/) browser extension observing Page view and Page ping events. ![](/assets/images/inspector-53e6adf2645e1fa0593a30ea2eff80c1.png)

## Add link click tracking

As a next step you will implement link click tracking for the main page link pointing to the TodoMVC website. To track this and other links on your pages, you can install the [Link click](/docs/sources/web-trackers/tracking-events/link-click/) tracking plugin. The plugin provides automatic link click tracking for all links on your page.

To enable it in the application, switch to the project root directory and then install it by running `npm install @snowplow/browser-plugin-link-click-tracking`.

Next, add the following piece of code at `src/main.tsx`.

```diff
import {
  enableActivityTracking,
  newTracker,
  trackPageView,
} from "@snowplow/browser-tracker";
+import {
+  LinkClickTrackingPlugin,
+} from "@snowplow/browser-plugin-link-click-tracking";

newTracker("t1", "{{COLLECTOR_URL}}", {
  appId: "todo-web-dev",
+  plugins: [LinkClickTrackingPlugin()],
});
```

And the following on `src/pages/index.tsx`.

```diff
import Info from "./components/Info";
+import { enableLinkClickTracking } from "@snowplow/browser-plugin-link-click-tracking";
+import { useEffect } from "react";

const Todo = function () {
+  useEffect(() => {
+    enableLinkClickTracking();
+  }, [])
```

With this code, all links that are initially rendered on the page will be tracked automatically.

You can verify this using the Snowplow inspector browser extension observing link click events after clicking the TodoMVC link. ![](/assets/images/inspector-link-32d06e32417a66782f3b57096a25e659.png)

---

# Create custom data structures for event specifications
> Define custom data structures for event and entity schemas specific to your application. Create todo_action and todo data structures to represent user interactions with specific properties and validations.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/setup-custom-ds/

Before you create the custom tracking plan for these interactions, you need to create a couple of Data Structures, `todo` and `todo_action`, fitting the use case of the Todo web application.

#### Todo action Data Structure

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "type": "object",
  "self": {
    "vendor": "com.your.organization",
    "name": "todo_action",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "description": "Event Data Structure representing an action taken for a todo",
  "properties": {
    "action": {
      "type": "string",
      "enum": [
        "add",
        "remove",
        "complete"
      ],
      "description": "The action taken for a specific todo item"
    }
  },
  "required": [
    "action"
  ],
  "additionalProperties": false
}
```

#### Todo Data Structure

```json
{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "type": "object",
  "self": {
    "vendor": "com.your.organization",
    "name": "todo",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "description": "Entity Data Structure representing a todo",
  "properties": {
    "title": {
      "type": "string",
      "description": "The title of the Todo"
    }
  },
  "required": [
    "title"
  ],
  "additionalProperties": false
}
```

_Note: You might need to publish those to the production environment depending on the pipeline you are using._

---

# Install Snowtype to enable code generation from tracking plans
> Install and initialize Snowtype CLI to generate tracking APIs from event specifications. Automatically attach event_specification entities and enable event count features without additional implementation work.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/snowtype/

After defining your Event Specifications, the next step is implementing those events on your app. To do that you are going to use Snowtype to generate the required APIs for tracking the Event Specifications.

> **Info:** For some Event Specification features, such as event counts, it is required that events are tracked together with the `event_specification` entity. Snowtype automatically attaches the entity without any additional work required from the implementation engineers.

## Installing Snowtype

After having set up a [Console API key](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#authenticating-with-the-console) you can install Snowtype on this project by switching to the project root directory and running `npm install @snowplow/snowtype@latest --save-dev`.

Since this is a project without previous Snowtype installation, we need to go through the [init flow](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#initializing-snowtype-for-your-project).

To do that, you can go to the tracking plan page and click on the `Implement tracking` button. There you can copy the second code command which relates to initializing a new Snowtype project.

![](/assets/images/sntp-init-b5daa7c6f5f4fd1508a487a3ef80b670.png)

The inputs should look like the following:

![](/assets/images/sntp-init-inputs-0b17b17c6e55a3c45786ef4825a425f1.png)

Next you add this tracking plan to the Snowtype project by copying the first code command.

![](/assets/images/sntp-patch-859dc88996d2b0cb3262be80ac075473.png)

Now your Snowtype configuration file should include the tracking plan in the `dataProductIds` array.

---

# Learn how to track web events with base tracking plans
> Learn how to implement web tracking using Event Studio, source applications, and Snowtype code generation for a React TodoMVC application.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/start/

This guide will help you understand some of the basic capabilities of tracking plans and how they can be used in practice for most tracking implementation setups.

## Prerequisites

- A [collector](/docs/pipeline/collector/) endpoint.
- A [Console API key](/docs/event-studio/implement-tracking/snowtype/using-the-cli/#authenticating-with-the-console) for generating code using Snowtype.

## What you'll be doing

This recipe will showcase how a basic tracking setup can be implemented using Event Studio capabilities such as Source Applications, tracking plans and their related Snowtype features.

This basic tracking setup will include:

- Page views and page pings
- Link clicks
- Custom events

For demonstration purposes we are going to be using a [TodoMVC](https://todomvc.com/) clone built with [React.js](https://react.dev/).

If you want to follow along you can:

1. Clone the repository using`git clone git@github.com:snowplow-incubator/data-products-basic-tracking-recipe.git`.
2. Change into the project directory and install the dependencies using `npm install`.
3. Run the development server using `npm run dev`.
4. Open <http://localhost:5173/> to see the app.

---

# Verify that your application is sending the expected tracking plan events
> Verify that your base tracking plan is receiving events from the correct environment by checking event volumes in the Event Studio Console.
> Source: https://docs.snowplow.io/tutorials/data-products-base-tracking/verify-dp-events/

As mentioned previously, if you check the `Todo - Base Web` tracking plan you created, you will be able to see the events coming in from the correct environment based on the app ID.

_Note: You might need to wait for a bit, up to a maximum of 2 hours, until the events are visible._ ![](/assets/images/base-counts-febf7ebeac5fdb5d0428b0cb307a35af.png)

---

# Automate data structure workflows with GitHub Actions
> Set up GitHub Actions workflows to automatically validate pull requests and publish data structures to development and production environments.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/automate-with-github-actions/

We'll not go into the details of creating github repositories and initial commits here, the [github docs](https://docs.github.com/) do an excellent job of that already. The next few steps will assume a working github repository containing the directory and data structure we created in the previous section. It will have two branches named `main` and `develop` which should be in sync.

## Publish to develop workflow

We would like pushes to our `develop` branch to be automatically published to our [development](/docs/testing/) environment. Github workflows can be [triggered](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/triggering-a-workflow) by all kinds of repository events. The one we are interested in here:

```yml
on:
  push:
    branches: [develop]
```

With our trigger point worked out we need to complete a series of steps:

1. Configure snowplow-cli via environment variables provided as [github action secrets](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions)
2. Checkout our repo
3. Install snowplow-cli. We'll use our [setup-snowplow-cli](https://github.com/snowplow/setup-snowplow-cli) github action here. Behind the scenes it is downloading the [latest](https://github.com/snowplow/snowplow-cli/releases/latest) snowplow-cli release and making it available via the workflow job's `path`.
4. Run the `snowplow-cli ds publish dev` command as we did earlier

The full action:

```yml
on:
  push:
    branches: [develop]

jobs:
  publish:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds publish dev --managed-from $GITHUB_REPOSITORY
```

> **Tip:** The value of the `--managed-from` flag will be displayed inside the 'This data structure is locked' banner we saw earlier in the UI. It is designed to help people track down the source of truth for this data structure.

## Publish to production workflow

In the same way we want our `develop` branch to deploy to our `develop` environment we want our `main` branch to deploy to our `production` environment.

As we saw earlier publishing to production is very similar to publishing to development. The only new thing we need here is a different workflow trigger.

```yml
on:
  push:
    branches: [main]

jobs:
  publish:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds publish prod --managed-from $GITHUB_REPOSITORY
```

## Validate on pull request workflow

A core component of version control based workflows is the [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). For our repository we would like to ensure as best we can that any data structure changes are valid and problem free before they get merged into develop. Lucky for us there is a github workflow event for that.

By combining the `snowplow-cli ds validate` command and the github workflow pull request event we arrive at this:

```yml
on:
  pull_request:
    branches: [develop, main]

jobs:
  validate:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds validate --gh-annotate
```

> **Tip:** The `--gh-annotate` flag will make the validate command output [github workflow command](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/workflow-commands-for-github-actions) compatible output. We'll see an example of what that looks like in the next section.

---

# Conclusions from the data structures in Git tutorial
> Complete your Git-based data governance workflow with version control, automated validation, approval workflows, and environment promotion for data structures.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/conclusion/

- We have seen how snowplow-cli can be used to work with data structures from the command line
- We have applied that knowledge to build github workflows which support automated validation and publication
- We have added source applications, tracking plans and event specifications to use the same approach as data structures.

This approach provides you with:

- **Version control** for your data structures with full change history
- **Automated validation** to catch errors before they reach production
- **Approval workflows** through pull requests
- **Environment promotion** from development to production
- **Integration with tracking plans** for comprehensive data governance

You now have a complete CI/CD pipeline for managing your Snowplow data structures using industry-standard tools and practices.

---

# Add GitHub workflow support for tracking plans
> Extend your Git workflow to manage source applications, tracking plans, and event specifications using Snowplow CLI and GitHub Actions automation.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/follow-up-with-data-products/

> **Info:** We originally called tracking plans "data products". You'll still find the old term used in some existing APIs and CLI commands.

Now that we have our data structures set up, we can define tracking plans to organize and document how these structures are used across our applications. We'll walk through creating source applications, tracking plans, and event specifications using the CLI, then integrate them into our automated workflows.

## Create a source application

First, we'll create a source application to represent our website that will send the `login` event we defined earlier.

```bash
snowplow-cli dp generate --source-app website
```

> **Note:** `dp` is an alias for `data-products`, from the previous name for tracking plans. Source applications and event specifications are also managed by this command.

This should provide the following output

```text
INFO generate wrote kind="source app" file=data-products/source-apps/website.yaml
```

The generated file is written to the default `data-products/source-apps` directory. Help for all the arguments available to `generate` is available by running `snowplow-cli dp generate --help`.

Let's examine the generated file:

```yml
apiVersion: v1
resourceType: source-application
resourceName: b8261a25-ee81-4c6a-a94c-7717ba835035
data:
    name: website
    appIds: []
    entities:
        tracked: []
        enriched: []
```

- `apiVersion` should always be `v1`
- `resourceType` should remain `source-application`
- `resourceName` is a unique identifier of the source applications. It must be a valid uuid v4
- `data` is the contents of the source app

> **Note:** For more information about available fields and values you can refer to the [source applications schema](https://raw.githubusercontent.com/snowplow/snowplow-cli/main/internal/validation/schema/source-application.json). Making your ide schema aware via a [language server](https://github.com/redhat-developer/yaml-language-server) should provide a much smoother editing experience.

Now let's customize our source application. We'll configure it to handle events from our production website as well as staging and UAT environments. We'll also add an owner field and remove the unused entities section.

```yml
apiVersion: v1
resourceType: source-application
resourceName: b8261a25-ee81-4c6a-a94c-7717ba835035
data:
    name: website
    appIds: ["website", "website-stage", "website-ua"]
    owner: me@example.com
```

Before syncing, we can validate our changes and preview what will happen:

```bash
snowplow-cli dp sync --dry-run
```

The command will show us the planned changes:

```text
sync will create source apps file=.../data-products/source-apps/website.yaml name=website resource name=b8261a25-ee81-4c6a-a94c-7717ba835035
```

When we're happy with the proposed changes, we can sync by removing the `--dry-run` flag:

```bash
snowplow-cli dp sync
```

After syncing, you'll be able to see your new source application in the Snowplow Console UI.

## Create a tracking plan and an event specification

Let's now create a tracking plan and an event specification by running the following command

```bash
snowplow-cli dp generate --data-product Login
```

This should provide the following output

```text
INFO generate wrote kind="data product" file=data-products/login.yaml
```

Let's see what it has created for us

```yml
apiVersion: v1
resourceType: data-product
resourceName: 0edb4b95-3308-40c4-b266-eae2910d5d2a
data:
    name: Login
    sourceApplications: []
    eventSpecifications: []
```

> **Note:** For more information about available fields and values you can refer to the [tracking plans schema](https://raw.githubusercontent.com/snowplow/snowplow-cli/main/internal/validation/schema/data-product.json). Making your ide schema aware via a [language server](https://github.com/redhat-developer/yaml-language-server) should provide a much smoother editing experience.

Let's amend it to add an event specification, and a reference to a source application:

```yml
apiVersion: v1
resourceType: data-product
resourceName: 0edb4b95-3308-40c4-b266-eae2910d5d2a
data:
    name: Login
    owner: me@example.com
    description: Login page
    sourceApplications:
        - $ref: ./source-apps/website.yaml
    eventSpecifications:
        - resourceName: cfb3a227-0482-4ea9-8b0d-f5a569e5d103
          name: Login success
          event:
            source: iglu:com.example/login/jsonschema/1-0-1
```

> **Note:** You'll need to come up with a valid uuid V4 for the `resourceName` of an event specification. You can do so by using an [online generator](https://www.uuidgenerator.net), or running the `uuidgen` command in your terminal

> **Warning:** The `iglu:com.example/login/jsonschema/1-0-1` data structure has to be deployed at least to a develop envinroment. Currently referencing local data structures is not supported

We can run the same `sync --dry-run` command as before, to see if the output is as expected. The output should contain the following lines

```bash
snowplow-cli dp sync --dry-run
```

```text
INFO sync will create data product file=.../data-products/login.yaml name=Login resource name=0edb4b95-3308-40c4-b266-eae2910d5d2a
INFO sync will update event specifications file=.../data-products/login.yaml name="Login success" resource name=cfb3a227-0482-4ea9-8b0d-f5a569e5d103 in data product=0edb4b95-3308-40c4-b266-eae29
```

We can apply the changes by using the sync command without the `--dry-run` flag

```bash
snowplow-cli dp sync
```

## Add tracking plans validation and syncing in the GitHub Actions

Now that we've modeled a source application, tracking plan and event specification, let's see how we can add them to the existing GitHub Actions workflows for data structures. You can customize your setup, use a separate repository or separate actions, but in this example we'll add tracking plan syncing and releasing into the existing workflows.

Let's modify the PR example, and add the following line. This command will validate and print the changes to the GitHub Actions log.

```yml
on:
  pull_request:
    branches: [develop, main]

jobs:
  validate:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds validate --gh-annotate

      - run: snowplow-cli dp sync --dry-run --gh-annotate
```

Tracking plans, source applications and event specifications don't have the dev and prod environments, so it's enough to sync them once. When merging to `develop`, use `sync` to push your changes as drafts. When merging to `main`, use `release` to also mark event specifications as published.

Add the `sync` command to the develop pipeline:

```yml
on:
  push:
    branches: [develop]

jobs:
  publish:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds publish dev --managed-from $GITHUB_REPOSITORY

      - run: snowplow-cli dp sync
```

Add the `release` command to the production pipeline to publish event specifications when merging to `main`:

```yml
on:
  push:
    branches: [main]

jobs:
  publish:
    runs-on: ubuntu-latest
    env:
      SNOWPLOW_CONSOLE_ORG_ID: ${{ secrets.SNOWPLOW_CONSOLE_ORG_ID }}
      SNOWPLOW_CONSOLE_API_KEY_ID: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY_ID }}
      SNOWPLOW_CONSOLE_API_KEY: ${{ secrets.SNOWPLOW_CONSOLE_API_KEY }}

    steps:
      - uses: actions/checkout@v4

      - uses: snowplow/setup-snowplow-cli@v1

      - run: snowplow-cli ds publish prod --managed-from $GITHUB_REPOSITORY

      - run: snowplow-cli dp release
```

---

# Learn how to manage data structures with Git
> Manage Snowplow data structures using Git and Snowplow CLI for version control, approval workflows, and automated validation with GitHub Actions.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/introduction/

Snowplow data structures are the artifacts defining the rules for event validation within the Snowplow data pipeline. As such, they are a description of event shapes that the pipeline will allow through, and essentially the basis for the shape of data in the warehouse.

The fact that data structures formalize what the warehouse tables look like makes them a cornerstone of Snowplow's facilities for data governance. An unintended change in a data structure could result in data consumers down the line being unable to process that data (e.g. data models breaking). That is why larger organizations guard Data Structure definitions closely, and need approval workflows to allow or disallow changes. For instance, a Data Protection Officer may want to have the final say about collected events, to ensure no PII is harvested.

On top of that, detailed change history can be crucial for such large organizations. It is important to be able to tell in fine detail what was changed, when, and by whom.

The Snowplow Console's UI has facilities to get started quickly with data structures (either using the Builder or the direct JSON editor), and is a solid tool for smaller teams. It doesn't implement such approval processes, neither does it offer such fine-grained visibility around changes.

A common solution when faced with these requirements is to move management to some form of version control platform (github/gitlab). This opens up an entire ecosystem of tools and patterns enabling all manner of custom workflows.

We have built [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) to help you bridge the gap between these repository-based workflows and Snowplow Console.

## Prerequisites

- A deployed Snowplow pipeline
- [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/) installed and configured
- A familiarity with [git](https://git-scm.com/) and an understanding of [GitHub Actions](https://docs.github.com/en/actions/writing-workflows)
- A sensible [terminal emulator](https://en.wikipedia.org/wiki/Terminal_emulator) and shell

## What you'll be doing

This tutorial will walk through creating and deploying a data structure from the command line using [Snowplow CLI](/docs/event-studio/programmatic-management/snowplow-cli/). It will then show how it is possible to automate the validation and deployment process using [GitHub Actions](https://docs.github.com/en/actions/writing-workflows).

---

# Create a local data structure with Snowplow CLI
> Generate a new data structure locally using Snowplow CLI with vendor and schema information in YAML format.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/local-setup/

Firstly we'll need a place to put things.

```bash
$ mkdir -p snowplow-structures/data-structures
$ cd snowplow-structures
```

> **Tip:** snowplow-cli data structures commands default to looking for data structures in `./data-structures`.

Now let's create our data structure. We'll create a custom event called 'login'.

```bash
$ snowplow-cli ds generate login --vendor com.example
```

> **Note:** `ds` is an alias for `data-structures`.

This should provide us the following output

```text
3:00PM INFO generate wrote=data-structures/com.example/login.yaml
```

The generated file is written to our default `data-structures` directory under a sub directory matching the `--vendor` we supplied with a filename that mirrors the name we gave the data structure. Help for all the arguments available to `generate` is available by running `snowplow-cli ds generate --help`.

> **Note:** This directory layout and file naming scheme is also followed by the [download](/docs/event-studio/programmatic-management/snowplow-cli/data-structures/#downloading-data-structures) command.

Let's see what it has created for us.

```yml
apiVersion: v1
resourceType: data-structure
meta:
  hidden: false
  schemaType: event
  customData: {}
data:
  $schema: http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#
  self:
    vendor: com.example
    name: login
    format: jsonschema
    version: 1-0-0
  type: object
  properties: {}
  additionalProperties: false
```

- `apiVersion` should always be `v1`
- `resourceType` should remain `data-structure`
- `meta.hidden` directly relates to showing and hiding [in Console UI](/docs/event-studio/data-structures/#hide-a-data-structure)
- `meta.schemaType` can be `event` or `entity`
- `meta.customData` is a map of strings to strings that can be used to send across any key/value pairs you'd like to associate with the data structure
- `data` is the actual [snowplow self describing schema](/docs/api-reference/iglu/common-architecture/self-describing-json-schemas/) that this data structure describes

---

# Modify, validate, and publish the data structure using Snowplow CLI
> Validate data structure schemas for errors and warnings, then publish to development and production environments using Snowplow CLI.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/validation-and-publishing/

Firstly we'll add a property to our data structure definition. We'd like to know if a login succeeded or failed. Our modified `login.yaml` should look like this

```yml
apiVersion: v1
resourceType: data-structure
meta:
  hidden: false
  schemaType: event
  customData: {}
data:
  $schema: http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#
  self:
    vendor: com.example
    name: login
    format: jsonschema
    version: 1-0-0
  type: object
  properties:
    result:
      enum: [success, failure]
  additionalProperties: false
```

## Validate

We should validate our changes before we attempt to publish them. Let's do that

```bash
$ snowplow-cli ds validate data-structures/com.example/login.yaml
```

> **Tip:** You can supply snowplow-cli with a directory and it will look for anything that looks like a data structure. Also given the default data structure directory is being used the previous command is equivalent to `snowplow-cli ds validate`.

You should see output similar to this:

```text
3:00PM INFO validating from paths=[data-structures/com.example/login.yaml]
3:00PM INFO will create file=data-structures/com.example/login.yaml vendor=com.example name=login version=1-0-0
3:00PM WARN validation file=data-structures/com.example/login.yaml
  messages=
  │ The schema is missing the "description" property (/properties/result)
  │ The schema is missing the "description" property (/)
```

## Publish to development

Apart from the missing descriptions everything looks good. We can fill them in later. Let's go ahead and publish our data structure to our [development](/docs/testing/) environment.

```bash
$ snowplow-cli ds publish dev
```

> **Tip:** We omit the directory here but as with other commands the default directory will get used and it will attempt to publish any data structures it can find.

The command should output something close to the following:

```text
3:00PM INFO publishing to dev from paths=[data-structures]
3:00PM INFO will create file=data-structures/com.example/login.yaml vendor=com.example name=login version=1-0-0
3:00PM WARN validation file=data-structures/com.example/login.yaml
  messages=
  │ The schema is missing the "description" property (/properties/result)
  │ The schema is missing the "description" property (/)
3:00PM INFO all done!
```

> **Note:** Publishing to `dev` will also run validation. It will only fail on ERROR notifications.

You should now be able to see your published data structure in [Console UI](https://console.snowplowanalytics.com/data-structures). If you click through from the data structure listing to view the `login` data structure you should see the following banner.

![](/assets/images/locked-57bc4e2e7e71494600acaa1b219d5b2c.png)

Any data structures published using snowplow-cli will automatically get this banner and have UI based editing disabled. It is a good idea to settle on one source of truth for each data structure to avoid potential conflicts.

## Publish to production

With our data structure deployed to develop and working as we expect we can safely publish it to production.

```bash
$ snowplow-cli ds publish prod
```

```text
3:00PM INFO publishing to prod from paths=[data-structures]
3:00PM INFO will update file=data-structures/com.example/login.yaml local=1-0-0 remote=""
3:00PM INFO all done!
```

> **Note:** Data structures must be published to `dev` before they can be published to `prod`

We have now seen how to create, validate and then publish a new data structure from the command line. Next we'll look at how to configure github actions to run validation and publishing automatically for us.

---

# Verify data structures GitHub Actions automation workflow
> Test your GitHub Actions workflow by creating a pull request with data structure changes and verifying automated validation and deployment.
> Source: https://docs.snowplow.io/tutorials/data-structures-in-git/verify-github-setup/

Now we have our workflows in place let's work through an example. Our login data structure needs some attention. Our requirements have changed and rather than 'success' and 'failure' the login result will now need to report numbers and not strings. So instead of `[success, failure]` it'll be `[200, 403]`.

Having created a [new branch](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging) called `login-results-error-codes` and making the changes locally we should end up here:

```bash
--- a/data-structures/com.example/login.yaml
+++ b/data-structures/com.example/login.yaml
@@ -10,9 +10,9 @@ data:
    vendor: com.example
    name: login
    format: jsonschema
-    version: 1-0-0
+    version: 1-0-1
  type: object
  properties:
    result:
-      enum: [success, failure]
+      enum: [200, 403]
  additionalProperties: false
```

That all looks good so we'll go ahead and push to github and [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).

We wait patiently for our validate on pull request workflow to run.

![](/assets/images/worked-pr-checks-617ef11e0920c15371b3f53399ddb864.png)

Validation has failed. To identify the problem we open the 'file' tab on the pull request and see..

![](/assets/images/worked-diff-annotated-b2288d79b1056cbcf9160efc0d829925.png)

> **Note:** Validation only takes your configured [destinations](https://console.snowplowanalytics.com/destinations) into account.

Together with the description warnings we forgot to fix earlier we have some errors. Changing the values of the enum would change the type of the `result` property which will cause problems further down the line for our data. The error suggests we need to make a major version bump to avert disaster. We'll do that (and add descriptions).

Our next attempt:

```bash
--- a/data-structures/com.example/login.yaml
+++ b/data-structures/com.example/login.yaml
@@ -10,9 +10,11 @@ data:
    vendor: com.example
    name: login
    format: jsonschema
-    version: 1-0-0
+    version: 2-0-0
  type: object
+  description: Login outcome event
  properties:
    result:
-      enum: [success, failure]
+      description: The resulting http error code of a login request
+      enum: [200, 403]
  additionalProperties: false
```

And the workflow result..

![](/assets/images/worked-pr-checks-ok-8029034775bea79ec28a93f51e200fa2.png)

Validation has passed. Now our colleagues can feedback on our changes and if everyone is happy we can merge to `develop` which will trigger our `publish-develop.yml` workflow.

![](/assets/images/worked-pub-dev-8f0d2efaa5f2dbead6a73d788976404e.png)

Finally, once we are convinced everything works we can open another pull request from `develop` to `main`, merge that and trigger our `publish-production.yml` workflow.

---

# Optional: Add a new calculation to the Flink live shopper features project
> Extend the Flink pipeline by adding a custom feature calculation to track the most viewed product brand using event data and window aggregations.
> Source: https://docs.snowplow.io/tutorials/flink-live-shopper-features/add-calculation/

This page describes how to add an additional feature to the product window. The process would be similar for other parts of the system, such as cart or purchase features.

The package `com.evoura.snowplow.operator.product` contains all classes related to the product features. The features used by the product windows are:

1. `product_view_count`
2. `avg_viewed_price`
3. `return_view_count`
4. `price_range_viewed`

These are generated as metrics for Flink by the `ProductMetricsGeneratorRichFunction` class. To generate an additional feature, you will need to update several other classes first.

In this example, you will track a new feature called `most_viewed_brand`, based on the user's most viewed product brand.

## 1. Add a field to `ProductViewEvent`

The web application tracks data on product brand—it's a field within the Snowplow [ecommerce `product_view` event](/docs/sources/web-trackers/tracking-events/ecommerce/#product-view). A full example payload is available at `.example/product_view.json`, showing how the payload would look after processing through the Snowplow pipeline.

The `brand` field isn't currently present in the `ProductViewEvent` class.

Add `brand` as a `String` instance variable, after `timestamp`.

The updated class should look like this:

```java
package com.evoura.snowplow.operator.product;

import com.evoura.operator.TimestampedEvent;

public class ProductViewEvent implements TimestampedEvent {
  public String eventId;
  public String userId;
  public String productId;
  public double productPrice;
  public long timestamp;
  public String brand;

  public ProductViewEvent() {}

  public ProductViewEvent(
      String eventId, String userId, String productId, double productPrice, long timestamp, String brand) {
    this.eventId = eventId;
    this.userId = userId;
    this.productId = productId;
    this.productPrice = productPrice;
    this.timestamp = timestamp;
    this.brand = brand;
  }

  @Override
  public long getTimestamp() {
    return timestamp;
  }

  @Override
  public String toString() {
    return "ProductViewEvent{"
        + "eventId='"
        + eventId
        + '\''
        + ", userId='"
        + userId
        + '\''
        + ", productId='"
        + productId
        + '\''
        + ", productPrice="
        + productPrice
        + ", brand='"
        + brand
        + '\''
        + '}';
  }
}
```

## 2. Deserialize the information in `ProductViewEventMap`

Because you updated the `ProductViewEvent` constructor, you need to update the `ProductViewEventMap` class as well. The class `com.evoura.snowplow.operator.product.ProductViewEventMap` is responsible for mapping a `SnowplowEvent` into a `ProductViewEvent`.

On line 32, provide the `brand` argument as `productNode.path("brand").asText()`.

The updated method should look like this:

```java
@Override
public ProductViewEvent map(SnowplowEvent snowplowEvent) throws Exception {
  String eventId = snowplowEvent.getEventId();
  String userId = snowplowEvent.getUserId();

  JsonNode payloadNode = MAPPER.readTree(snowplowEvent.payload);
  JsonNode productNode = payloadNode.path("product");

  String productId = productNode.path("id").asText("");
  JsonNode priceNode = productNode.path("price");
  double productPrice = priceNode.isNumber() ? priceNode.asDouble() : 0.0;

  return new ProductViewEvent(
      eventId, userId, productId, productPrice, snowplowEvent.getTimestamp(), productNode.path("brand").asText());
}
```

## 3. Calculate the most viewed brand

You can now calculate which brand is the most viewed by the user. All the current features are already segmented by `user_id`, so the segmentation won't be part of this.

Look at the class `com.evoura.snowplow.operator.product.ProductFeatureRollingWindow`. The method `ProductFeatureRollingWindow.processWindow` receives `Iterable<ProductViewEvent> events` as the first argument.

This variable will contain all the `ProductViewEvent`s in the defined window. For products you can choose a 5 minute or a 1 hour window. Iterate through these items to calculate the most viewed brand by the user in this window.

It will look like something like this:

```java
String mostViewedBrand =
        uniqueEvents.values().stream()
            .collect(
                Collectors.groupingBy(
                    productViewEvent -> productViewEvent.brand, Collectors.counting()))
            .entrySet()
            .stream()
            .max(Map.Entry.comparingByValue())
            .map(Map.Entry::getKey)
            .orElse("");
```

## 4. Forward the `mostViewedBrand` downstream

The value of the feature needs to be collected so that it's available for Flink.

The class `com.evoura.snowplow.operator.product.ProductFeature` is responsible for forwarding all calculations that happened in `ProductFeatureRollingWindow.processWindow`.

Start by updating the class to store the new `mostViewedBrand` feature as a `String` variable.

```java
public class ProductFeature implements Serializable {
  public String userId;
  public int uniqueProductCount;
  public double averagePrice;
  public double minPrice;
  public double maxPrice;
  public String windowSize;
  public Map<String, Long> views;
  public String mostViewedBrand;

  public ProductFeature() {}

  public ProductFeature(
      String userId,
      int uniqueProductCount,
      double averagePrice,
      double minPrice,
      double maxPrice,
      String windowSize,
      Map<String, Long> views,
      String mostViewedBrand) {
    this.userId = userId;
    this.uniqueProductCount = uniqueProductCount;
    this.averagePrice = averagePrice;
    this.minPrice = minPrice;
    this.maxPrice = maxPrice;
    this.windowSize = windowSize;
    this.views = views;
    this.mostViewedBrand = mostViewedBrand;
  }

  @Override
  public String toString() {
    return String.format(
        "Window: %s, User: %s, Unique Products: %d, Avg Price: %.2f, Min Price: %.2f, Max Price: %.2f, Views: %s, Most Viewed Brand: %s",
        windowSize, userId, uniqueProductCount, averagePrice, minPrice, maxPrice, views, mostViewedBrand);
  }
  ...
}
```

This class contains a `TypeInfoFactory` class. A `TypeInfoFactory` is a way to tell Flink what's being forwarded to the next operator.

This is required because we decided to turn off Kryo deserialization on Flink. This trades some convenience for the long-term ability to perform rolling upgrades, rescale jobs, and keep historical state readable as your data model inevitably changes.

Kryo's "black-box" binary format doesn't carry any structural information, so even a small change to a field name, order, or data type breaks compatibility and can corrupt state when jobs are upgraded. Flink's own type serializers (POJO, Avro, Protobuf, etc.) embed explicit schema metadata and versioning hooks, allowing the runtime to evolve the schema safely and transparently migrate state across savepoints. We chose POJO for this project to avoid requiring any external schema registry.

Update the `ProductFeature` `TypeInfoFactory` to forward the `mostViewedBrand` feature.

The desired output will be:

```java
public class ProductFeature implements Serializable {
  ...

  public static class InfoFactory extends TypeInfoFactory<ProductFeature> {

    public static TypeInformation<ProductFeature> typeInfo() {
      InfoFactory factory = new InfoFactory();
      return factory.createTypeInfo(null, null);
    }

    @Override
    public TypeInformation<ProductFeature> createTypeInfo(
        Type t, Map<String, TypeInformation<?>> genericParameters) {
      Map<String, TypeInformation<?>> fields = new HashMap<>();

      fields.put("userId", Types.STRING);
      fields.put("uniqueProductCount", Types.INT);
      fields.put("averagePrice", Types.DOUBLE);
      fields.put("minPrice", Types.DOUBLE);
      fields.put("maxPrice", Types.DOUBLE);
      fields.put("windowSize", Types.STRING);
      fields.put("views", Types.MAP(Types.STRING, Types.LONG));
      fields.put("mostViewedBrand", Types.STRING);

      return Types.POJO(ProductFeature.class, fields);
    }
  }
}
```

Next, go back to `ProductFeatureRollingWindow.processWindow` and pass `mostViewedBrand` on at line 75.

```java
    out.collect(
        new ProductFeature(
            ctx.getCurrentKey(),
            uniqueViewsCount,
            avgUniquePrice,
            uniqueViewsCount == 0 ? 0.0 : priceStats.getMin(),
            uniqueViewsCount == 0 ? 0.0 : priceStats.getMax(),
            windowIdentifier,
            productViewCounts,
            mostViewedBrand));
```

## 5. Get the feature in Redis

The class `com.evoura.snowplow.operator.product.ProductMetricsGeneratorRichFunction` is responsible for converting the generated metrics into something the Redis operator understands. The method `ProductMetricsGeneratorRichFunction.processElement` has multiple feature collections.

Add your new `mostViewedBrand` feature after the existing blocks:

```java
out.collect(
        new MetricValue(
            String.format(
                "user:%s:most_viewed_brand_%s", productFeature.userId, productFeature.windowSize),
            String.valueOf(productFeature.mostViewedBrand)));
```

## 6. Re-run the pipeline

After these changes, re-run the script `./up.sh`. It will rebuild the docker image and restart the containers. You should be able to see in Redis the most viewed brand by the user.

![image.png](/assets/images/live-shopper-add-calculation-8c81055955e4a38fd50764ba080102ce.png)

---

# Understand the live shopper feature calculations
> Learn how rolling and session windows process Snowplow events in Flink to calculate product views, cart behavior, and session analytics metrics.
> Source: https://docs.snowplow.io/tutorials/flink-live-shopper-features/calculations/

This section explains what is calculated, how each metric is updated, and where the values are stored.

- Metrics are calculated in-stream, so they’re always fresh
- Rolling windows offer an up-to-date view over the last N minutes or hours
- Session windows account for user inactivity, producing clean session data
- Redis keys are predictable, making metrics easy to use across services

## How features are calculated

Everything starts with the entry point class `com.evoura.snowplow.SnowplowAnalyticsPipeline`, which is responsible for:

- Creating a Kafka source (a Flink operator to read the data)
- Parsing the data into a known object (`com.evoura.snowplow.model.SnowplowEvent`)
- Branching the data for multiple windowed calculations
- Defining how each window processes the data
- Sinking the features into Redis

At a high level, the workflow is:

1. The Snowplow tracker sends raw events to the Snowplow pipeline, which processes them and forwards them to Flink
2. Flink parses events, assigns more timestamps, and sends each event down a dedicated branch
3. Each branch runs a windowed function that maintains just enough state to update the metric
4. The function emits a `MetricValue` object at a fixed cadence (30 seconds or 1 minute)
5. Flink writes the metric to Redis using a predictable key (`user:{id}:feature:{name}_{window}`)

## Window types

| Window type | Purpose                  | Size                    | Emit every    | Ends when            | Used in                           |
| ----------- | ------------------------ | ----------------------- | ------------- | -------------------- | --------------------------------- |
| Rolling     | Continuous, sliding view | 5 m, 1 h, 24 h          | 30 s or 1 min | Never; always shifts | Product, Category, Cart, Purchase |
| Session     | Group events per visit   | Gap-based (30 min idle) | 30 s          | No events for 30 min | Session metrics                   |

This architecture diagram shows more details about the windowing logic:

![live-shopper-calculations-architecture.png](/assets/images/live-shopper-calculations-architecture-66d8b09fcc96a1e2544ae37edf3276d6.png)

- All rolling windows use a custom `RollingWindowProcessFunction`
- The session view uses `SnowplowSessionWindow`

For example, aggregation on a rolling window of 5 seconds and a session window with a 3-second gap would look like:

![live-shopper-calculations-window.png](/assets/images/live-shopper-calculations-window-c2782b8c083ec80a69814b3d19049c7e.png)

![live-shopper-calculations-window2.png](/assets/images/live-shopper-calculations-window2-34d42ff53db3e07982b21ceb6f6a4ce2.png)

More detail is available in [The case for a custom window in Flink: Expanding your streaming use-cases](https://pedromazala.substack.com/p/the-case-for-a-custom-window-in-flink?utm_source=snowplow\&utm_medium=accelerator\&utm_campaign=live-shopper) blog post.

## Per-feature logic

### Product views

- Event filter: `product_view`
- Key: `userId`
- Windows: 5 min, 1 hour
- Metrics: view count, average price, min-max price range _Emitted as_: `product_view_count_5m`, `avg_viewed_price_1h`, etc.

### Category engagement

- Event filter: `list_view`, `product_view`
- Key: `userId`
- Windows: 5 min, 1 hour, 24 hour
- Metrics: category view count, repeat views, top category in window

### Cart behavior

- Event filter: `add_to_cart`, `remove_from_cart`
- Key: `userId`
- Windows: 5 min, 1 hour
- Metrics: adds, removes, net cart value, cart change frequency

### Purchase history

- Event filter: checkout events (`add_to_cart`, `remove_from_cart`, `checkout_step`)
- Key: `userId`
- Window: 24 hour
- Metrics: order count, total spend, average order value

### Session analytics

- Event filter: all high-level engagement events
- Key: `sessionId`
- Window: session gap (30 min)
- Metrics: session duration, pages per session, bounce flag, cart-to-page ratio

## Redis key pattern

```text
user:{user_id}:{feature}_{window}
```

Examples:

- `user:trent@snowplowanalytics.com:product_view_count_5m`
- `user:lucas@snowplowanalytics.com:session_duration`

Downstream apps such as dashboards, ML models, or chat bots read these keys directly.

---

# Conclusions from the Flink live shopper features accelerator
> Build sub-second real-time personalization with Snowplow, Flink, Kafka, and Redis to drive higher conversion and smarter marketing.
> Source: https://docs.snowplow.io/tutorials/flink-live-shopper-features/conclusion/

Real-time streams let you see what a shopper is doing while the session is still active. Events leave the browser, flow through Snowplow, and reach Flink within milliseconds. This is important because the system can trigger actions that improve conversion.

Rolling and session windows turn raw events into metrics, and Redis makes each metric available to any service—chat bots, ML models, pricing engines, dashboards, etc.—all before the user moves on.

## Stack recap

- **Snowplow tracker**: captures clean, strictly defined events
- **Kafka**: buffers and scales traffic without spikes
- **Flink**: windows and aggregates in memory for sub-second output
- **Redis**: serves metrics with microsecond reads

All components run in Docker, so you can spin up the full pipeline with a single script—no cloud bill required.

## What you built

1. Started the stack with `./up.sh`
2. Tracked product views, cart events, and purchases in the demo store
3. Watched events flow through **Kafka** (via **AKHQ**) and metrics appear in **Redis** (via **Redis Insights**)
4. Added your own calculation (most viewed brand) and published it live
5. Verified that low-latency metrics fed downstream apps in real time

## Business impact

- **Higher conversion**: personalized offers and live chat triggers reach shoppers before they leave
- **Smarter marketing**: real-time, segmented audiences feed email, ad, and push platforms with fresh intent data
- **Data consistency**: a single source of truth powers both real-time and long-term analytics, reducing duplication and errors

With these steps, you now have a repeatable blueprint for turning live behavioral data into immediate, actionable insight.

---

# Learn how to calculate real-time shopper features using Apache Flink
> Build real-time ecommerce analytics with Apache Flink to calculate shopper features like product views, cart behavior, and session metrics for in-session personalization.
> Source: https://docs.snowplow.io/tutorials/flink-live-shopper-features/introduction/

Welcome to the **real-time shopper features using Apache Flink** solution accelerator for ecommerce.

This accelerator demonstrates how to leverage Snowplow's behavioral data to monitor and act on shopper behavior while they're still navigating your site.

Traditional analytics stacks focus on deriving insights after the fact—business intelligence dashboards that show what happened. Real-time monitoring enables proactive actions such as initiating a live chat, or sending a unique discount code based on surges in shopper activity.

In this accelerator, you'll learn how to calculate key metrics, in near real time, that can power other systems.

After you set up the project, you can explore the system via three tools:

- [**Snowplow ecommerce store**](https://github.com/snowplow-industry-solutions/ecommerce-nextjs-example-store) (left panel in below image)
- [**AKHQ**](https://akhq.io/), a Kafka visualization tool (center panel)
- [**Redis Insights**](https://redis.io/insight/), a Redis visualization tool (right panel)

All interactions on the store flow through [Snowplow Local](https://github.com/snowplow-incubator/snowplow-local) and are streamed into Kafka, where they are processed using Flink.

![Three panel screenshot showing ecommerce store and visualizations](/assets/images/live-shopper-introduction-467353b777bcfccfe7c51b1586822ec5.webp)

The computed metrics can also feed longer-term dashboards, enhancing the quality and reusability of your data.

## Prerequisites

This accelerator is fully Dockerized. The only prerequisites are Docker 28+ and Git.

## Solution accelerator code

The code for this accelerator [is available on GitHub](https://github.com/snowplow-industry-solutions/flink-live-shopper).

The project structure is:

- `apps/` contains the Flink processing application (`flink/`) and the demo ecommerce store (`ecommerce/`)
- `infra/` contains Docker Compose configurations for the infrastructure components: Snowplow, Kafka, Redis, and Localstack
- `docker-compose.yaml` is the main Docker Compose file including all services
- `up.sh` is a convenience script to initialize and start all services

## Key technologies

- Snowplow: event tracking pipeline (Collector, Enrich, Kafka sink)
- Apache Flink: stream processing engine for real-time analytics
- Apache Kafka: message broker for decoupling event producers and consumers
- Redis: in-memory data store for storing computed metrics
- Next.js: framework for the demo ecommerce application
- Docker and Docker Compose: containerization for easy setup and deployment of all services
- AKHQ: web UI for Kafka management and inspection
- Redis Insight: web UI for Redis data visualization and management
- Grafana: visualization tool for monitoring and analyzing metrics

## Architecture

This diagram shows the architectural overview:

![Architecture diagram](/assets/images/live-shopper-setup-architecture-6f92ebf8cbee9bd84ff993246c2c3924.svg)

Benefits of this architecture:

- Sub-second freshness: metrics are computed in the stream, not via nightly batch jobs, so they're actionable in-session
- Single source of truth: the same logic powers dashboards and in-session nudges
- Composable and portable: the entire system runs in Docker, and can be adapted to cloud-managed Kafka/Flink/Redis with minimal changes

The key subsystems are described below.

### Event capture and ingestion with Snowplow

- E-store front-end and Snowplow JavaScript tracker: user activity is captured as Snowplow ecommerce events
- Snowplow Local to Kafka: the Snowplow pipeline validates the events, enriches them with device and geolocation data, then forwards them into Kafka

### Real-time stream processing in Flink

- Source: a single Flink job reads from the `enriched-good` topic

- Branching by event type: the stream is split into four logical lanes (product, category, cart, purchase)

- There are two options for keying and windowing:

  - Rolling windows (5 min, 1 h, 24 h): keyed by `user_id` for always-fresh "last-N-minutes" stats
  - Session windows: keyed by `session_id`, grouping events into sessions that end after 30 minutes of inactivity

- Aggregations: each lane computes its own features e.g. view counts, average price, cart value, session duration

- Metric parsers: convert aggregated values into one or more metrics, e.g. a unique product count may feed both product view metrics and average viewed price metrics

### Feature store and action loop

- Sink to Redis: Flink writes each metric to Redis using deterministic keys like `user:{id}:{feature}_{window}` or `session:{sid}:{metric}`, making Redis a low-latency feature store

- Backend consumers: the ecommerce store back-end (or any downstream app like ML models or dashboards) can retrieve metrics in microseconds to:

  - Trigger live-chat prompts when high-value carts stall
  - Send discounts based on price sensitivity
  - Feed both real-time dashboards and long-term analytics using consistent definitions

## Acknowledgements

Thank you to the Data Streaming experts [evoura](https://evoura.com/?utm_source=snowplow\&utm_medium=accelerator\&utm_campaign=live-shopper) for their support with building this accelerator.

---

# Run the Flink live shopper features project
> Start the Dockerized Flink streaming pipeline to process Snowplow events from Kafka and store computed shopper metrics in Redis.
> Source: https://docs.snowplow.io/tutorials/flink-live-shopper-features/run/

1. [Clone the repository](https://github.com/snowplow-industry-solutions/flink-live-shopper)

2. Run the startup script. This will:

   - Ensure `.env` files exist, copying from `.env.example` if needed
   - Update Git submodules
   - Start all the services defined in the Docker Compose files, in detached mode

   ```bash
   up.sh
   ```

3. Access the main interfaces:

   - **Web application ecommerce store**: `http://localhost:3000`
   - **AKHQ (Kafka UI)**: `http://localhost:8085`
   - **Redis Insights**: `http://localhost:5540`
   - **Flink dashboard**: `http://localhost:8081`
   - **Grafana**: `http://localhost:3001`

4. When finished, stop the containers

   ```bash
   docker compose down
   ```

## Dataflow steps

1. Client-side tracking:

   - The Next.js-based ecommerce store emits user interactions (e.g., `product_view`, `add_to_cart`) using the Snowplow JavaScript tracker

2. Ingestion pipeline:

   - Events are captured by the Collector and processed via Enrich
   - Snowbridge forwards enriched events into Kafka topics

   ![Screenshot showing the snowplow-enriched-good topic](/assets/images/live-shopper-setup-kafka-dda0bf0769ae3219845c29189fb0a2d9.png)

3. Stream processing (Apache Flink):

   - Events are parsed, filtered, and routed into logical branches

   - Processing includes:

     - Product features: count views, average viewed price, price range
     - Cart behavior: add/remove counts, cart value, update frequency
     - Category engagement: category views, repeat views
     - Purchase history: aggregate purchases over rolling windows (e.g., 24 hours) to calculate total spend, order count
     - Session analytics: duration, bounce rate, marketing source

   ![Screenshot showing the Flink dashboard](/assets/images/live-shopper-setup-flink-70d69fc83f917b8701fe190d5f1a936a.png)

4. Feature store (Redis):

   - Metrics are written to Redis using deterministic keys (e.g., `user:{user_id}:{feature}_{window}`)
   - These metrics are available for real-time lookups by downstream systems

   ![Screenshot showing the Redis dashboard](/assets/images/live-shopper-setup-redis-5c2074aa029d500e04c66c45443031f3.png)

## Testing

To test the system, log in to the ecommerce store using one of the [mock users](https://github.com/snowplow-industry-solutions/ecommerce-nextjs-example-store/blob/main/src/mocks/users.ts).

Open a product listed on the homepage. You should see data flowing to the `enriched-good` topic via [AKHQ](http://localhost:8085). This is the same data ingested by Flink to derive metrics.

All calculated metrics will appear in [Redis Insights](http://localhost:5540).

If you don't see any data flowing, check that the ecommerce store is sending events correctly. Ad blockers can interfere with event tracking: turn them off for full functionality.

The data in Redis is now available to be consumed by any application, system, or process—for example, an AI agent can use this data to determine the next best action.

---

# Conclusions and next steps from the Kafka live viewer profiles accelerator
> Complete your real-time event-driven architecture for video streaming analytics with Snowplow, Kafka, and DynamoDB for live viewer insights.
> Source: https://docs.snowplow.io/tutorials/kafka-live-viewer-profiles/conclusion/

In this tutorial, you've explored the **live viewer profiles** solution accelerator for video streaming, gaining practical insights into building, deploying, and extending real-time, event-driven architectures using Snowplow and Kafka.

You have successfully built a real time system for processing event data including:

- **Web tracking application** for collecting media events ![Application Output](/assets/images/video-4b9da5d6d44a98bf239574a7d56ebcf1.png)
- **Snowplow Collector and Snowbridge** for event processing and forwarding
- **Live Viewer back-end** for managing real-time data with Kafka and DynamoDB
- **Live Viewer front-end** for visualizing real-time user activity on the web tracking application ![Live viewer frontend](/assets/images/live-viewer-df27c3494d3a88598b62d266d80769b6.png)

This architecture highlights how real-time insights can be achieved using event-driven systems in a streaming context.

## What you achieved

You explored how to:

1. Use LocalStack to emulate AWS services for local development and testing
2. Launch and interact with the system components, such as the Kafka UI and LocalStack UI
3. View and verify the real-time event data from the browser using Snowplow's media tracking capabilities
4. Deploy the solution in an AWS environment using Terraform

This tutorial can be extended to utilize Snowplow event data for other real-time use cases, such as:

- Web Engagement analytics
- Personalized recommendations
- Ad performance tracking

## Next steps

- **Extend tracking:** extend the solution to track more granular user interactions or track on a new platform such as mobile
- **Extend dashboard:** extend the Live Viewer to include information on the media being watched and the user
- **Replace the state store:** replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB

By completing this tutorial, you're equipped to harness the power of event-driven systems and Snowplow's analytics framework to build dynamic, real-time solutions tailored to your streaming and analytics needs.

---

# Deploy the Kafka live viewer profiles application on AWS using Terraform
> Deploy the live viewer profiles solution to AWS infrastructure using Terraform scripts with EC2, DynamoDB, and Kafka components.
> Source: https://docs.snowplow.io/tutorials/kafka-live-viewer-profiles/deploy-aws-terraform/

The following steps will deploy the solution accelerator to AWS using Terraform. This is an alternative to the Localstack method. There is no need to manually install [Terraform](https://www.terraform.io/). It is executed via [Docker](https://www.docker.com/) using the `terraform.sh` script.

## Step 0: Prerequisites

1. Open a terminal
2. Install **Docker** and **Docker Compose**
3. [Clone the project](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) and navigate to its directory

```bash
git clone https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles.git
```

4. Create a `.env` file based on `./docker/.env.example` and configure AWS variables.

```bash
ACCEPT_LICENSE="true"
AWS_REGION=eu-west-2
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxx
```

## Step 1: Initialize the project

```bash
$ ./terraform/terraform.sh init
```

## Step 2: Create the infrastructure

```bash
$ ./terraform/terraform.sh apply
```

## Step 3: Access the EC2 instance that runs the apps in AWS

```bash
$ ./terraform/apps/ssh.sh
```

Inside the EC2 instance, you can control the Docker images in a similar way to how you do locally:

```bash
$ cd snowplow-demo

$ ./stats.sh # <- show the statistics for the Docker containers
$ ./down.sh  # <- stop the Docker containers
$ ./up.sh    # <- start the Docker containers
```

## Step 4: Open access to the applications

Review the [LocalStack guide](/tutorials/kafka-live-viewer-profiles/quickstart-localstack/) for the default configuration for each component. Open public access to the two front-end applications and the Snowplow Collector using a HTTP load balancer so that anyone can watch the video, submit events to the pipeline, and see information on concurrent users.

The applications listen for HTTP traffic on the following ports

- Web tracker front-end - 3000
- Live viewer front-end - 8280
- Snowplow Collector - 9090

## Next Steps

- You can implement Snowplow media tracking on any [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/) or [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/) media of your choice
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user.
- Replace Amazon DynamoDB with an alternative to be cloud agnostic, e.g. Google Bigtable or MongoDB.

***

## Other commands

### Check versions

```bash
$ ./terraform/terraform.sh --version
```

Example response

```bash
Terraform v1.10.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.79.0
+ provider registry.terraform.io/hashicorp/local v2.5.2
+ provider registry.terraform.io/hashicorp/tls v4.0.6
```

### Check the Terraform plan

```bash
$ ./terraform/terraform.sh plan
```

### Generate a PNG image for the Terraform modules in this project

```bash
$ ./terraform/terraform.sh png
```

Current PNG image of the available modules:

![Terraform Modules](/assets/images/terraform-10e5f3377b0d5e1909fe44c79924f7a8.png)

### Destroy the infrastructure

```bash
$ ./terraform/terraform.sh destroy
```

---

# Learn how to create live viewer profiles using Kafka
> Build real-time viewer profiles for video streaming platforms using Snowplow, Kafka, Java, and DynamoDB to track live user interactions and engagement.
> Source: https://docs.snowplow.io/tutorials/kafka-live-viewer-profiles/introduction/

Welcome to the **live viewer profiles** solution accelerator for video streaming.

This accelerator demonstrates how to build a real-time use case leveraging **Snowplow event data** to create live viewer profiles for a video streaming site. By combining Snowplow's streaming pipeline with **Apache Kafka**, a **Java application** and **AWS DynamoDB**, the solution processes live streaming events to visualize user interactions with video content and advertisements.

On the left side of the image below, someone is watching a video. Their events are sent through a Snowplow pipeline to Kafka where they are consumed and processed by an application. The result of this processing is displayed in the right window. This shows the number of active users and their current state.

![Application Output](/assets/images/one-viewer-2db2b92ead2e4e0cdce602d8d0c74d0f.png)

Through this hands-on guide, you'll learn how to build, deploy, and extend real-time, event-driven architectures using Snowplow and Kafka, enabling personalized recommendations, real-time insights, and dynamic analytics for streaming platforms. The framework is inspired by common challenges in video streaming, including tracking user behavior, ad engagement, and session activities, with the goal of maintaining up-to-date viewer profiles in DynamoDB.

This accelerator is open source and can serve as the foundation to build practical applications like real-time viewer insights, engagement analytics, ad performance tracking, and personalized recommendations. Whether you're optimizing ad placements or enhancing viewer satisfaction, this guide equips you to unlock the full potential of Snowplow event data.

Please start by reviewing how the application works in the next page on Localstack, even if you're planning to deploy with Terraform.

## Solution Accelerator code

The code for this infrastructure is available [here on GitHub](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles).

## Architecture

The solution comprises several interconnected components:

- **Web tracking application**:

  - A React application with a video to watch
  - Snowplow's media tracking has been configured to send events (e.g., play, pause, ad skipped) to the [Snowplow Collector](/docs/fundamentals/)
  - Code available in [tracker-frontend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/tracker-frontend) folder in GitHub

- **Snowplow Collector**:

  - Collects and forwards events via [Stream Enrich](/docs/fundamentals/) and Kinesis to [Snowbridge](/docs/api-reference/snowbridge/)

- **Snowplow Snowbridge**:

  - Publishes events to Kafka for the Live Viewer back-end to consume

- **Live Viewer back-end**:

  - A Java application which processes events from Kafka, stores the data in DynamoDB, and generates JSON state data for the Live Viewer front-end
  - Code available in [live-viewer-backend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/live-viewer-backend) folder in GitHub

- **Live Viewer front-end**:

  - A HTML website which displays the state of users currently watching the video
  - Code available in [live-viewer-frontend](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles/tree/main/live-viewer-frontend) folder in GitHub

The following diagram maps out where each component sits in the end-to-end communication flow. ![Architecture Diagram](/assets/images/architecture-5be2ba8f8c4e631e43172a4a2d1e493a.png)

### Components and configuration

The following files in the [GitHub repository](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) can be used to configure the project's components:

- **Snowplow components**: `docker/compose.snowplow.yaml`
- **Kafka infrastructure**: `docker/compose.kafka.yaml`
- **Application components**: `docker/compose.apps.yaml`
- **LocalStack setup**: `docker/compose.localstack.yaml`
- **AWS setup**: Terraform scripts (located in the `docs/terraform` folder)

## Acknowledgements

Thank you to the Kafka experts [OSO](https://oso.sh/) for their support with building this accelerator.

---

# Run the Kafka live viewer profiles application using LocalStack
> Run the complete live viewer profiles stack locally with Docker, LocalStack, and Snowplow to process media tracking events and display real-time user states.
> Source: https://docs.snowplow.io/tutorials/kafka-live-viewer-profiles/quickstart-localstack/

The following steps will deploy the solution accelerator using Localstack.

## Step 0: Prerequisites

1. Open a terminal
2. Install **Docker** and **Docker Compose**
3. [Clone the project](https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles) and navigate to its directory

```bash
git clone https://github.com/snowplow-industry-solutions/kafka-live-viewer-profiles.git
```

4. Create a `.env` file based on `./docker/.env.example`
5. You can leave the AWS variables as placeholders when using Localstack

```bash
ACCEPT_LICENSE="true"
AWS_REGION=eu-west-2
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxx
AWS_ENDPOINT_URL=http://localstack:4566
```

## Step 1: Start the containers

Run the following command to download and run everything in Docker:

```bash
./docker/up.sh
```

Details on everything that's installed can be found in the [architecture](/tutorials/kafka-live-viewer-profiles/introduction/#architecture) section on the previous page.

**Tips:**

- Use `Ctrl+C` to stop services but keep containers running
- Pass service-specific options to `./docker/up.sh` (e.g., `./docker/up.sh kafka-services`)

## Step 2: Open the web tracking front-end

Visit <http://localhost:3000> to configure the Stream Collector endpoint and start tracking events. Enter the Collector URL: `localhost:9090` and click `Create tracker`.

![First page of tracking website](/assets/images/tracker-demo-721e93110218fc7e08aa453985b6d823.png)

On the next screen, click `Custom media tracking demo`. This will bring up a video and a screen that displays information on what events are sent from the browser to the pipeline. If you want to simulate multiple users watching the video at the same time, you can open this in separate browsers.

![Welcome page on tracking website](/assets/images/welcome-page-aadae418d8f57badba9bb14166e6cf3e.png)

You must keep this window open with the video playing because everything here is running in real-time. You can use the [Snowplow Chrome Plugin](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm) to verify that the events are successfully sent from the web browser.

![Video playing on the website](/assets/images/video-4b9da5d6d44a98bf239574a7d56ebcf1.png)

## Step 3: Open the Live Viewer front-end

Open <http://localhost:8280> in a separate window. This will display the active users and their current state (e.g. watching video, watching advertisement, paused).

![Live viewer frontend](/assets/images/live-viewer-df27c3494d3a88598b62d266d80769b6.png)

Congratulations! You have successfully run the accelerator to stream web behavior through Snowplow and Kafka to a real-time dashboard.

## Next Steps

- You can implement Snowplow media tracking on any [HTML5](/docs/sources/web-trackers/tracking-events/media/html5/) or [YouTube](/docs/sources/web-trackers/tracking-events/media/youtube/) media of your choice
- Look into the output from Kafka and extend the Live Viewer to include information on the media being watched and the user.
- Use our supplied Terraform in the next section to run this on AWS and make it publicly available.

## Other things you can do

### View Events in Kafka UI

Access <http://localhost:8080> to review events within the Kafka UI.

### Manage containers with LazyDocker

Run the following command to manage containers visually:

```bash
sudo ./docker/lazy.sh
```

### Inspect infrastructure with LocalStack UI

Visit the [LocalStack UI](https://app.localstack.cloud/) to inspect infrastructure components such as Kinesis and DynamoDB. Please note that a LocalStack account is required to view this.

## Cleaning up

### Stop the containers

Shut down all running containers:

```bash
./docker/down.sh
```

### Clean up and delete

To remove all containers and images, use:

```bash
./docker/clean.sh
```

This command will delete all generated data.

---

# Conclusions and next steps from the Real-time Editorial Analytics accelerator
> Complete your real-time event-driven architecture for editorial analytics with Snowplow and ClickHouse for real-time insights.
> Source: https://docs.snowplow.io/tutorials/realtime-editorial-analytics-clickhouse/conclusion/

You've completed the real-time editorial analytics solution accelerator. You've gained practical experience building, deploying, and extending real-time, event-driven architectures using Snowplow and ClickHouse.

You have successfully built a real-time system for processing event data including:

- **Web tracking application** for collecting article interaction and ad performance events.
- **Snowplow Micro and Snowbridge** for event processing and forwarding. You explored how to use Snowplow Micro to emulate a full Snowplow pipeline for local development and testing.
- **ClickHouse** for processing and storing real-time event-level data.
- **Editorial Analytics Dashboard front-end** for visualizing real-time content engagement behavior on the web tracking application.

![The Daily Query Analytics Dashboard showing Trending Articles, Trending Categories, and Ad Performance sections, each with metrics for the last 30 minutes including impressions, views, scroll depth, and engaged time](/assets/images/realtime-dashboard-f03b1b16e3bd3aed3b6f41938004109b.png)

This architecture highlights how real-time insights can be achieved using event-driven systems in a streaming context.

This accelerator can be extended to use Snowplow event data for other real-time use cases, such as:

- Web engagement analytics
- Personalized content recommendations
- Ad performance tracking
- Search and content re-ranking

## Next steps

The following extensions are a good starting point for building on this accelerator:

- **Extend tracking:** extend the solution to track more granular user interactions or track on a new platform such as mobile
- **Personalize content recommendations:** extend the solution to change the Featured Article on the homepage based on the most-viewed article from the last 30 minutes
- **Exclude content:** extend the solution to exclude any content already viewed by the current user
- **Real-time content scoring:** extend the solution to compute a real-time score for each article, based on the latest engagement

By completing this tutorial, you're equipped to build event-driven systems that deliver real-time analytics tailored to your streaming needs.

---

# Learn how to perform real-time editorial analytics with Snowplow and ClickHouse
> Perform real-time editorial analytics for media publishers using Snowplow and ClickHouse to track real-time article engagement and performance.
> Source: https://docs.snowplow.io/tutorials/realtime-editorial-analytics-clickhouse/introduction/

Welcome to the **Real-time Editorial Analytics** solution accelerator for media publishers.

This accelerator demonstrates how to leverage real-time **Snowplow event data** with **ClickHouse Cloud** to understand article engagement and user behavior on a media publisher site.

The image below shows the accelerator in action. On the left side, a user is engaging with a media website, reading articles and clicking different advertisements. The website sends events through a local Snowplow Micro pipeline.

The left side of the image shows the Snowplow events in the Snowplow Micro dashboard.

![Side-by-side view of The Daily Query demo website on the left and the Snowplow Micro dashboard on the right, showing captured article engagement events in a table](/assets/images/snowplow-tracking-website-3cc7cb851a7e762272b3a1221fbbdfbf.png)

From here, events are forwarded to a ClickHouse table in near real-time latency. Each event is stored as an individual row in a single table, as seen on the left side of the image below.

On the right is an example dashboard, hosted as part of the accelerator website, which queries ClickHouse for real-time article engagement and ad performance metrics from the previous 30 minutes.

![Side-by-side view of the ClickHouse SQL console showing the snowplow\_article\_interactions table on the left, and The Daily Query Analytics Dashboard showing trending articles and ad performance metrics on the right](/assets/images/editorial-analytics-dashboard-71fa110b1be2c2aebc6c99e5ba3ce720.png)

Through this hands-on guide, you'll learn how to build, deploy, and extend real-time, event-driven architectures using Snowplow and ClickHouse. The framework is inspired by real customer use cases in media.

This accelerator is open source (Apache 2.0). Feel free to use it as the foundation for practical applications such as real-time viewer insights, engagement analytics, ad performance tracking, or personalized content recommendations.

## Solution accelerator code

The code for this infrastructure is available [here on GitHub](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics/tree/main).

## Architecture

The solution comprises several interconnected components:

- **Web tracking application**:

  - A Next.js application with a number of articles and advertisements
  - Snowplow tracking for events related to article engagement, e.g., article impressions, article views, ad impressions, ad clicks, or page pings, sent to the [Snowplow Collector](/docs/fundamentals/)
  - Code available in the [`snowtype.ts`](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics/blob/main/website/snowtype/snowplow.ts) file in GitHub

- **Snowplow Micro**:

  - [Snowplow Micro](/docs/testing/snowplow-micro/) is a lightweight version of the Snowplow pipeline which can be run locally
  - It passes validated and enriched events to [Snowbridge](/docs/api-reference/snowbridge/)

- **Snowplow Snowbridge (also known as [Event Forwarding](/docs/destinations/forwarding-events/))**:

  - Filters incoming Snowplow events to only forward a subset of events and dimensions to ClickHouse
  - Publishes events and lands events in a single table in ClickHouse using ClickHouse's [HTTP interface](https://clickhouse.com/docs/interfaces/http)

- **ClickHouse Cloud**:

  - [ClickHouse Cloud](https://clickhouse.com/) receives and stores events from Snowplow
  - Stored data can be queried using ClickHouse's UI or via API

The following diagram maps out where each component sits in the end-to-end communication flow.

![Architecture diagram showing event data flowing from a front-end website through Snowplow Micro (Collector, Schema Validation, Enrichments) and Snowbridge (Event Forwarding, Transformation and Filters) to a ClickHouse table, which is queried by a Personalization Service and the Content Editorial Team](/assets/images/architecture-eb542ecb2ae024c72d67248fcfe5892e.png)

## Prerequisites

You'll need a ClickHouse Cloud account to receive the Snowplow events. A [30-day free trial](https://clickhouse.com/cloud) signup is available.

## Acknowledgments

Thank you to the [ClickHouse](https://clickhouse.com/) team for their support and collaboration with building this accelerator.

---

# Run the application using Snowplow Micro and ClickHouse
> Run the complete real-time editorial analytics stack locally with Docker, Snowplow and ClickHouse to process user behavioral events and display real-time content engagement insights and ad performance metrics.
> Source: https://docs.snowplow.io/tutorials/realtime-editorial-analytics-clickhouse/quickstart/

The following steps will deploy the solution accelerator using Docker.

## Step 0: Prerequisites

1. Open a terminal
2. Install **Docker** and **Docker Compose**. You can run the following commands to check if it's already installed. If not installed, you can install Docker / Docker Compose by following these [instructions](https://docs.docker.com/compose/install/).

```bash
  docker --version

  docker-compose --version
```

3. [Clone the project](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics) and navigate to its directory

```bash
git clone https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics.git
```

4. Run the SQL query in `./clickhouse-queries/create-table-query.sql` within ClickHouse's SQL console. This table will store the Snowplow events. You'll need a `CLICKHOUSE_DATABASE` value in the next step. It's labeled in this image:

![ClickHouse Cloud SQL console home screen with the left navigation showing the Snowplow database and its tables, with the CLICKHOUSE\_DATABASE value highlighted in the breadcrumb](/assets/images/clickhouse-database-name-2a822d0a496e1fea4f1213efcc53f5d2.png)

5. Create a `.env` file by copying `.env.example`. To populate the values, go to your ClickHouse account and select the [HTTPS connection](https://clickhouse.com/docs/getting-started/quick-start/cloud#connect-with-your-app) method. This will display a sample `curl` command containing your connection details.

Map each value to the correct environment variable:

| Variable              | Where to find it                                                                                          |
| --------------------- | --------------------------------------------------------------------------------------------------------- |
| `CLICKHOUSE_HOST`     | Line 3 of the sample `curl` command. Format: `https://<your-clickhouse-host>.aws.clickhouse.cloud:<port>` |
| `CLICKHOUSE_USER`     | The **Username** field                                                                                    |
| `CLICKHOUSE_KEY`      | The **Password** field                                                                                    |
| `CLICKHOUSE_DATABASE` | The database name from the SQL Console where the SQL query was run. This is typically `default`           |
| `CLICKHOUSE_TABLE`    | Always set this to `snowplow_article_interactions`                                                        |

![ClickHouse Connect dialog with Username and Password fields annotated as CLICKHOUSE\_USER and CLICKHOUSE\_KEY, and a curl command with line 3 annotated as CLICKHOUSE\_HOST](/assets/images/clickhouse-credentials-2d4755d109c7b620538c6c1748f3a809.png)

## Step 1: Start the containers

Run the following command to download and run everything in Docker:

```bash
docker-compose up -d
```

The [architecture](/tutorials/realtime-editorial-analytics-clickhouse/introduction/#architecture) section on the previous page has the details on everything that's installed.

## Step 2: Open the web tracking front-end

Wait for about 30 seconds for the website container to start. Once it's ready, visit [`http://localhost:3000`](http://localhost:3000) to view the website application and start tracking events.

![The Daily Query demo website homepage showing a featured article about AI in journalism and a grid of new articles below](/assets/images/homepage-433833d669609a13cef1c2ba91ea3dd9.png)

2.1 Click on any of the articles that are on the homepage. Scroll down on the new page which opens. Wait for about 10 seconds to simulate a user reading.

2.2 Click on the advertisement which appears on the right-hand sidebar. Return to the homepage by clicking the **The Daily Query** logo in the header.

![The Daily Query demo website article page with a Professional Development Courses advertisement in the right sidebar](/assets/images/advertisement-ee4a00ab1bd585c370c43c86ddeecd03.png)

2.3 Select a different article from the homepage. Scroll down on the new page which opens. Wait for about 10 seconds to simulate a user reading.

2.4 Click on the advertisement which appears on the right-hand sidebar as you did in Step 2.2. Return to the homepage by clicking the **The Daily Query** logo in the header.

## Step 3: Open the Snowplow Micro front-end

Open Snowplow Micro on [`http://localhost:9090/micro/ui`](http://localhost:9090/micro/ui) in a separate window. Press the **Refresh** button located in the header. This will display the current Snowplow events which are being tracked (e.g. `page_view`, `page_ping`, `article_interaction`, `ad_interaction`).

You can use the **Pick Columns** button to select certain dimensions. Try selecting the following:

- `event_name` from the "Events" section
- `com_demo_ad_interaction_1.type` from the "Events" section
- `com_demo_media_article_interaction_1.type` from the "Events" section
- `com_demo_media_article_1.title` from the "Entities" section

## Step 4: Query the data in ClickHouse Console

Run the following query in ClickHouse's SQL Console. You should see events landing in real-time within the ClickHouse table.

```sql
select * from snowplow_article_interactions
order by dvce_created_tstamp desc
```

![ClickHouse SQL console showing query results from the snowplow\_article\_interactions table, with rows of event data ordered by creation timestamp](/assets/images/clickhouse-results-489eb6997e47976b047ec59398d787ae.png)

## Step 5: View the editorial analytics data in a sample real-time dashboard

Visit the real-time editorial analytics dashboard at [`localhost:3000/dashboard`](http://localhost:3000/dashboard), which is querying data from ClickHouse. Press the **Load Data** button to see the article engagement and ad performance metrics for the last 30 minutes.

![The Daily Query Analytics Dashboard showing Trending Articles, Trending Categories, and Ad Performance sections, each with metrics for the last 30 minutes including impressions, views, scroll depth, and engaged time](/assets/images/realtime-dashboard-f03b1b16e3bd3aed3b6f41938004109b.png)

If you're interested in the queries powering these insights, take a look at the code here:

- [Trending Articles report](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics/blob/main/website/app/api/dashboard/route.ts#L52)
- [Trending Categories report](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics/blob/main/website/app/api/dashboard/route.ts#L122)
- [Ad Performance](https://github.com/snowplow-industry-solutions/clickhouse-realtime-editorial-analytics/blob/main/website/app/api/dashboard/route.ts#L204)

## Step 6: Generate more insights

Try selecting different news articles, or clicking on different displayed ads. Repeat Step 4 or Step 5 and the data will refresh in real-time.

## Clean up and delete

Shut down and delete all running containers:

```bash
docker-compose down
```

> **Tip:** There will still be data in your ClickHouse Cloud account. If you want to delete the generated data, run the following command in ClickHouse's SQL Console:
>
> ```sql
> DROP TABLE snowplow_article_interactions
> ```

---

# Conclusions and next steps from the Signals batch engine tutorial
> Complete the Snowplow Signals batch engine tutorial for warehouse-based attribute calculation. Learn how to use dbt projects to compute and sync historical attributes to the Profiles Store.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/conclusion/

In this tutorial you've learned how to calculate attributes from your warehouse data, and apply them to Signals.

This is the process workflow:

- Define batch attribute group configurations and apply them to Signals
- Initialize dbt projects
- Generate models
- Configure the projects with dbt
- Create tables and snapshot by running dbt
- Connect the snapshot to Signals by syncing

Supported warehouses:

- Snowflake
- BigQuery

## Next steps

Learn more about the Signals possibilities in our other Signals tutorials:

- [Signals Quick Start](/tutorials/signals-quickstart/start/)
- [Score prospects in real time using Signals and ML](/tutorials/signals-ml-prospect-scoring/intro/)

---

# Generate dbt data models using the batch engine CLI
> Generate dbt SQL models for Snowplow Signals batch attribute processing including base models, filtered events, daily aggregates, and final attribute definitions.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/generate-models/

Each project will have its own set of models generated based on its specific schema and requirements.

For each project, the generation process will:

1. Create dbt configuration files
2. Generate SQL models based on the batch attribute group's schema
3. Set up necessary macros and functions
4. Update any existing files if needed

For each batch attribute group, the generated models are specifically designed for batch processing:

- Base models: raw data transformations
- Filtered events: event filtering and cleaning
- Daily aggregates: time-based aggregations
- Attributes: final feature definitions

## Run generate command

Depending on how you initialized your projects, you can generate models in two ways.

If you created projects for all attribute groups, you can generate models for all of them at once:

```bash
# For all attribute groups
snowplow-batch-engine generate --verbose
```

To generate models for a specific project:

```bash
snowplow-batch-engine generate \
  --project-name "user_attributes_1" \
  --target-type snowflake \
  --verbose
```

Adjust the target-type to `bigquery`, if relevant.

Remember that project names follow the format `{attribute_group_name}_{attribute_group_version}`.

## Project structure

After generation, each project in your repository will have this expanded structure:

```text
my_snowplow_repo/
├── user_attributes_1/
│   ├── dbt_project.yml        # Main dbt configuration
│   ├── packages.yml           # Dependencies configuration
│   ├── models/                # SQL models
│   │   ├── base/              # Base models
│   │   ├── filtered_events/   # Event filtering
│   │   ├── daily_aggregates/  # Aggregated data
│   │   └── attributes/        # Feature definitions
│   ├── configs/               # Configuration files
│   │   ├── base_config.json
│   │   └── dbt_config.json
│   │   └── batch_source_config.json
│   └── macros/                # Reusable SQL functions
├── product_attribute_groups_2/
│   └── ... (same structure)
└── user_segments_1/
    └── ... (same structure)
```

## Troubleshooting

If you encounter any issues during generation:

1. Check that your projects were properly initialized in the correct path
2. Review the `base_config.json` file in each project for configuration issues
3. Check that your API credentials have the necessary permissions
4. Use the `--verbose` flag for more detailed error messages

---

# Create attribute group projects using the batch engine CLI
> Initialize separate dbt projects for each Snowplow Signals batch attribute group with automated folder structure and base configuration files.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/initialize-project/

Having tested the connection, you can now initialize your projects.

When you run the initialization command, the CLI will:

1. Create a separate project directory for each relevant attribute group
2. Set up the basic configuration files for each project
3. Initialize the necessary folder structure for each project
4. Prepare each project for model generation

## Run initialize

You can generate projects for all the relevant attribute groups in Signals at once, or one at a time. Change your target-type to `bigquery` if relevant.

```bash
# For all attribute groups
snowplow-batch-engine init \
  --target-type snowflake \
  --verbose

# For a specific attribute group
snowplow-batch-engine init \
  --attribute-group-name "user_attributes" \
  --attribute-group-version 1 \
  --target-type snowflake \
  --verbose
```

Each attribute group will have its own separate dbt project, with the project name following the format `{attribute_group_name}_{attribute_group_version}`.

The files will be generated at the path specified in your `SNOWPLOW_REPO_PATH` environment variable.

## Project structure

After initialization, your repository will have a structure like this:

```text
my_repo/
├── my_attribute_group_1/
│   └── configs/
│       └── base_config.json
├── etc.
```

In this example, projects were generated for three attribute groups: `user_attributes` v1, `product_attribute_groups` v2, and `user_segments` v3:

```text
my_snowplow_repo/
├── user_attributes_1/
│   └── configs/
│       └── base_config.json
├── product_attribute_groups_2/
│   └── configs/
│       └── base_config.json
└── user_segments_1/
    └── configs/
        └── base_config.json
```

## Troubleshooting

If you run into any issues during initialization:

1. Check that you have write permissions in the target directory
2. Check that you don't already have a project with the same name as one you're trying to initialize
3. Check that your API credentials have the necessary permissions
4. Use the `--verbose` flag to get more detailed error messages

---

# Install the Signals Python SDK with batch engine CLI
> Install the Snowplow Signals batch engine CLI tool with Python pip, and configure environment variables for API authentication and warehouse attribute processing.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/install/

Choose where to generate your Signals dbt models. We recommend creating a new repository.

Navigate into your repo, and check you're in the intended Python environment.

The batch engine is part of the Signals Python SDK. It's not installed by default, as not all use cases will need it. To install it, run the following command:

```bash
pip install 'snowplow-signals[batch-engine]'
```

This will install the CLI tool as `snowplow-batch-engine`, along with the necessary dependencies.

## Available commands

The available options are:

```bash
  init              # Initialize dbt project structure and base configuration
  generate          # Generate dbt project assets
  sync       # Registers the attribute table as a data source with Signals
  test_connection   # Test the connection to the authentication and API services
```

A `--verbose` flag is available for every command.

## Set up environment variables

To make your workflow smoother, set up your Signals credentials as environment variables. This way, you won't need to type them in every command:

```bash
export SNOWPLOW_API_URL="YOUR_API_URL"
export SNOWPLOW_API_KEY="YOUR_API_KEY"
export SNOWPLOW_API_KEY_ID="YOUR_API_KEY_ID"
export SNOWPLOW_ORG_ID="YOUR_ORG_ID"
export SNOWPLOW_REPO_PATH="./my_snowplow_repo" # Path to the repo where you want to generate projects
```

The variables must have these exact names.

---

# Run and test your new attribute dbt models
> Configure dbt connection profiles, run models with full refresh to create attribute tables in Snowflake or BigQuery, and validate data quality.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/run-dbt/

Now that your models are generated, it's time to run them and verify that everything works as expected. This step allows you to test your models locally before moving them to production.

During the run process:

- dbt will compile your SQL models
- Tables will be created in your data warehouse
- You'll see progress updates in the terminal
- Any errors will be clearly displayed

Best practice to ensure successful model runs:

- Always test your models after generation
- Review the generated SQL for accuracy
- Document any custom modifications you make
- Keep track of model versions
- Regularly update your models as your data evolves

## Configure dbt

Before running your new models, you'll need to configure their dbt connection profile. Read more about this in the [dbt documentation](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles). The batch engine doesn't generate a `profiles.yml` because it isn't best practice to store credentials in the same place as models.

## Run the models

After configuring dbt, you can run your models locally using the `dbt run` command.

For your first run, you'll want to do a full refresh to ensure all tables are created properly:

```bash
dbt run --full-refresh
```

![dbt first run](/assets/images/dbt_first_run-22571b35554de4e00d266894a4285914.png)

For later runs, you can use the standard command:

```bash
dbt run
```

Running the models will create tables of your newly calculated attributes.

## Test and validate

It's important to test your models before moving them to production. The local testing phase is your opportunity to:

- Verify that the generated models meet your requirements
- Make any necessary adjustments to the models
- Explore the data transformations
- Ensure data quality and accuracy

Follow your standard dbt testing process.

## Make sure all your data is processed

The first time the model is run, not all your data might get processed. Processing is dictated by the variable `snowplow__backfill_limit_days`. You could increase this to a larger number depending on the data volume and how much data you need to backfill. You might be able to process everything in one run, or after a few runs.

## Run dbt snapshot

Once your data is fully backfilled, you are ready to run `dbt snapshot`. The attributes table captures the latest values for each attribute key in a drop and recompute fashion. Once you start running the dbt snapshot linked to this table, the sync engine will only process changes to specific attribute keys. It will only sync attribute keys where at least one attribute value changed since the last time data was sent to the profile store.

This optimized syncing saves processing cost. Once you start syncing, you'll need to incorporate dbt snapshot runs after each time you run your dbt models. More on this in the next step.

---

# Learn how to set up the Signals batch engine for warehouse attributes
> Set up the Snowplow Signals batch engine to calculate historical behavioral data attributes from warehouse data using dbt. Generate modeled datasets and sync attributes to the Profiles Store for real-time access.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/start/

Welcome to the [Snowplow Signals](/docs/signals/introduction/) batch engine tutorial.

Snowplow Signals is a real-time personalization engine for customer intelligence, built on Snowplow's behavioral data pipeline. It allows you to compute, access, and act on in-session stream and historical user data, in real time.

The Signals batch engine is a CLI tool to create the attributes in your warehouse to compute over larger historical data that otherwise would not be possible / efficient to do so in real time. It isn't required to use Signals: it's only necessary if you want to:

- Calculate attributes from Snowplow events in your warehouse
- Sync those attributes to the Profiles Store so they can be served in real time alongside stream attributes

The batch engine helps by:

- Generating separate dbt projects for each batch attribute group definition
- Building efficient modeled datasets at different aggregation levels, instead of querying directly against large atomic event tables
- Producing attributes tables optimized for downstream use
- Syncing the calculated attributes to Signals, making them available for production use

To use tables of pre-existing, already calculated values, read up on external batch sources in the [Signals documentation](/docs/signals/concepts/).

This guide will walk you through the steps to set up the batch engine and calculate attributes.

## Prerequisites

This tutorial assumes that you have:

- Python 3.11+ installed in your environment

- Snowflake or BigQuery warehouse with your atomic Snowplow events ready to use as the data source

- [dbt](https://www.getdbt.com/) with your warehouse [target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) set up

- Basic [dbt](https://www.getdbt.com/) knowledge

- Valid API credentials for your Signals account:

  - Signals API URL
  - Snowplow API key
  - Snowplow API key ID
  - Snowplow organization ID

- Batch attribute groups already created for Signals, but not yet published

The batch source configuration can't be done before the attributes table has been created.

Check out the [Signals configuration](/docs/signals/introduction/) documentation to find out where to find these credentials, and how to apply attribute configurations.

---

# Sync the attribute tables with the Signals Profiles Store
> Register warehouse attribute tables as batch sources in Signals to enable hourly syncing to the Profiles Store for real-time access in applications.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/sync-tables/

Syncing is the process of making your calculated attributes available in Signals for production use. After the attributes table is ready for production use, you should run `dbt snapshot`. The sync engine will use this table to understand which records have been changed since the last sync. In order for it to work you need to let the sync engine know about the location of your dbt snapshot.

There are two steps to enable syncing:

1. Fill out the `batch_source_config.json` file for each dbt project (one project per attribute group)
2. Run the `sync` command

## Update configuration file

During data model generation, a config file is generated in `config/batch_source_config.json`. It will have a similar structure to this:

```yml
{
    "database": "",       # Add your database name
    "wh_schema": "",      # Add your schema
    "table": "user_attributes_1_attributes_snapshot",
    "name": "user_attributes_1_attributes",
    "timestamp_field": "dbt_valid_from",
    "description": "Table containing attributes for user_attributes_1 attribute group",
    "tags": {},
    "owner": ""
}
```

Fill out the `database` (for BigQuery this should be the project) and `wh_schema` values as per your [dbt target](https://docs.getdbt.com/reference/dbt-jinja-functions/target) setup.

The warehouse schema should be the `schema` defined in your dbt target, suffixed with `_derived` (`{target_schema}_derived`). This is where the generated attributes table and snapshot are located by default.

## Run the sync command

Adjust the following CLI command according your use case (e.g. Change the target type to `bigquery`) and run it:

```bash
snowplow-batch-engine sync \
  --attribute-group-name "user_attributes" \
  --attribute-group-version 1 \
  --target-type snowflake \
  --verbose
```

The batch engine will first register the batch source for the attribute group. It will also publish the attribute group so that syncing can begin.

Signals will check for updates to the table every hour. Changes will be captured in the next sync if both the dbt run and the dbt snapshot update have finished. Your attributes will then become available to retrieve in your applications.

```bash
# Progress messages
✅ Successfully added Batch Source information to attribute group user_attributes_1
✅ Successfully registered table user_attributes_1_attributes
```

---

# Test the Signals batch engine connections
> Verify connectivity to Snowplow Signals API and check database, cache, and storage service health before initializing dbt projects.
> Source: https://docs.snowplow.io/tutorials/signals-batch-engine/test-connection/

The first step is to confirm that you can connect to all the necessary services.

The connection test checks several important components:

- Verifies your API credentials

- Ensures the main Signals API service is accessible

- Checks the status of:

  - Database connections
  - Cache service
  - Storage systems

Test your connection using the following command:

```bash
snowplow-batch-engine test-connection --verbose
```

If you didn't set up environment variables, you can also provide the credentials as command-line flags:

```bash
snowplow-batch-engine test-connection \
  --api-url "YOUR_API_URL" \
  --api-key "YOUR_API_KEY" \
  --api-key-id "YOUR_API_KEY_ID" \
  --org-id "YOUR_ORG_ID" \
  --verbose
```

When everything is working correctly, you'll see a clear success message:

```bash
🔐 Testing authentication service...
✅ Authentication service is healthy

🌐 Testing API service...
✅ API service is healthy
📊 Dependencies status:
   ✅ database: ok
   ✅ cache: ok
   ✅ storage: ok

✨ All services are operational!
```

You can continue to the next step.

## Troubleshooting

If you encounter any problems:

1. Double-check your API credentials
2. Verify your network connection
3. Ensure your API key has the required permissions
4. Use the `--verbose` flag for detailed error messages
5. Check if your organization's services are up and running

---

# Conclusions and next steps for the Signals interventions tutorial
> Complete the Signals Sandbox tutorial and explore next steps for real-time personalization.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/conclusion/

Congratulations! You've successfully completed the Signals Sandbox tutorial and experienced real-time personalization in action.

In this tutorial, you:

- Deployed a Signals Sandbox instance
- Used the Signals Python SDK to programmatically define attributes
- Created a service to expose calculated attributes
- Defined rule-based interventions for common ecommerce scenarios
- Tested your configuration with an interactive demo application
- Saw real-time personalization triggered by actual user behavior

You've worked with the core Signals concepts:

- **[Attributes](/docs/signals/attributes/attributes/)**: real-time calculations of user behavior patterns
- **[Attribute groups](/docs/signals/attributes/attribute-groups/)**: organized collections of related attributes
- **[Services](/docs/signals/attributes/services/)**: interfaces for applications to retrieve attributes
- **[Interventions](/docs/signals/interventions/)**: rules that trigger personalized experiences

Thank you for trying Signals. We hope this tutorial has inspired ideas for how you can use real-time behavioral data to create personalized experiences for your users.

## Next steps

Now that you understand how Signals works, here are some ways to continue your journey:

### Explore more Signals capabilities

- Try defining attributes with different [aggregations](/docs/signals/attributes/attributes/#aggregation-options) (min, max, average, etc.)
- Experiment with [time windows](/docs/signals/attributes/attributes/#time-period) for attributes
- Create more complex intervention criteria using multiple attributes
- Define attributes based on different [event types](/docs/fundamentals/events/)

### Try other tutorials

These tutorials require a [Snowplow account](/docs/get-started/private-managed-cloud/) with Signals configured:

- [Set up Signals for real-time calculation](/tutorials/signals-quickstart/start/) using the Snowplow Console UI
- [Score prospects in real time using Signals and ML](/tutorials/signals-ml-prospect-scoring/intro/) for machine learning integration
- [Set up the Signals batch engine](/tutorials/signals-batch-engine/start/) for warehouse-based attribute calculation

### Move to production

Ready to use Signals in production? You'll need:

- A [Snowplow](/docs/get-started/private-managed-cloud/) account
- A [Signals connection](/docs/signals/connection/) configured
- Integration of the [browser tracker plugin](/docs/signals/interventions/subscribe/#using-the-browser-tracker-plugin) or API calls in your application

### Learn more

- Read the full [Signals documentation](/docs/signals/introduction/)
- Explore the [Signals Python SDK reference](https://pypi.org/project/snowplow-signals/)
- Join the [Snowplow community](https://community.snowplow.io/) to discuss use cases and best practices

## Sandbox limitations

Remember that the Signals Sandbox is designed for exploration and learning:

- Sandbox instances are temporary and will be deleted after some period
- Data is not persisted long-term
- Performance and rate limits apply
- For production use cases, use Snowplow CDI with Signals

---

# Define attributes and service configuration using the Signals Python SDK
> Use the Signals Python SDK to programmatically define real-time ecommerce attributes.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/define-attributes-python/

In this section, you'll define [attributes](/docs/signals/concepts/#attribute-groups) that calculate real-time user behavior metrics from ecommerce events. These attributes will track product views, cart additions, and cart value.

You'll create three attributes to track user shopping behavior:

- **`count_product_views`** counts the number of product view events
- **`count_add_to_cart`** counts the number of add-to-cart events
- **`total_cart_value`** sums the prices of items added to the cart

These attributes use the [Snowplow ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/), which tracks standardized ecommerce events.

## Import required classes

Start by importing the classes you'll need from the [Signals Python SDK](https://pypi.org/project/snowplow-signals/):

```python
from snowplow_signals import (
    Attribute,
    Event,
    Criterion,
    Criteria,
    EventProperty,
    EntityProperty,
    StreamAttributeGroup,
    Service,
    domain_userid,
)
```

## Define the attributes

Each [attribute](/docs/signals/attributes/using-python-sdk/attribute-groups/attributes/) defines which event it will be calculated from, and what kind of aggregation will be performed.

### Product views counter

This attribute counts product view events:

```python
count_product_views = Attribute(
    name="count_product_views",
    type="int32",
    events=[Event(name="snowplow_ecommerce_action")],
    criteria=Criteria(
        all=[
            Criterion.eq(
                property=EventProperty(
                    vendor="com.snowplowanalytics.snowplow.ecommerce",
                    name="snowplow_ecommerce_action",
                    major_version=1,
                    path="type",
                ),
                value="product_view",
            )
        ]
    ),
    aggregation="counter",
)
```

This attribute:

- Uses the `snowplow_ecommerce_action` [event](/docs/fundamentals/events/)
- Filters for events where the action type is `product_view`
- Counts matching events using the `counter` aggregation

### Add to cart counter

This attribute counts when users add items to their cart:

```python
count_add_to_cart = Attribute(
    name="count_add_to_cart",
    type="int32",
    events=[Event(name="snowplow_ecommerce_action")],
    criteria=Criteria(
        all=[
            Criterion.eq(
                property=EventProperty(
                    vendor="com.snowplowanalytics.snowplow.ecommerce",
                    name="snowplow_ecommerce_action",
                    major_version=1,
                    path="type",
                ),
                value="add_to_cart",
            )
        ]
    ),
    aggregation="counter",
)
```

### Total cart value

This attribute sums the prices of products added to the cart:

```python
total_cart_value = Attribute(
    name="total_cart_value",
    type="double",
    events=[Event(name="snowplow_ecommerce_action")],
    criteria=Criteria(
        all=[
            Criterion.eq(
                property=EventProperty(
                    vendor="com.snowplowanalytics.snowplow.ecommerce",
                    name="snowplow_ecommerce_action",
                    major_version=1,
                    path="type",
                ),
                value="add_to_cart",
            )
        ]
    ),
    property=EntityProperty(
        vendor="com.snowplowanalytics.snowplow.ecommerce",
        name="product",
        major_version=1,
        path="price",
    ),
    aggregation="sum",
)
```

This attribute:

- Filters for `add_to_cart` events
- Extracts the `price` field from the `product` [entity](/docs/fundamentals/entities/)
- Sums the prices using the `sum` aggregation

## Create an attribute group

[Attribute groups](/docs/signals/concepts/#attribute-groups) organize related attributes together. They can be considered as "tables" of attributes.

Create a group to hold your ecommerce attributes:

```python
attribute_group = StreamAttributeGroup(
    name="ecom_attributes",
    version=1,
    attribute_key=domain_userid,
    attributes=[
        count_product_views,
        count_add_to_cart,
        total_cart_value,
    ],
    owner="user@company.com",
)
```

The `attribute_key` parameter specifies the [user identifier](/docs/signals/concepts/#attribute-keys) that the attributes are grouped by. In this case, `domain_userid` means the attributes track behavior for each anonymous user.

This is a `StreamAttributeGroup`, because Signals will process events from the real-time event stream.

> **Note:** Attribute groups are immutable and versioned. If you need to modify attributes, create a new version of the group.

## Create a service

[Services](/docs/signals/concepts/#services) provide an interface for applications to retrieve attributes. Create a service that includes your attribute group:

```python
stream_service = Service(
    name="ecom_attributes",
    attribute_groups=[attribute_group],
    owner="user@company.com",
)
```

## Publish to Signals

Now publish both the attribute group and service to your Signals instance:

```python
sp_signals.publish([attribute_group, stream_service])
```

Once published, Signals begins calculating these attributes in real time as events arrive at the Collector.

> **Tip:** You can retrieve attribute values for a specific user by calling:
>
> ```python
> stream_service.get_attributes(
>     signals=sp_signals,
>     attribute_key="domain_userid",
>     identifier="your-domain-userid-here", # the value will be a UUID
> )
> ```

Your attributes are now live and ready to power personalization experiences. Next, you'll define interventions that trigger based on these attribute values.

---

# Define real-time ecommerce intervention triggers
> Create rule-based interventions that trigger personalized experiences based on user behavior.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/define-interventions/

[Interventions](/docs/signals/concepts/#interventions) are rules that trigger when user attributes meet specific conditions. They enable real-time personalization by activating experiences based on current behavior.

When an intervention triggers:

1. Signals evaluates attribute values in real time
2. If the criteria are met, the intervention activates
3. Your Web application can receive intervention notifications via the [browser tracker plugin](/docs/signals/interventions/subscribe/#using-the-browser-tracker-plugin) directly on the frontend
   - This uses a Server-Sent Events (SSE) connection for real-time streaming

In this section, you'll create three interventions for common ecommerce scenarios: cart abandonment, discount offers for engaged browsers, and free shipping promotions.

## Import intervention classes

Add the intervention-related imports to your notebook:

```python
from snowplow_signals import (
    RuleIntervention,
    InterventionCriterion,
    LinkAttributeKey,
)
```

## Define intervention rules

These [rule-based interventions](/docs/signals/interventions/using-python-sdk/) will be triggered automatically when their criteria are met.

Each `InterventionCriterion` specifies:

- `attribute`: the attribute to evaluate, in the format `group_name:attribute_name`
- `operator`: the comparison operator (`>`, `<`, `>=`, `<=`, `==`, `!=`)
- `value`: the threshold value for comparison

### Cart abandonment intervention

This intervention triggers when a user has added items to their cart but hasn't completed checkout:

```python
cart_abandonment_intervention = RuleIntervention(
    name="cart_abandonment",
    description="Show banner to users who have added at least one item to cart.",
    criteria=InterventionCriterion(
        attribute="ecom_attributes:count_add_to_cart",
        operator=">",
        value=0,
    ),
    target_attribute_keys=[
        LinkAttributeKey(name="user_id"),
        LinkAttributeKey(name="domain_userid"),
    ],
    owner="user@company.com",
)
```

Key components:

- **`criteria`**: triggers when `count_add_to_cart > 0`.
- **`target_attribute_keys`**: specifies which user identifiers to use when matching interventions. This intervention targets both the authenticated `user_id` and anonymous `domain_userid` identifiers, to reach as many users as possible.

### Discount intervention

This intervention offers a discount to users who are actively browsing products:

```python
discount_intervention = RuleIntervention(
    name="discount",
    description="Show banner to users who have viewed more than 3 products.",
    criteria=InterventionCriterion(
        attribute="ecom_attributes:count_product_views",
        operator=">",
        value=3,
    ),
    target_attribute_keys=[
        LinkAttributeKey(name="user_id"),
        LinkAttributeKey(name="domain_userid"),
    ],
    owner="user@company.com",
)
```

This intervention activates after a user views more than three products, indicating high purchase intent.

### Free shipping intervention

This intervention encourages larger purchases by offering free shipping:

```python
free_shipping_intervention = RuleIntervention(
    name="free_shipping",
    description="Show banner to users who plan to spend at least $100.",
    criteria=InterventionCriterion(
        attribute="ecom_attributes:total_cart_value",
        operator=">",
        value=100,
    ),
    target_attribute_keys=[
        LinkAttributeKey(name="user_id"),
        LinkAttributeKey(name="domain_userid"),
    ],
    owner="user@company.com",
)
```

This triggers when the cart value exceeds $100, incentivizing users to complete their purchase.

## Publish the interventions

Publish all three interventions to your Signals Sandbox:

```python
sp_signals.publish(
    [
        cart_abandonment_intervention,
        discount_intervention,
        free_shipping_intervention,
    ]
)
```

Once published, these interventions are active and will trigger in real time as users interact with your application.

In the next section, you'll use the demo e-shop application to see these interventions in action. The app uses the [browser tracker plugin](/docs/signals/interventions/subscribe/#using-the-browser-tracker-plugin) to automatically receive and display intervention banners when they trigger.

---

# Connect to Signals within a notebook using the Signals Python SDK
> Enable Signals through Snowplow Console or deploy a Signals Sandbox instance and obtain your access credentials.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/setup/

The first step is to set up your Signals connection. Follow the instructions in the [Signals connection documentation](/docs/signals/connection/) for your chosen deployment method:

- **Snowplow Console**: [enable Signals through Snowplow Console](/docs/signals/connection/#snowplow-console) if you have a Snowplow account
- **Signals Sandbox**: [deploy a Sandbox instance](/docs/signals/connection/#signals-sandbox) to experiment without a Snowplow account

Once your connection is set up, gather the required credentials as described in the [connection credentials section](/docs/signals/connection/#connection-credentials).

## Set up your Jupyter notebook environment

You can use either Google Colab or a local Jupyter notebook environment for this tutorial.

If you want to skip ahead, you can make use of the following notebook that contains all the Python code you'll need for this tutorial:

1. [Python notebook on Google Colab](https://colab.research.google.com/github/snowplow-incubator/signals-interventions-demo/blob/main/attributes_and_interventions.ipynb)
2. [Jupyter notebook on GitHub](https://github.com/snowplow-incubator/signals-interventions-demo/blob/main/attributes_and_interventions.ipynb)

### Using Google Colab

You can use the provided notebook, or create your own. To create a new notebook, open a new notebook at [Google Colab](https://colab.research.google.com/).

You'll need to add credentials as Colab secrets. Click the key icon in the left sidebar, and add the required secrets:

**Snowplow Console:**

- `SP_API_URL`: your Signals API URL
- `SP_API_KEY`: your API key
- `SP_API_KEY_ID`: your API key ID
- `SP_ORG_ID`: your Snowplow Console organization ID

**Signals Sandbox:**

- `SP_API_URL`: your Profiles API URL
- `SP_SANDBOX_TOKEN`: your Sandbox Token

***

When you run the notebook, it will ask for access to the secrets. Choose to grant access.

If you're using your own notebook, follow these steps:

1. Install the Signals Python SDK:

```python
%pip install snowplow-signals
```

2. Load your credentials in the notebook:

**Snowplow Console:**

```python
from google.colab import userdata
import os

os.environ["SP_API_URL"] = userdata.get('SP_API_URL')
os.environ["SP_API_KEY"] = userdata.get('SP_API_KEY')
os.environ["SP_API_KEY_ID"] = userdata.get('SP_API_KEY_ID')
os.environ["SP_ORG_ID"] = userdata.get('SP_ORG_ID')
```

**Signals Sandbox:**

```python
from google.colab import userdata
import os

os.environ["SP_API_URL"] = userdata.get('SP_API_URL')
os.environ["SP_SANDBOX_TOKEN"] = userdata.get('SP_SANDBOX_TOKEN')
```

***

### Using local Jupyter

Navigate into your working directory and environment, then follow these steps:

1. Install Jupyter and the Signals SDK:

```bash
pip install jupyter snowplow-signals python-dotenv
```

2. Create a `.env` file in your working directory with your credentials:

**Snowplow Console:**

```text
SP_API_URL=your_signals_api_url
SP_API_KEY=your_api_key
SP_API_KEY_ID=your_api_key_id
SP_ORG_ID=your_organization_id
```

**Signals Sandbox:**

```text
SP_API_URL=your_profiles_api_url
SP_SANDBOX_TOKEN=your_sandbox_token
```

***

3. Start Jupyter notebook:

```bash
jupyter notebook
```

4. In your notebook, load the environment variables:

```python
from dotenv import load_dotenv
load_dotenv()
```

## Connect to Signals

Now you're ready to connect to your Signals instance using the Python SDK.

**Snowplow Console:**

```python
from snowplow_signals import Signals
import os

sp_signals = Signals(
    api_url=os.environ["SP_API_URL"],
    api_key=os.environ["SP_API_KEY"],
    api_key_id=os.environ["SP_API_KEY_ID"],
    org_id=os.environ["SP_ORG_ID"],
)
```

**Signals Sandbox:**

```python
from snowplow_signals import SignalsSandbox
import os

sp_signals = SignalsSandbox(
    api_url=os.environ["SP_API_URL"],
    sandbox_token=os.environ["SP_SANDBOX_TOKEN"],
)
```

> **Tip:** The `SignalsSandbox` class is specifically designed for Sandbox environments. For production Snowplow deployments, you would use the `Signals` class instead, which requires API keys.

***

You're now ready to start defining attributes and interventions in Signals.

---

# Learn how to implement real-time interventions in an ecommerce app using Signals
> Get hands-on with Snowplow Signals, using Snowplow Console or the Signals Sandbox trial environment, to create real-time personalization.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/start/

Welcome to the **Snowplow Signals** tutorial.

[Snowplow Signals](/docs/signals/introduction/) is a real-time personalization engine for customer intelligence, built on Snowplow's behavioral data pipeline. It allows you to compute, access, and act on in-session stream and historical user data, in real time.

This tutorial provides a hands-on introduction to Signals. You'll use Python to programmatically define attributes, services, and interventions, then test them with a demo ecommerce application.

You can follow this tutorial using either:

- **Snowplow Console**: if you have a Snowplow account with Signals enabled
- **Signals Sandbox**: a trial environment where you can experiment without needing a Snowplow account

This tutorial should take approximately 20-30 minutes to complete.

## What you'll learn

In this tutorial, you will:

- Set up your Signals connection (via Console or Sandbox)
- Use the Signals Python SDK to define attributes, services, and interventions
- Calculate real-time user behavior metrics from ecommerce events
- Create intervention rules that trigger personalized experiences
- Test your Signals configuration with an interactive demo application

## Deployment options

### Snowplow Console

If you have a Snowplow account, you can [enable Signals through Snowplow Console](/docs/signals/connection/#snowplow-console). This provides:

- Integration with your existing Snowplow data pipeline
- Access to your production behavioral data
- Full production capabilities and support

### Signals Sandbox

[Signals Sandbox](https://try-signals.snowplow.io/) provides a temporary Signals deployment that you can use to explore the platform without needing a Snowplow account. The Sandbox includes:

- A dedicated Profiles API endpoint for your attributes and interventions
- A Snowplow Collector endpoint for tracking events
- Access credentials (Sandbox Token) for authentication
- Limited-time access to experiment with Signals capabilities

> **Note:** The Signals Sandbox is designed for experimentation and learning. For production use cases, you'll need a [Snowplow](/docs/get-started/private-managed-cloud/) account with [Signals enabled](/docs/signals/connection/).

## Prerequisites

This tutorial requires:

- Python 3.7 or higher
- A Jupyter notebook environment (local or Google Colab)
- Basic familiarity with Python
- A modern web browser for testing the demo application
- Either a Snowplow account with Signals enabled, or access to Signals Sandbox

---

# Test your Signals interventions in the demo application
> Use the interactive demo e-shop to see your Signals configuration in action.
> Source: https://docs.snowplow.io/tutorials/signals-interventions/test-with-demo-app/

Now that you've defined attributes and interventions, it's time to see them work in a real application. We've prepared a demo React application with Snowplow tracking that can integrate with your Signals instance to demonstrate real-time personalization.

The demo application features:

- **Product catalog**: products fetched from an external API
- **Snowplow tracking**: the [JavaScript tracker](/docs/sources/web-trackers/) with the [ecommerce plugin](/docs/sources/web-trackers/tracking-events/ecommerce/)
- **Intervention display**: the [browser tracker plugin](/docs/signals/interventions/subscribe/#using-the-browser-tracker-plugin) to receive and show intervention banners
- **Logging**: the app prints to the console when it tracks an event, or receives an intervention
- **Reset functionality**: ability to clear your user data and start fresh

## Access the demo application

The demo application is deployed at:

<https://snowplow-incubator.github.io/signals-interventions-demo/>

When you first open the app, you'll see a configuration screen.

![Screenshot showing the configuration page of the demo app.](/assets/images/demo-configure-0759b8ea89e72ae691dbe3b1dcbb98ba.png)

Enter your credentials from the setup step.

**Snowplow Console:**

- **Collector URL**: your Snowplow Collector endpoint for the pipeline you enabled Signals on (see the "Pipelines" section in Snowplow Console)
- **Profiles API URL**: your Signals API URL (from Snowplow Console > **Signals** > **Overview**)

**Signals Sandbox:**

- **Collector URL**: your Snowplow Collector endpoint provided by Signals Sandbox
- **Profiles API URL**: your Signals Profiles API endpoint provided by Signals Sandbox

***

The app will store these values in your browser's local storage, so you won't need to enter them again during this session.

Click **Start Shopping** to proceed to the e-shop.

## Test your interventions

Follow these steps to trigger each intervention.

### Trigger the discount intervention

1. Browse the product catalog

2. Click on different products to view their details

3. After viewing four products, you should see a banner appear at the top:

   - **Message**: "10% off your next purchase!"
   - **Code**: `SAVE10`

### Trigger the cart abandonment intervention

1. Click **Add to Cart** on any product
2. After adding at least one item, you should see:
   - **Message**: "Don't forget your items in cart!"

### Trigger the free shipping intervention

1. Add multiple items to your cart until the total value exceeds $100

2. You should see:

   - **Message**: "Free shipping on orders over $100!"
   - **Code**: `FREE`

![Screenshot showing a triggered intervention in the demo app.](/assets/images/demo-intervention-e9c05f07f0877ce583bd66b4085131c1.png)

## Inspecting events, attributes and interventions using Snowplow Inspector

We highly recommend installing the [Snowplow Inspector](/docs/testing/snowplow-inspector/) Chrome extension that lets you validate which Snowplow events are triggering on the website and also check the values of Signals attributes and interventions.

You can install the extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm). Once you add the extension to Chrome, you can view it by [opening Developer Tools](https://developer.chrome.com/docs/devtools/open/) (usually `Ctrl`+`Shift`+`I` or on Mac `Cmd`+`Option`+`I`), where it has its own tab named 'Snowplow'. Right away, you will see the tracked events as you open a product or add a product to cart.

![Screenshot showing the Snowplow Inspector extension with tracked events from the demo app.](/assets/images/inspector-events-cc8f7737b748821cc26528aa7c0f0be0.png)

**Snowplow Console:**

To configure the extension to show Signals attributes and interventions:

1. In Chrome, navigate to [the Extensions settings page](chrome://extensions/)
2. Click on "Details" for the Snowplow Inspector extension
3. Scroll down and click on "Extension options"
4. Click on "Add new organization" and enter your Organization ID and API key details.

**Signals Sandbox:**

To configure the extension to show Signals attributes and interventions:

1. In Chrome, navigate to [the Extensions settings page](chrome://extensions/).
2. Click on "Details" for the Snowplow Inspector extension.
3. Scroll down and click on "Extension options".
4. Open "Advanced options" and enter your Signals Sandbox URL and token.

![Screenshot showing the Snowplow Inspector extension options page with fields to enter Signals API URL and Token.](/assets/images/inspector-advanced-options-372d76b109ed9b6eafc501325134ee88.png)

***

Now, in addition to seeing the Snowplow events tracked, you will be able to see the attributes and interventions calculated by Signals after switching the section on the left side of the extension.

![Screenshot showing the Snowplow Inspector extension with Signals attributes displayed.](/assets/images/inspector-attributes-067b8df903f7e2a74c7c8deb6dc37b3a.png)

![Screenshot showing the Snowplow Inspector extension with Signals interventions displayed.](/assets/images/inspector-interventions-6c70a032c7a0aec14f6f79c4521336bf.png)

## Behind the scenes

Here's what happens when you interact with the demo app:

1. **Event tracking**: when you view a product or add to cart, the JavaScript tracker sends an ecommerce event to your Sandbox Collector
2. **Attribute calculation**: Signals processes the event and updates your user attributes in real time
3. **Intervention evaluation**: Signals checks if any intervention criteria are now met
4. **Delivery**: if an intervention triggers, Signals sends it to the browser via Server-Sent Events (SSE)
5. **Display**: the browser tracker plugin receives the intervention and your application displays the appropriate banner

## Reset and experiment

You can click **Reset User Data** to clear your current user session and start testing again with a fresh profile. This generates a new `domain_userid` so you can re-trigger interventions.

The demo app is open source. You can explore the implementation in the [GitHub repository](https://github.com/snowplow-incubator/signals-interventions-demo):

- **Tracker setup**: see how the Snowplow tracker and Signals plugin are initialized in `app/src/snowplow.ts`
- **Intervention handling**: see how interventions are received and displayed in `app/src/App.tsx`
- **Attribute definitions**: review the Python notebook `attributes_and_interventions.ipynb` that you followed in this tutorial

You've successfully configured Signals to calculate real-time attributes and trigger personalized interventions based on user behavior.

---

# Call the API to see prospect scores in your browser
> Deploy the prospect scoring API, and display real-time conversion predictions in the browser console.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/access-api-browser/

The final requirement is to see the prospect scores and predictions in the browser. In this tutorial, we'll call the API every 10 seconds.

You need an API endpoint that you can access from your local machine, or from JavaScript in the browser.

In the previous step we used TryCloudflare tunnels to expose Colab notebook behind a public HTTPS endpoint. In the server output you will find your public URL similar to this one:

```text
...
 * Running on https://aaa-bbb-ccc-ddd.trycloudflare.com
```

## Test with cURL

Test the endpoint using `cURL`, passing in your `domain_userid` that you got earlier using the Snowplow Inspector.

Update the URL with your TryCloudflare tunnel URL.

```bash
curl -X POST \
  "https://aaa-bbb-ccc-ddd.trycloudflare.com/predict" \
  -H "Content-Type: application/json" \
  -d '{"domain_userid": "8e554b10-4fcf-49e9-a0d8-48b6b6458df3"}'
```

You should see an output like this:

```json
{
  "score": 0.35080923483101656,
  "scoring_attributes": {
    "num_customers_views": 0,
    "num_page_views": 0,
    "num_pricing_views": 0
  },
  "signals": {
    "domain_userid": "00000000-1111-2222-3333-444455556666",
    "num_customers_views": null,
    "num_page_views": null,
    "num_pricing_views": null
  }
}
```

## See scores in the browser

Finally, run the code below in your browser console on your website to see your live prospect score. The code retrieves your `domain_userid` directly from the tracker, and calls the intermediary API to get the scores.

You may need to update this if your tracker name is different. Check the tracker name in outbound events using the Snowplow Inspector.

Run this in your browser console to see your predictions:

```js
let api_url = "https://aaa-bbb-ccc-ddd.trycloudflare.com/predict"; // UPDATE THIS
let tracker_name = "sp"; // MAYBE UPDATE THIS

// Calls the API every 10s from the front-end
setInterval(function () {
    // assuming the Snowplow tracker is available at 'window.snowplow(...)'
    window.snowplow(function () {
        // Gets domain_userid from the tracker instance
        var sp = this[tracker_name];
        var domainUserId = sp.getDomainUserId();

        // Calls the API
        fetch(api_url, {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({ domain_userid: domainUserId })
        })
        .then(response => response.json())
        .then(result => {
            console.log("Prediction: ", domainUserId, " - ", result.score);

            // Acts on prediction
            if (result.score >= 0.9) console.log('Prospect is likely to convert!');
        })
        .catch(console.error);
    });
}, 10 * 1000);
```

Every 10 seconds it will print out the prediction score that you'll convert, as well as the full API response.

Adjust the timing interval to call the endpoint APIs as often as needed in your use case.

![](/assets/images/console_output-fea3b67475be2c3cdbc93d115d89dcb5.png)

In a real use case, you'd be able to take actions based on these scores and predictions.

---

# Conclusions and next steps from the Signals ML prospect scoring accelerator
> Complete the Snowplow Signals machine learning prospect scoring tutorial, and explore advanced real-time personalization use cases.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/conclusion/

In this accelerator you've learned how to build a prospect scoring system using Snowplow Signals together with a machine learning model, your own website, and your own Snowplow data.

What you achieved:

- Used Signals to calculate attributes
- Scored prospects using an ML model
- Served live prospect scores in the browser

This accelerator is a starting point for exploring and using Signals APIs for your own needs and use cases.

![](/assets/images/console_output-fea3b67475be2c3cdbc93d115d89dcb5.png)

## Next steps

Here are some ideas for further exploration:

- Define more Signals attributes that are specific for your use cases and tracking
- Act on the prospect scores and predictions in your application
- Use attributes to trigger actions and decisions without the ML step

---

# Create an API endpoint to serve the prospect scores in real time
> Build an API endpoint to serve Snowplow Signals attributes and ML predictions for real-time prospect scoring.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/create-endpoint/

You now have attribute values in your Profiles Store, and a trained ML model. The next step is to create and deploy an endpoint to serve attributes and predictions.

It needs to do the following:

- Get calculated attribute values from Signals
- Score the values using the model
- Return the results

This tutorial uses Flask together with TryCloudflare tunnels to expose the API endpoint outside of the Colab notebook environment.

## Install dependencies

Run `%pip install flask-cloudflared` to install the required libraries.

## Create ML scoring endpoint

Start by loading your model, and defining the methods needed to retrieve and process your Signals data.

```python
from flask import Flask, request
from flask_cloudflared import run_with_cloudflared

# Define and launch Colab API Proxy
app = Flask(__name__)
run_with_cloudflared(app) # Open Cloudflared demo tunnel

# Load model
model = joblib.load("model.joblib")

# Process the Signals data for individual domain_userids
def get_duid_values(duid: str):
    # Retrieve attributes from Signals
    response = sp_signals.get_service_attributes(
        name="prospect_scoring_tutorial_service",
        attribute_key="domain_userid",
        identifier=duid,
    )
    signals_df = pd.DataFrame([response])
    # Prepare ML dataframe
    ml_df = signals_df.fillna(0).reindex(columns=x_columns, fill_value=0)
    return (signals_df, ml_df)

# Score the data against the model
def get_predictions(df):
    return float(model.predict_proba(df)[:, 1][0])

@app.route("/predict", methods=['POST'])
def predict():
    input_dict = request.get_json() # Parse JSON input

    # Get Signals data and prepare dataframe for scoring
    signals_df, ml_df = get_duid_values(input_dict['domain_userid'])

    # Score dataframe using the trained model
    prediction = get_predictions(ml_df)

    # Return the result
    print(f"P: {round(prediction, 4)} - {input_dict}")
    return {
        "signals": signals_df.to_dict(orient='records')[0],
        "scoring_attributes": ml_df.to_dict(orient='records')[0],
        "score": prediction
    }

app.run()
```

This `/predict` endpoint does four things:

1. Receives a `domain_userid`
2. Calls the Signals API to get the current attribute values, using the `prospect_scoring_tutorial_service` service
3. Scores the attribute values using the ML model
4. Returns Signals attributes, and ML prediction score

---

# Define page view attributes for real-time prospect scoring
> Define behavioral data attributes in Snowplow Signals for prospect scoring, including page views, sessions, and conversion events.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/define-prospect-attributes/

To use Signals, you need to define which attributes to calculate, and then apply the configuration. Signals will calculate the attributes from your real-time event stream.

Let's imagine that for this use case our data science team came up with the following set of attributes. They'll be calculated against the `domain_userid` device attribute key. Choosing this attribute key allows Signals to calculate attributes from events across multiple sessions for each prospect.

| Feature name          | Description                                             | Type | Aggregation |
| --------------------- | ------------------------------------------------------- | ---- | ----------- |
| `num_page_views`      | Number of `page_view` events                            | int  | `counter`   |
| `num_pricing_views`   | Number of `page_view` events of the `/pricing` page     | int  | `counter`   |
| `num_customers_views` | Number of `page_view` events of the `*customers*` pages | int  | `counter`   |

## Install Signals Python SDK

Run `%pip install snowplow-signals`.

## Define attributes

Next, define the attributes to calculate.

```python
# Imports
from snowplow_signals import Attribute, Criteria, Criterion, AtomicProperty, EntityProperty

# Define an event
sp_page_view = Event(
    vendor="com.snowplowanalytics.snowplow",
    name="page_view",
    version="1-0-0"
)

# Define attributes
num_page_views = Attribute(
    name="num_page_views",
    type="int32",
    events=[sp_page_view],
    aggregation="counter"
)

num_pricing_views = Attribute(
    name="num_pricing_views",
    type="int32",
    events=[sp_page_view],
    aggregation="counter",
    criteria=Criteria(
        all=[
            Criterion.like(
                AtomicProperty(name="page_url"),
                "%pricing%"
            )
        ]
    )
)

num_customers_views = Attribute(
    name="num_customers_views",
    type="int32",
    events=[sp_page_view],
    aggregation="counter",
    criteria=Criteria(
        all=[
            Criterion.like(
                AtomicProperty(name="page_url"),
                "%customers%"
            )
        ]
    )
)
```

Group the attributes into an attribute group with the `domain_userid` device attribute key.

```python
from snowplow_signals import StreamAttributeGroup, domain_userid

user_attributes_group = StreamAttributeGroup(
    name='prospect_scoring_tutorial',
    owner='your_email@example.com',
    version=1,
    attribute_key=domain_userid,
    attributes=[
        num_page_views,
        num_pricing_views,
        num_customers_views,
    ],
)
```

Next, define a service for retrieving the calculated attributes. Again, provide your own email address for the `owner` field.

```python
from snowplow_signals import Service

prospect_scoring_tutorial_service = Service(
    name='prospect_scoring_tutorial_service',
    owner='your_email@example.com',
    attribute_groups=[user_attributes_group],
)
```

**Details**

For Signals Customers: test attribute group on your warehouse data

If you're using Signals in the Console, you can test the attribute outputs on a subset of recent event data. The `test` command uses the last hour of data from your atomic events table. Here we're restricting the results to events with the application ID `website`. This filtering is optional.

```python
sp_signals_test = sp_signals.test(
    attribute_group=user_attribute_group,
    app_ids=["website"]
)
sp_signals_test
```

The result should look similar to this:

![](/assets/images/signals_test_output-0fd3caaf6492e3f5ea9ef9ed0b940c74.png)

## Deploy configuration to Signals

Apply the attribute group and service configurations to Signals.

```python
applied = sp_signals.publish([user_attribute_group, prospect_scoring_tutorial_service])

# This should print "2 objects applied"
print(f"{len(applied)} objects applied")
```

Signals will start populating your Profiles Store with attributes calculated from your real-time event stream.

## Look at your attributes

> **Note:** This section expects that you [integrated](/docs/sources/web-trackers/quick-start-guide/) `sp.js` into a website and have events flowing to the collector. If you're using Signals Sandbox your collector URL for `sp.js` is next to the Signals token.

Go to your website, and use the [Snowplow Inspector](/docs/testing/snowplow-inspector/) browser plugin to find your own `domain_userid` in outbound web events.

![](/assets/images/get_domain_userid-8cbe50832395dfe0e5e83cbdb6581064.png)

Use your `domain_userid` to retrieve the attributes that Signals has calculated just now from your real-time event stream.

```python
sp_signals_result = sp_signals.get_service_attributes(
    name="prospect_scoring_tutorial_service",
    attribute_key="domain_userid",
    identifier="00000000-1111-2222-3333-444455556666", # UPDATE THIS
)
sp_signals_result
```

The result should look something like this:

```yaml
{
    'domain_userid': '00000000-1111-2222-3333-444455556666',
    'num_page_views': None,
    'num_pricing_views': None,
    'num_customers_views': None
}
```

---

# Learn how to score prospects in real time using Signals and machine learning
> Build a real-time prospect scoring system using Snowplow Signals and machine learning to predict conversion likelihood.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/intro/

Welcome to the [Snowplow Signals](/docs/signals/introduction/) real-time prospect scoring tutorial.

Snowplow Signals is a real-time personalization engine for customer intelligence, built on Snowplow's behavioral data pipeline. It allows you to compute, access, and act on in-session stream and historical user data, in real time.

This guide will through the process of building a real-time prospect scoring system using Signals together with a machine learning model. You'll learn how to leverage Snowplow event data to predict a prospect's likelihood to convert on your website, and how to trigger personalized engagements.

Use this tutorial as a starting point for how to integrate Signals data with any ML use cases or other back-end services.

The tutorial uses the Snowplow marketing website as an example. Follow along using your own website and Snowplow data.

At the end of the tutorial, you will:

- See live prospect attribute updates in the browser console
- Score prospects using an ML model
- Be ready to use the outputs to drive decisions on your website

## Business case overview

Prospects on the Snowplow marketing website visit product and pricing pages, watch videos, etc. The key call to action is a form submission to request a demo.

We want to predict if a specific prospect is likely to submit a form in the next hour, given their behavior before the prediction moment.

![](/assets/images/prediction-structure-6110e68ef2574e58b5b22a2ad73c2390.png)

For simplicity in this tutorial, we'll score prospects every 10 seconds as they browse the website. Depending on your requirements, you could score prospects at the start of each `page_view`, after certain events, or at some other time interval.

We're calculating aggregated attributes based off real-time stream event data, so won't have much user history to work with initially. Over time, as prospects visit and revisit the website, the attributes will become more meaningful.

## Prerequisites

1. Please follow [Connect to Signals](/docs/signals/connection/) page to setup the Signals connection (Console or Sandbox)

   1. Signals Sandbox: follow the Sandbox setup flow
   2. Signals Customers: follow the Console setup flow

2. Open this [Google Colab](https://colab.research.google.com/github/snowplow-incubator/signals-notebooks/blob/main/web/web_prospect_scoring_end_to_end.ipynb) notebook to follow along

3. \[Optional] Integrate Snowplow into your website. If you use Signals Sandbox - find your collector URL in the UI

## Architecture

The system consists of three main blocks:

1. **Your marketing website**: as users browse the website, Snowplow events are sent to the Snowplow Collector
2. **Snowplow Infrastructure**: the Collector captures the events, and Signals calculates aggregated user attributes in real time e.g., `num_page_views`, `num_customers_views`
3. **Intermediary `/predict` API**: an API that calls the Signals API to get the latest attributes, runs an ML model on the Signals output, and sends the response back

![](/assets/images/solution_overview-268d0581d22caed9db29407d4260f603.png)

> **Note:** You can replace the ML model with any other back-end system that you'd use to act on live prospects' attributes.

---

# Add credentials to the Signals prospect scoring ML notebook
> Configure Google Colab notebook with Snowplow Signals credentials for the prospect scoring machine learning tutorial.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/setup/

This tutorial uses this Google [Colab notebook](https://colab.research.google.com/github/snowplow-incubator/signals-notebooks/blob/main/web/web_prospect_scoring_end_to_end.ipynb).

For ML training data we will use an artificial set of Snowplow events. Feel free to replace with your warehouse connection to connect to your own dataset.

Start by configuring your Signals credentials in the notebook.

![](/assets/images/colab_credentials-75693a10119a3fbb3b31d8567f3c1a7c.jpeg)

Here's a list of all the credentials you may need to configure, saved as variables:

```python
from google.colab import userdata

# Snowplow Signals Sandbox credentils
SP_SANDBOX_URL = userdata.get('SP_SANDBOX_URL')     # https://{{123abc}}.svc.snplow.net
SP_SANDBOX_TOKEN = userdata.get('SP_SANDBOX_TOKEN') # 12345678-0000-1111-2222-123456789012

# Snowplow Signals Console credentials
SP_API_URL = userdata.get('SP_API_URL')       # Signals API URL
SP_API_KEY = userdata.get('SP_API_KEY')       # Signals API key
SP_API_KEY_ID = userdata.get('SP_API_KEY_ID') # Signals API key ID
SP_ORG_ID = userdata.get('SP_ORG_ID')         # Snowplow org ID
```

Once you've added the secrets, start working through the tutorial. If you prefer to run the cells in one go with Run all, update your details in the required places first - they're marked with `UPDATE THIS`.

---

# Prepare a training dataset and train the ML prospect scoring model
> Create and train a machine learning model using historical Snowplow behavioral data to predict prospect conversion.
> Source: https://docs.snowplow.io/tutorials/signals-ml-prospect-scoring/train-ml-prospect-scoring/

As prospects browse your website, Signals will calculate the aggregated attributes in real time.

We want to score the combination of these attributes using an ML model to better understand if a specific prospect is likely to submit a form.

Here's the prediction structure and timeline:

![](/assets/images/prediction-structure-6110e68ef2574e58b5b22a2ad73c2390.png)

The next task is to prepare historical data resembling the same Signals features, and train a LogisticRegression model on top.

## Prepare training dataset

For this tutorial we will use an artificial set of Snowplow events in a CSV that resemble web traffic.

We will connect to it using duckdb and model data using SQL. You can replace this section with your warehouse data, just update the connection block. SQL syntax should translate to major warehouses easily.

Here's the SQL:

```sql
with
-- Change to your events table
events as (
    select * from read_csv_auto('{csv_filename}')
    where app_id = 'website' -- filter for your app_ids
        and domain_userid is not null -- filter to traffic with domain_userids
        and event_name in ('page_view', 'submit_form') -- filter the events you need
        -- and derived_tstamp >= dateadd('day', -90, current_date) -- filter by time
        -- filter out bots, new/returning users, etc
),
-- Filter out post-conversion events
post_conv as (
    select
        domain_userid,
        min(derived_tstamp) as first_conv
    from events
    where event_name = 'submit_form'
    group by 1
),
-- Prepare the target column
targets_as_of_event as (
    -- Target: will this person 'submit_form' in the next 1 hour?
    select
        -- identifiers
        er.event_id,
        -- target
        count_if(ef.event_name = 'submit_form') > 0 as target_had_submit_form_next1h,
    from events er
    left join events ef on er.domain_userid = ef.domain_userid
        and er.derived_tstamp < ef.derived_tstamp -- only future events
        and datediff('second', er.derived_tstamp, ef.derived_tstamp) <= 60 * 60 -- only the next 1h of events
    group by er.event_id
),
-- Prepare training features and dataset
final_training as (
    select
            er.domain_userid,
        er.derived_tstamp,
        er.event_name,
        coalesce(count_if(eh.event_name = 'page_view'), 0) as num_page_views,
        coalesce(count_if(eh.event_name = 'page_view' and eh.page_url like '%pricing%'), 0) as num_pricing_views,
        coalesce(count_if(eh.event_name = 'page_view' and eh.page_url like '%customers%'), 0) as num_customers_views,
        coalesce(t.target_had_submit_form_next1h, false) as target_had_submit_form_next1h,
    from events er
    left join events eh on er.domain_userid = eh.domain_userid
        and eh.derived_tstamp < er.derived_tstamp
    left join targets_as_of_event t on er.event_id = t.event_id
    left join post_conv p on er.domain_userid = p.domain_userid
        and er.derived_tstamp >= p.first_conv
    group by all
    having not p.domain_userid is not null -- remove post-conversion events
)
select * from final_training
```

Next we will connect to the CSV to run the query and retrieve data into a pandas DataFrame:

```python
csv_filename = 'https://github.com/snowplow-incubator/signals-notebooks/raw/refs/heads/main/web/sample_events.csv.gz'
query = f"""..."""

import duckdb
import pandas as pd

conn = duckdb.connect(database=':memory:')
db_df = conn.sql(query).df()
conn.close()

db_df
```

> **Info:** Adjust the query/connection here if you want to try the query on your own warehouse. duckdb SQL syntax translates easily to all the major warehouses.

### Best practices for atomic events training datasets

1. Check that key behaviors you want to use for predictions are captured in your events, and exclude events not needed for your use case like `application_error`.

2. Choose a subset of `domain_userid`s (or your other entities, depending on what you're using) to include in training. For example, sometimes you may want to exclude bots, new/returning users, specific segments, etc.

3. Choose historical training time periods that allow your `domain_userid`s (or other entities) to reach the target.

   - Let's say you expect your `domain_userid` to convert in 7 days. Then choose `domain_userid`s who first appeared between 90 and 30 days ago, and set a cutoff of 7 days for them to convert. Don't just choose the last 30 days, as a `domain_userid` who appeared yesterday wouldn't have enough time to reach the cutoff date.
   - If you're trying to predict churn, give your `domain_userid`s enough time to become "churned". For example, you could make an assumption that anyone who doesn't have events for 30 days is churned. So you'd need to train only on data that happened up until `today - 30d`.

4. Make sure there's no target leakage in your dataset. The most common pitfalls are:

   - Including events in historical training that happened after the prediction point in the journey.
   - Including information that wouldn't be available at the time of prediction in training, e.g., accidentally enriching the training dataset with some "as of today" data from the warehouse or CRM.

## Train and evaluate the model

In this tutorial, we're training a regular LogisticRegression model on top of numerical features.

> **Note:** Use this template to adjust to your own ML needs. Training an ML model for your specific use cases goes far beyond this tutorial.

In the next code block we prepare data for training, fit the model, and persist the model binary to use it later in the API.

```python
import joblib
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# preprocessing
x_columns = [
    'num_page_views',
    'num_pricing_views',
    'num_customers_views',
]
y_column = 'target_had_submit_form_next1h'

seed = 42
model = LogisticRegression(random_state=seed, class_weight='balanced')

# Split, Train, and Evaluate
X = db_df[x_columns]
y = db_df[y_column]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)

model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

# Export
joblib.dump(model, "model.joblib")
```

Follow along in the notebook to evaluate your model's performance.

![](/assets/images/model_evaluate-9b22ced5cfe609c9feeb05ac047c0eb2.png)

> **Note:** This model is built on an artificial subset of events. In real-life use-cases you'll most likely use a lot more Signals attributes that capture key user's behaviours including geo, useragent, traffic sources, various categorical attributes, and more.

---

# Conclusions and next steps from the Signals personalized travel site accelerator
> Summary of what you've accomplished and recommended next steps for expanding your Snowplow Signals implementation.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/conclusion/

You've successfully built a personalization system using Snowplow Signals that demonstrates the power of real-time behavioral data. Your implementation captures user interactions, processes them into meaningful attributes, and applies these insights to create personalized experiences.

You've learned how to transform raw behavioral events into actionable personalization signals. You've seen how micro-segmentation based on user behavior can drive content customization, from changing images and descriptions to reordering information based on user preferences. The integration with an AI chatbot shows how behavioral data can enhance conversational experiences, making them more contextual and relevant.

The intervention system you implemented showcases proactive personalization, where the system anticipates user needs and responds accordingly. This represents a shift from reactive to predictive user experiences, where the application adapts to user behavior patterns rather than waiting for explicit requests.

Your personalization system now:

- Processes behavioral data in real-time
- Segments users dynamically based on their interactions
- Customizes content presentation based on demonstrated preferences
- Provides personalized chatbot responses using behavioral context
- Triggers proactive interventions based on user engagement patterns

## Next steps

You can expand your personalization capabilities by exploring these opportunities:

- **Advanced attribute definitions**: create more sophisticated attributes that combine multiple behavioral signals or include temporal elements
- **Multi-session personalization**: extend attributes to persist across sessions, building long-term user preference profiles
- **A/B testing integration**: use Signals interventions to deliver different personalization strategies to different user segments
- **Cross-channel personalization**: apply the same behavioral insights to personalize email campaigns, push notifications, or other customer touchpoints
- **Enhanced chatbot capabilities**: integrate more complex conversational flows that adapt based on user segment
- **Performance optimization**: implement caching strategies for frequently accessed attributes and optimize attribute calculations for high-traffic scenarios

---

# Define travel preference attributes using the Signals Python SDK
> Define behavioral attributes using the Snowplow Signals Python SDK to capture user preferences from website interactions.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/defining-attributes/

You'll now define behavioral attributes that capture different types of travel preferences based on how users interact with the website. These attributes will serve as the foundation for both on-site content personalization and AI chatbot customization.

The attributes you create will track user engagement with different types of content, destinations, and features. For example, users who frequently view food-related content will have their culinary interest attribute incremented, while those who browse luxury destinations will see their luxury preference attribute increase.

## Open the Jupyter notebook

You'll use the Snowplow Signals Python SDK in a Jupyter notebook to define your attributes. You can run the notebook directly using Google Colab [here](https://colab.research.google.com/github/snowplow/documentation/blob/main/static/notebooks/signals-personalize-travel-demo.ipynb), or download it locally.

## Set up your credentials

You'll need the same connection credentials you used in the previous step:

**CDI:**

```python
API_URL = 'example.signals.snowplowanalytics.com'
API_KEY = 'YOUR_API_KEY'
API_KEY_ID = 'YOUR_API_KEY_ID'
ORG_ID = 'YOUR_ORG_ID'
```

**Sandbox:**

```python
API_URL = 'you.signals.snowplowanalytics.com'
ACCESS_TOKEN = ''
```

***

## Define attributes

Run the notebook cell that defines the attributes. The notebook creates a series of attributes that represent different user preferences and behaviors, including interests in various types of travel experiences like luxury, budget, adventure, and culinary.

The attributes are based on these content tags:

```python
cultural_explorer_tags = ["culture", "history", "heritage", "ancient", "temples", "art", "traditional"]
modern_urbanite_tags = ["urban", "nightlife", "shopping", "modern", "architecture"]
tranquil_seeker_tags = ["nature", "peaceful", "wellness", "beaches", "mountains", "river", "wellness"]
family_fun_tags = ["family-friendly", "beaches", "nature", "food", "mountains", "culture"]
culinary_tourist_tags = ["food", "street food", "multicultural", "traditional", "urban", "shopping"]
```

The notebook defines these counter attributes:

- `page_view_count`
- `dest_page_view_count`
- `family_destination_count`
- `cultural_explorer`
- `modern_urbanite`
- `tranquil_seeker`
- `family_fun`
- `culinary_tourist`
- `budget_conscious`
- `luxury_inclined`

And these attributes with the `last` aggregation:

- `preferred_experience_length`
- `latest_schedule`

Running this cell defines the attributes locally but doesn't yet publish them to your Signals instance.

## Create an attribute group

Run the next notebook cell to define an attribute group for your attributes. This creates a `StreamAttributeGroup` with `domain_sessionid` as the attribute key:

```python
session_attributes_group = StreamAttributeGroup(
    name="travel_view",
    version=1,
    attribute_key=domain_sessionid,
    attributes=[page_view_count, dest_page_view_count, family_destination_count, cultural_explorer,
                modern_urbanite, tranquil_seeker, family_fun, culinary_tourist,
                preferred_experience_length, budget_conscious, luxury_inclined, latest_schedule],
    owner = 'you@email.com',
)
```

## Create a service

Run the notebook cell that defines a service to expose the attribute group via the API:

```python
travel_service = Service(
    name="travel_service",
    description="A service for our travel demo website.",
    attribute_groups=[session_attributes_group],
    owner='you@email.com'
)
```

## Publish to Signals

Finally, run the cell that publishes the attribute group and service to your Signals instance:

```python
response = sp_signals.publish([session_attributes_group, travel_service])
print(response)
```

Signals will start processing attributes from your real-time event stream.

---

# Define a Signals intervention to automatically start the travel AI agent
> Use Snowplow Signals interventions to automatically trigger chatbot appearance based on specific user behaviors.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/interventions/

Use Snowplow Signals interventions to control when the chatbot appears, transforming reactive customer support into proactive assistance based on behavioral triggers.

In the previous section, you personalized chatbot responses based on user behavior. This section shows how to use interventions to automatically present the chatbot when users demonstrate specific behaviors that suggest they might benefit from help.

Instead of waiting for users to click the chat icon, you can automatically trigger the agent to appear when users exhibit certain behaviors, such as browsing extensively without taking action.

## How it works

You'll create an intervention in Signals that triggers the agent to appear when a user has viewed more than three destination pages in a single session. This demonstrates how you can use interventions to control when an agent appears based on user behavior.

When the user views their fourth destination page, the agent will automatically appear and offer assistance.

## Create the intervention

Open your Jupyter notebook and run the final cell. This creates an intervention based on the `dest_page_view_count` attribute.

```python
from snowplow_signals import RuleIntervention, InterventionCriterion, LinkAttributeKey

destination_assistance_intervention = RuleIntervention(
    name="destination_help",
    owner="you@email.com",
    description="Assist the user if they are looking at 3 or more destinations in a given session.",
    target_attribute_keys=[LinkAttributeKey(name="domain_sessionid")],
    version=1,
    criteria=InterventionCriterion(
        attribute="travel_view:destination_page_view_count",
        operator=">=",
        value=3,
    ),
)

sp_signals.publish([destination_assistance_intervention])
```

## Test the intervention

The travel site automatically subscribes to interventions for the current user and session using the [Signals browser plugin](/docs/signals/interventions/subscribe/).

Once you've created the intervention, test it by:

1. Navigating to the travel site
2. Viewing more than three destination pages
3. When you view the fourth page, the agent should automatically expand, rather than requiring a click, and offer assistance

This proactive approach can significantly improve user experience by providing help at the right moment based on behavioral signals.

---

# Learn how to build a personalized travel agent with Signals
> Build personalized travel experiences using Snowplow Signals to customize content and chatbot responses based on real-time user behavior.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/intro/

Welcome to the **Build a personalized travel agent with Signals** tutorial.

[![Personalized Travel Experiences](/assets/images/demo-site-2a8ce88acac39ecf2c9d90d0c34ecc9c.jpg)](/assets/files/demo-site-2a8ce88acac39ecf2c9d90d0c34ecc9c.jpg/)

This tutorial walks you through building personalized experiences on a travel website using Snowplow Signals. You'll learn how to capture user behavior, define attributes that represent user preferences, and use these insights to customize both website content and chatbot responses in real-time.

Personalization has become essential for creating engaging user experiences. Traditional approaches often rely on static user profiles or require manual segmentation. Snowplow Signals enables dynamic personalization by processing behavioral data in real-time and making it immediately available for customization.

This tutorial is designed for developers, engineers, and analysts who want to implement real-time personalization. You'll work with a demo travel website focused on Southeast Asian destinations, creating a practical example of how behavioral data can drive personalized experiences.

You'll build a complete personalization system that captures user interests through their browsing behavior, processes this data into meaningful attributes using Snowplow Signals, and applies these insights to customize both the content displayed on the website and the responses provided by an AI-powered chatbot.

## Prerequisites

- Signals enabled for your CDI pipeline or [Snowplow Local](https://github.com/snowplow-incubator/snowplow-local) set up with [Signals Sandbox](https://try-signals.snowplow.io/dashboard)
- The [Snowplow Inspector](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm) browser extension installed
- Familiarity with running Jupyter notebooks, either locally or in Google Colab
- [Docker](https://www.docker.com/) installed and configured
- Access to the Jupyter [notebook](https://colab.research.google.com/github/snowplow/documentation/blob/main/static/notebooks/signals-personalize-travel-demo.ipynb)
- Optional: an [OpenAI API key](https://platform.openai.com/api-keys) to customize agent responses

---

# Provide Signals personalization context to the travel AI agent
> Integrate Snowplow Signals with an AI chatbot to provide personalized travel recommendations based on user behavioral data.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/personalizing-agent/

You'll now personalize a chatbot experience by integrating Signals with an AI agent that uses OpenAI as the backend. The chatbot fetches user attributes via a tool call, allowing it to provide contextual, personalized responses based on the user's browsing behavior.

The demo site includes all necessary tools and SDKs, so you'll only need an OpenAI API key to implement this feature.

## How it works

When a user asks the agent a question, the system fetches the attribute values for the user using the Signals API and uses these to modify the prompt sent to OpenAI. This allows personalized responses based on the user's preferences and behavior.

A toggle button on the chat widget allows you to enable or disable the tool call so you can see the difference in responses with and without Signals personalization.

## Data flow

```mermaid
sequenceDiagram
  Agent->>+OpenAI: Message
  Agent->>+Tool call: domain_sessionid
  Tool call->>+Typescript SDK: domain_sessionid
  Typescript SDK->>+Signals API: domain_sessionid, service_name
  Signals API->>+Typescript SDK: attributes
  Typescript SDK->>+Tool call: attributes
  Tool call->>+OpenAI: attributes
  OpenAI->>+Agent: personalized response
```

## Configure the agent

Add these variables to your `.env` file in the `snowplow-local` directory:

```text
AI_MODEL_PROVIDER=openai
AI_MODEL_NAME=gpt-4o-mini
OPENAI_API_KEY=your_openai_api_key
```

## Generate personalization data

You'll need some attribute data to personalize the agent responses. Browse different filters, destinations, and blog pages to generate additional attribute values.

Check your attribute values using the Snowplow Inspector **Attributes** tab. You should see values under the `travel_view` label.

## Test the agent

Start chatting with the agent by selecting the chat icon in the lower right portion of the screen.

First, test without personalization:

1. Turn the toggle switch off (gray means off, green means on)
2. Ask a question like "What are some good destinations for me in Southeast Asia?"
3. Note the generic response
4.

![Chat widget without personalization](/assets/images/chat-nops-7668c9509e3c08326232cd9b2970e845.jpg)

Then test with personalization:

1. Turn the toggle switch on (green)
2. Ask the same question
3. Compare the response - it should now be tailored to your browsing behavior

![Chat widget with personalization](/assets/images/chat-ps1-144ee2bbe76757306a80409195e261ac.jpg)

![Chat widget with personalization continued](/assets/images/chat-ps2-a7995e7ad8284f4492092bc4218672ec.jpg)

The personalized responses should provide destination suggestions and justifications that align with the preferences you've demonstrated through your behavior on the site.

---

# Add Signals personalization options to the travel site contents
> Use Snowplow Signals attributes to customize website content including images, descriptions, and layout based on user behavior.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/personalizing-site/

You'll now use your behavioral attributes to personalize the website experience in real-time. The travel website uses micro-segmentation based on behavioral data to customize various elements of the user experience.

The personalization system uses five different attributes that represent micro-segments based on user behavior patterns:

- `culinary_tourist`: users interested in food and culinary experiences
- `cultural_explorer`: users interested in cultural experiences
- `family_fun`: users interested in family friendly experiences and destinations
- `modern_urbanite`: users interested in urban experiences like architecture, nightlife, and shopping
- `tranquil_seeker`: users interested in relaxing experiences like beaches, spas, and nature

To determine the most appropriate micro-segment for a user, look at the value for each of these attributes (they are counters) and use the highest value.

## Data flow

Here's how the personalization works on the **Destinations** page:

```mermaid
sequenceDiagram
  React Frontend->>+Signals SDK (Node): domain_sessionid
  Signals SDK (Node)->>Signals API: domain_sessionid, service_name
  Signals API->>+Signals SDK (Node): Attribute values
  Signals SDK (Node)->>+React Frontend: Save to LocalStorage
```

To achieve this personalization you'll use the [Typescript Signals SDK](https://github.com/snowplow-incubator/snowplow-signals-typescript-sdk) to fetch the attribute values from the Signals API (requires authentication), and store these in local storage in the browser. You can then use these values to personalize the content on the destinations page.

## Customizing the page

The travel site has a **Destinations** home page, which you'll customise by:

1. Selecting images relevant to the segment
2. Modifying the description of the location based on the segment
3. Re-ordering the star ratings of different destination attributes, if relevant

Provide a JSON structure that defines which properties to use for each micro-segment. For example, here are the options for Bangkok, Thailand:

```json
[
  {
    "name": "Bangkok",
    "country": "Thailand",
    "image": "https://images.unsplash.com/photo-1508009603885-50cf7c579365?q=80&w=1950&ixlib=rb-4.0.3&auto=format&fit=crop&w=800&q=80",
    "images_category": {
        "cultural_explorer": "https://images.unsplash.com/photo-1690299490301-2eb3865bee58?q=80&w=2069&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
        "family_fun": "https://images.unsplash.com/photo-1733150632166-8d8752da4ff6?q=80&w=2232&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
        "modern_urbanite": "https://images.unsplash.com/photo-1593103499244-6c882f0163cf?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
        "tranquil_seeker": "https://images.unsplash.com/photo-1591233244269-d8c4bcbbf1dd?q=80&w=987&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
        "culinary_tourist": "https://images.unsplash.com/photo-1506781961370-37a89d6b3095?q=80&w=1674&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"
      },
    "description": "Vibrant capital with street food, temples, and bustling markets",
    "descriptions": {
        "cultural_explorer": "Grand palaces, ancient temples, and rich history await. Discover the soul of Thailand's capital.",
        "family_fun": "Kids love the tuk-tuks, canal tours, and vibrant markets. A city of endless family adventures.",
        "culinary_tourist": "A street food paradise with an explosion of flavours. Authentic Thai cuisine at every turn.",
        "tranquil_seeker": "Find peace in serene temples, tranquil gardens, and riverside long-tail boat journeys.",
        "modern_urbanite": "A vibrant, non-stop metropolis with world-class nightlife, shopping, and cutting-edge art."
      }
    }
]
```

For simplicity, these values are hardcoded but you could also generate them using generative AI. This results in more unique, but more randomized, results.

## Test the personalization

You can test the personalization by performing different actions across the website to increase the counter for a given micro-segment. Go back to your Jupyter notebook and examine the attribute definitions to see what actions trigger each segment.

Once you've performed 4 or more actions that align with a specific segment, refresh the destinations page. It should now be customized to your segment.

Here's what the page looks like before:

![Before personalization](/assets/images/no-pers-d34fc246d7d40c47e762bec8a54d4b5b.jpg)

After personalization for a `culinary_tourist` user, the images and descriptions have been updated to focus on food, and the food star rating has been put first:

![After personalization (culinary tourist segment)](/assets/images/with-pers-499344d057f9f424d4fd912e10e360dc.jpg)

The same destination is presented differently based on the user's demonstrated interests and behaviors.

---

# Run the demo Signals React travel application
> Set up and run the demo travel website using Docker and Snowplow Local.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/setting-up/

You'll now install and run the example travel website that you'll use to test Snowplow Signals personalization. This website represents a typical ecommerce travel platform where users browse destinations, read content, and interact with various features.

The demo website is part of the [Snowplow Local](https://github.com/snowplow-incubator/snowplow-local) repository. It's a React application.

## Clone Snowplow Local

Clone the Snowplow Local repository to your machine:

```text
git clone git@github.com:snowplow-incubator/snowplow-local.git
```

Change directory into the `snowplow-local` folder:

```text
cd snowplow-local
```

## Configure environment variables

Create an `.env` file based on the example file:

```text
cp .env.example .env
```

Edit the `.env` file with your Signals connection credentials:

**CDI:**

```bash
NEXT_PUBLIC_SNOWPLOW_SIGNALS_API_URL=signals.snowplow.com
SNOWPLOW_SIGNALS_API_KEY=
SNOWPLOW_SIGNALS_API_KEY_ID=
SNOWPLOW_SIGNALS_ORGANIZATION_ID=
NEXT_PUBLIC_SNOWPLOW_COLLECTOR_URL=
```

**Sandbox:**

```bash
NEXT_PUBLIC_SNOWPLOW_SIGNALS_API_URL=sandbox.signals.snowplow.com
SNOWPLOW_SIGNALS_INGEST_URL=
SNOWPLOW_SIGNALS_TRIAL_TOKEN=
```

***

If you plan to use the AI agent add an OpenAI API key (`OPENAI_API_KEY`) to your `.env` file.

## Run the travel website

Run the following Docker command:

**CDI:**

```bash
docker compose --profile travel-site up
```

**Sandbox:**

```bash
docker compose --profile travel-site --profile signals up
```

***

## Test the website

Open the travel website in your browser at <http://localhost:8086>. You should see the homepage of the travel site.

Open your browser's developer console (Ctrl+Shift+I or equivalent) and go to the Snowplow Inspector tab. Generate some events by clicking on filters on the [destinations](http://localhost:8086/destinations) page. You should see self-describing events firing into your Snowplow pipeline.

Explore the site to get familiar with its features before you define the attributes in the next step.

---

# Test your Signals attribute definitions using the Snowplow Inspector
> Verify that your Snowplow Signals attributes are working correctly by generating events and checking attribute values.
> Source: https://docs.snowplow.io/tutorials/signals-personalize-travel/testing-attributes/

You'll now test your attributes to verify they're working correctly by generating behavioral events and checking that the attribute values update as expected.

You'll use the Snowplow Inspector browser extension to monitor both the events being sent to your collector and the resulting attribute values.

## Generate events

Start by generating events on your travel website:

1. Go to your [demo site](http://localhost:8086)
2. Click on **Destinations** in the top navigation bar
3. Click on any destination page
4. Refresh the page once you're on the destination page

## Check attribute values

Open your browser's developer console (Ctrl+Shift+I or equivalent) and navigate to the [Snowplow Inspector](https://chromewebstore.google.com/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm) tab:

![Snowplow Inspector](/assets/images/inspector-efdb4aa7960e09c637a516b8b89f1db8.jpg)

1. In the **Events** tab, verify you can see page view events being sent to your collector endpoint
2. In the **Attributes** tab, check that the `page_view_count` and `dest_page_view_count` attributes have non-zero values

If you're not using the Chrome extension, you can check the developer console logs instead. Note that the logs are output when the page is refreshed, so they may lag behind the values shown in the Inspector.

## Troubleshoot

If your attribute values appear as null or zero:

- Check that your events are being successfully sent to your collector
- If you're using event forwarding, verify your forwarding address is correct
- Ensure your Signals credentials are configured properly in the `.env` file

You can now move on to personalizing the website content using these attributes.

---

# Conclusions and next steps from the Signals quick start tutorial
> Complete the Snowplow Signals quick start tutorial, and explore next steps for real-time customer intelligence.
> Source: https://docs.snowplow.io/tutorials/signals-quickstart/conclusion/

In this tutorial you've learned how to use Signals to calculate and serve information about user behavior in real time.

This is the process workflow:

- Create attribute group
- Publish the attribute group to Signals
- Create service
- Retrieve calculated attribute values from the Profiles Store

## Next steps

Here are some ideas for further exploration:

- Try out other entities, e.g. `domain_userid`, to calculate the attributes against

- Define attributes based off other event types

- Retrieve calculated attributes in your real applications

- Explore our other Signals tutorials:

  - [Implement Real-Time Interventions in an Ecommerce App Using Signals](/tutorials/signals-interventions/start/).
  - [Score prospects in real time using Signals and ML](/tutorials/signals-ml-prospect-scoring/intro/)
  - [Set up the Signals batch engine](/tutorials/signals-batch-engine/start/) for calculating attributes from your warehouse

---

# Define a Signals attribute group for session metrics
> Create an attribute group in Snowplow Signals to calculate session metrics like page views, browser, and referrer data in real time.
> Source: https://docs.snowplow.io/tutorials/signals-quickstart/define-attribute-group/

[Attribute groups](/docs/signals/concepts/#attribute-groups) are where you define the data you want to calculate. To create an attribute group, go to **Signals** > **Attribute groups** in Snowplow Console and click **Create attribute group**.

![Create attribute group form for quickstart tutorial](/assets/images/attribute-group-create-05c9671a86bc1133c6625497bb434064.png)

Follow these instructions to configure Signals to calculate three different session metrics from page views in your real-time event stream:

- How many page views in the last 15 minutes for each session
- The last seen browser name for each session
- The first seen page referrer for each session

To learn how to define attribute groups using the Python SDK, check out the [Quick Start notebook](https://colab.research.google.com/github/snowplow-incubator/signals-notebooks/blob/main/quickstart.ipynb), hosted on Google Colab.

## Configure group information

Specify the basic configuration for your attribute group:

- **Name**: `quickstart_group`
- **Description**: Quick Start tutorial: page view session metrics
- **Source**: stream
- **Primary owner**: your email address

The name will be the group's unique identifier. The description and owner are optional.

Under the Configuration section, select the following:

- **Attribute key**: `domain_sessionid`
- **TTL**: leave as default

## Define attributes

Click **Add attribute** to create each one.

![Add attribute button in attribute group configuration](/assets/images/attribute-group-add-attribute-9c768a8601d840dcb5dfc584e5cbbbd3.png)

### Page view counter

The first attribute is a count of the number of page view events within the last 15 minutes. Enter `page_view_count` in the attribute name field.

To set the event to calculate this attribute from:

1. Click on the event filter field to bring up the event selection options
2. Page views are a built-in Snowplow event, so they'll be listed within the default **Snowplow events** tab
3. Click in the search box to find `page_view`
4. Click **Confirm** to add the event to the attribute

![Event selection showing page\_view event in Snowplow events list](/assets/images/attribute1-specify-page-view-cfbc1204a7d596b7c79114f00b078d3e.png)

Leave the aggregation as `Counter`. No property is used for this aggregation, so leave the property field blank.

To set the time period:

1. Click on the **More** button
2. Update the time period to 15 minutes
3. Click **Done** to save

![Time period configuration set to 15 minutes for page view counter](/assets/images/attribute1-set-period-1245a531daa2acb8b7019decfda6c451.png)

The purple dot next to **More** indicates that you have extended settings.

![Completed page view counter attribute with time period indicator](/assets/images/attribute1-complete-1eddcadd29839f811a771f978316bf1b.png)

### Most recent browser

The second attribute is the last seen browser name. The calculation makes use of the [YAUAA enrichment](/docs/pipeline/enrichments/available-enrichments/yauaa-enrichment/): the browser name is a field in the `yauaa_context` entity.

Create an attribute named `most_recent_browser`, and select `page_view` as before.

Choose the `Last` aggregation.

To set the property:

1. Click on the property selection field
2. Choose the **Entities** tab to search through all schemas that have been tracked as entities with your events
3. Use the search bar to search for `yauaa_context`
4. Select the entity `yauaa_context (nl.basjes)`
5. Select the `agentName` property
6. Click **Confirm** to save

![Property selection showing YAUAA context agentName field for browser attribute](/assets/images/attribute2-property-selector-d2cb6ca2f9acc7207cf1a5eaa5c35f9c.png)

### First referrer

The third attribute stores the first seen referrer path, based on the `refr_urlhost` [atomic event property](/docs/fundamentals/canonical-event/#web-specific-fields).

Create an attribute named `first_referrer`, and select `page_view` as before.

Choose the `First` aggregation.

To set the property:

1. Click on the property selection field
2. Stay on the default **Atomic** tab to search through all atomic properties
3. Use the search bar to search for and select `refr_urlhost`
4. Click **Confirm** to save

![Property selection showing refr\_urlhost atomic field for referrer attribute](/assets/images/attribute3-property-selector-cae8bd66bb4c82c99f0c4eb19a856ee8.png)

For a trivial example of using criteria filters, add a filter to only consider events where the referrer is not an empty string.

To set the criteria filter:

1. Click on the **More** button
2. Click **Add criteria**
3. Choose the `page_referrer` atomic property and click **Confirm**
4. Change the operator to `not equals`
5. Leave the value blank
6. Click **Done** to return to the group details page

![Criteria configuration filtering for non-empty referrer values](/assets/images/attribute3-criteria-c7f348fa7a4d62046e99bb08d56ded51.png)

## Test the attribute definitions

Once you've added attributes, click **Run preview** to test your attribute group configuration.

![Attribute group with three attributes ready for preview testing](/assets/images/attribute-group-ready-to-test-5aa563c932d646a7e65d6d9cd17c665e.png)

This will calculate the attributes from your atomic events table using 10 random events from the last hour.

You should see something like this:

![Test results showing calculated attributes for sample session IDs](/assets/images/attribute-group-test-results-6d563dd6fffe08fc3f9d74502b2e51e6.png)

The first column shows the unique attribute key values, in this case for the session attribute key `domain_sessionid`.

## Save the attribute group

Once you're satisfied with the preview results, click **Create attribute group** to save it as a draft.

![Draft attribute group page after saving configuration](/assets/images/attribute-group-draft-c1d87d4475afae4157d9f077bd4b0d77.png)

Click **Publish** to push this configuration to Signals and start calculating attributes.

![Published attribute group showing active calculation status](/assets/images/attribute-group-published-2cf1af60f705a14cbe0ba0d61c5390f6.png)

---

# Define a Signals tutorial service
> Configure a Snowplow Signals service to group attribute groups for easy consumption in applications.
> Source: https://docs.snowplow.io/tutorials/signals-quickstart/define-service/

[Services](/docs/signals/concepts/#services) group attribute groups together for serving to your applications.

To create a service, go to **Signals** > **Services** in Snowplow Console and click **Create service**.

![Create service form for quickstart tutorial service](/assets/images/service-create-85585d669c129ebb7cb30d74209dfc12.png)

## Configure basic service information

Specify the configuration for your service:

- **Name**: `quickstart_service`
- **Description**: Quick Start tutorial service
- **Owner**: your email address

The name will be the service's unique identifier. The description and owner are optional.

## Select attribute groups

When choosing which attribute groups to include, you'll select a specific version of each attribute group.

To add an attribute group:

1. Click the attribute group selector
2. Search for and select the group you just created, `quickstart_group (v1)`

If `quickstart_group (v1)` attribute group isn't showing up, check that it's published.

![Attribute group selector showing quickstart\_group v1 selection](/assets/images/service-choose-groups-3eb324c39c910489d866001bf3cb9cc5.png)

## Create the service

Click **Create service** to save your service configuration. Services are automatically published as soon as they're created.

![Published quickstart service ready for attribute retrieval](/assets/images/service-published-5f5116562383bfe95d332ac7fd6b070c.png)

Your service is ready to use for retrieving attributes from the Profiles Store.

---

# Retrieve calculated attributes using the Signals Python SDK
> Access and consume calculated behavioral data attributes from the Snowplow Signals Profiles Store using the Python SDK.
> Source: https://docs.snowplow.io/tutorials/signals-quickstart/retrieve-attributes/

For a real use case, you'll want to consume calculated attributes in your applications. Read more about this [in the Signals documentation](/docs/signals/attributes/).

For this tutorial, we've provided a [Jupyter notebook](https://colab.research.google.com/github/snowplow-incubator/signals-notebooks/blob/main/quickstart.ipynb) so you can quickly explore attribute retrieval using the Signals Python SDK.

## Finding your current session ID

In your real application code, you can access the current session ID and use it to retrieve the relevant attribute values. The attributes are being calculated in real time, in session. Read about how to access IDs such as `domain_sessionid` in your web application in [the JavaScript tracker](/docs/sources/web-trackers/cookies-and-local-storage/getting-cookie-values/) documentation.

To test this out, use the [Snowplow Inspector](/docs/testing/snowplow-inspector/) browser extension to find out your current session ID on your web application. Click around and generate some page view events. Then find your `Domain Session ID` in the Inspector.

![Screenshot showing the session ID in the Snowplow Inspector](/assets/images/inspector-session-8a359813742a08d72749eacb5c4727b5.png)

## Connecting to Signals

Install the [Signals Python SDK](https://pypi.org/project/snowplow-signals/) into the notebook, and connect to Signals.

1. Go to **Signals** > **Overview** in Snowplow Console to find your Signals credentials
2. Add them to the notebook secrets:

![Screenshot showing how to add secrets](/assets/images/notebook-secrets-0e5fd8ee477440010bd1671d5b6e2908.png)

3. Install the SDK:

```python
%pip install snowplow-signals
```

4. Connect to Signals:

```python
from snowplow_signals import Signals
from google.colab import userdata

sp_signals = Signals(
    api_url=userdata.get('SP_API_URL'),
    api_key=userdata.get('SP_API_KEY'),
    api_key_id=userdata.get('SP_API_KEY_ID'),
    org_id=userdata.get('SP_ORG_ID'),
)
```

## Retrieving your session attributes

Use your current session ID to retrieve the attributes that Signals has just calculated about your session.

```python
response = sp_signals.get_service_attributes(
    name="quickstart_service",
    attribute_key="domain_sessionid",
    identifier="472f97c1-eec1-45fe-b081-3ff695c30415", # UPDATE THIS
)

df=response.to_dataframe()
df
```

The result should look something like this:

|   | `domain_sessionid`                     | `page_view_count` | `most_recent_browser` | `first_referrer` |
| - | -------------------------------------- | ----------------- | --------------------- | ---------------- |
| 0 | `472f97c1-eec1-45fe-b081-3ff695c30415` | 2.0               | `Firefox`             | `snowplow.io`    |

### Retrieving single attributes

To retrieve individual attributes rather than using a service, use the `get_group_attributes()` method.

```python
response = sp_signals.get_group_attributes(
    name="quickstart_group",
    version=1,
    attributes=["page_view_count"],
    attribute_key="domain_sessionid",
    identifiers=["472f97c1-eec1-45fe-b081-3ff695c30415"]
)

df=response.to_dataframe()
df
```

---

# Learn how to set up Signals for real-time calculation
> Get started with Snowplow Signals to calculate user behavior attributes in real time, to build personalization use cases.
> Source: https://docs.snowplow.io/tutorials/signals-quickstart/start/

Welcome to the [Snowplow Signals](/docs/signals/introduction/) Quick Start tutorial.

Snowplow Signals is a real-time personalization engine for customer intelligence, built on Snowplow's behavioral data pipeline. It allows you to compute, access, and act on in-session stream and historical user data, in real time.

This guide will walk you through the steps to calculate user behavior attributes from your Snowplow event stream, and to retrieve them for use in your application. This will unlock real-time personalization use cases for your business.

It should take less than 10 minutes from starting to retrieving calculated attributes.

This tutorial shows how to define attributes using the Snowplow Console UI, as well as programmatically using the [Signals Python SDK](https://pypi.org/project/snowplow-signals/).

## Prerequisites

This tutorial assumes that you have:

- Snowplow page view tracking on a web application
- Snowflake warehouse
- A Signals connection

## Connecting to Signals

Log in to [Console](https://console.snowplowanalytics.com) and navigate to the **Signals** section.

You'll need to [set up a Signals connection](/docs/signals/connection/) if you don't have one yet.

---

# Create and publish a data structure using the Snowplow CLI MCP tool
> Step-by-step walkthrough of creating, validating, and publishing a data structure and tracking plan using the Snowplow CLI MCP tool.
> Source: https://docs.snowplow.io/tutorials/snowplow-cli-mcp/basic-workflow/

Here's a typical interaction pattern for creating a data structure.

## 1. Get context

> **Note:** Always ensure `get_context` is called at the start of your conversation. If you don't see it happen, then ask for it.

```text
Please call get_context before we start working.
```

The assistant will retrieve the built-in schema and rules that define how Snowplow components should be structured. This provides the structural templates and requirements for your tracking implementation.

## 2. Create a data structure

```text
Create a data structure for tracking when users view a product page.

Include properties for product ID, product name, category, and price.
```

The assistant will:

- Generate a proper UUID for the data structure
- Create a valid event schema following Snowplow conventions
- Save the file locally to the appropriate location

**Note**: Files are created locally only. Use `snowplow-cli data-structures publish` to sync to Console when ready.

## 3. Validate (automatic)

The assistant should automatically call `validate_data_structures` on the created file and report any validation issues.

## 4. Iterate if needed

```text
The price should be optional, not required.

Also add a description field.
```

The assistant will modify the structure and re-validate.

## 5. Tracking plan creation

```text
Create a tracking plan for ecommerce product interactions. Include:
- The existing product page views
- Add to cart events
- A source application for our website
- Proper validation of all components
```

The assistant will:

1. Create the necessary data structures for events (locally)
2. Create a source application definition (locally)
3. Create a tracking plan linking everything together (locally)
4. Validate all components together (including cross-references)

## 6. Publish to Console

Use `snowplow-cli data-structures publish` and `snowplow-cli data-products publish` to push changes to Console.

---

# Conclusion
> Summary of what you learned in the Snowplow CLI MCP tool tutorial.
> Source: https://docs.snowplow.io/tutorials/snowplow-cli-mcp/conclusion/

You've successfully set up the Snowplow CLI MCP tool and learned how to use AI assistants to manage your Snowplow data structures and tracking plans through natural conversation.

In this tutorial, you:

- Installed and configured the Snowplow CLI with an MCP-compatible AI assistant
- Learned how to create and validate data structures using natural language prompts
- Built complete tracking plans including events, entities, and source applications
- Published your locally created structures to Snowplow Console

## Next steps

- Read the [Snowplow CLI documentation](/docs/event-studio/programmatic-management/snowplow-cli/) for further workflows and commands
- Follow the [Snowplow CLI Git tutorial](/tutorials/data-structures-in-git/introduction/) to manage your tracking plans with Git version control
- Check out our [tracking design best practise guide](/docs/fundamentals/tracking-design-best-practice/)

---

# Learn how to set up the Snowplow CLI MCP tool
> Introduction to the Snowplow CLI MCP tool for managing data structures and tracking plans through AI assistants like Claude, Cursor, or Copilot.
> Source: https://docs.snowplow.io/tutorials/snowplow-cli-mcp/introduction/

The Snowplow CLI MCP (Model Context Protocol) tool integrates Snowplow's data structure management capabilities directly into AI assistants like Claude. This enables natural language interaction for creating, validating, and managing your Snowplow tracking plans **locally**.

**Important**: the MCP tool creates and validates files on your local filesystem only. To sync changes to Console, you'll use the regular CLI commands like `snowplow-cli ds publish` afterward.

## What you'll learn

- How to set up the Snowplow CLI MCP tool with AI assistants like Claude Desktop, Cursor, or Copilot
- Available MCP tools and their functions
- Creating and validating data structures through conversation
- AI-powered analysis for strategic tracking plan development

## Demo: Using the Snowplow CLI MCP with Claude Desktop

## Prerequisites

- Snowplow CLI installed ([installation guide](/docs/event-studio/programmatic-management/snowplow-cli/#install))
- Snowplow CLI configured with your Console credentials ([configuration guide](/docs/event-studio/programmatic-management/snowplow-cli/#configure))
- Claude Desktop or another MCP-compatible client (Cursor or Copilot)
- **Filesystem access**: if using Claude Desktop, you must run alongside an MCP filesystem server (e.g., `@modelcontextprotocol/server-filesystem`) to enable file operations. Other MCP clients (Cursor, Copilot, etc.) have filesystem access by default.

## Available MCP tools

The Snowplow CLI MCP server provides these tools:

### Core tools

- **`get_context`** - Retrieves the built-in schema and rules that define how Snowplow data structures, tracking plans, and source applications should be structured.
- **`get_uuid`** - Generates valid v4 UUIDs required by many Snowplow components.

### Validation tools

- **`validate_data_structures`** - Validates data structure files (events/entities). Must be called after creating or modifying any data structure.
- **`validate_data_products`** - Validates tracking plans and source applications. Must include both tracking plan files AND their referenced source application files.

---

# Next steps from the Snowplow CLI MCP tool tutorial
> Advanced use cases for the Snowplow CLI MCP tool, including business process analysis, schema evolution planning, and cross-platform tracking.
> Source: https://docs.snowplow.io/tutorials/snowplow-cli-mcp/next-steps/

Beyond creating individual files, AI assistants can help analyze tracking requirements and suggest comprehensive solutions:

### Business process analysis

```text
We want to track user engagement on our blog
```

The assistant will suggest:

- Page view events with article entity
- Custom events for shares and subscriptions
- Reading progression metrics
- Time-based engagement tracking

### Schema evolution planning

```text
Our product_viewed event is missing context about where users found the product
```

The assistant will provide:

- Recommendations for adding entities
- Versioning strategy
- Guidelines for maintaining consistency with related events

### Journey mapping

```text
Create a tracking plan for our checkout funnel
```

The assistant will create:

- Complete tracking plan covering cart management through payment
- Error state tracking
- Abandonment scenario tracking

### Cross-platform consistency

```text
We're adding mobile app tracking to our existing web tracking
```

The assistant will analyze:

- Which entities should be consistent across platforms
- Platform-specific contexts needed
- Tracking plan structure recommendations

---

# Install the Snowplow CLI and configure an MCP client
> How to install the Snowplow CLI and configure MCP clients such as Claude Desktop, VS Code, and Cursor.
> Source: https://docs.snowplow.io/tutorials/snowplow-cli-mcp/setup/

To use the MCP tools with your AI assistant, you'll need to install the Snowplow CLI and configure your chosen MCP client. This guide walks you through both steps.

## 1. Install Snowplow CLI

```bash
# Using Homebrew
brew install snowplow/taps/snowplow-cli

# Or download binary directly
curl -L -o snowplow-cli https://github.com/snowplow/snowplow-cli/releases/latest/download/snowplow-cli_linux_x86_64
chmod u+x snowplow-cli
```

If you have `node.js` set up then no need to install, you can run via `npx`.

## 2. Configure your MCP client

The Snowplow CLI MCP server can be used with any MCP-compatible client. Below are configuration examples for popular clients. For a complete list of supported clients and their configurations, see the [MCP reference](/docs/event-studio/programmatic-management/snowplow-cli/reference/#mcp).

### Claude desktop

**Config location**:

- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "snowplow-cli",
      "args": ["mcp"]
    }
  }
}
```

### VS code

Add to `.vscode/mcp.json` in your workspace:

```json
{
  "servers": {
    "snowplow-cli": {
      "type": "stdio",
      "command": "snowplow-cli",
      "args": ["mcp"]
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` in your workspace:

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "snowplow-cli",
      "args": ["mcp", "--base-directory", "."]
    }
  }
}
```

### Using with npx

If using via `npx`, use this configuration format instead:

```json
{
  "mcpServers": {
    "snowplow-cli": {
      "command": "npx",
      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]
    }
  }
}
```

For VS Code, adjust the `type` field accordingly:

```json
{
  "servers": {
    "snowplow-cli": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@snowplow/snowplow-cli", "mcp"]
    }
  }
}
```

## 3. Start using MCP

After configuring your chosen client, start a new conversation or session. The Snowplow CLI MCP tools should be available for use.

---

# Learn how to set up the Unified Digital dbt package
> Learn how to set up the Snowplow Unified Digital dbt package for web and mobile analytics. Covers prerequisites for implementing unified cross-platform tracking with DBT transformations.
> Source: https://docs.snowplow.io/tutorials/unified-digital/intro/

This tutorial walks you through the process of setting up our Unified Digital dbt package.

## Prerequisites

- [DBT](https://github.com/dbt-labs/dbt) installed

- Connection to a warehouse

- For web:

  - web events dataset being available in your database
  - [Snowplow Javascript tracker](/docs/sources/web-trackers/) version 2 or later implemented.
  - Web Page context [enabled](/docs/sources/web-trackers/tracker-setup/initialization-options/) (enabled by default in version 3+).
  - [Page view events](/docs/sources/web-trackers/tracking-events/page-views/) implemented.

- For mobile:

  - Mobile events dataset being available in your database
  - Snowplow Android, iOS [mobile tracker](/docs/sources/mobile-trackers/) version 1.1.0 (or later) or [React Native tracker](/docs/sources/react-native-tracker/) implemented
  - Mobile session context enabled
  - Screen view events enabled and tracked

---

# Set up the Unified Digital dbt package locally
> Install and configure the Snowplow Unified Digital dbt package locally with detailed variable configuration for web and mobile data modeling. Includes optimization tips for partition pruning, entity enablement, and warehouse-specific setup.
> Source: https://docs.snowplow.io/tutorials/unified-digital/setting-up-locally/

Installing the package:

1. Run the following command in a new directory to create a new dbt project

   ```bash
   dbt init
   ```

2. Set up a profile to connect to the warehouse where you have your Snowplow events.

   ```yaml
   profile: 'demo_project'
   ```

3. Create a new `packages.yml` file.

4. Paste the latest version of the Snowplow Unified package. You could also use a range here if you want to keep up to date with more recent versions or you can you can hard pin to a specific version.

   ```bash

   packages:
     - package: snowplow/snowplow_unified
       version: [">=0.4.0"]
   ```

5. Type the following to install the package into this project:

   ```bash
   dbt deps
   ```

6. In your `project.yml` copy across our dispatch piece so that you're using our macros over the dbt core ones. We do this because we have a slightly more optimized upsert which saves you time and cost on your on your cloud warehouse bill.

   ```bash
   dispatch:
     - macro_namespace: dbt
       search_order: ['snowplow_utils', 'dbt']
   ```

7. Delete some of the other default pieces that are in the default project as they are not needed.

   ![](/assets/images/Screenshot_2024-07-04_at_17.14.37-9c416c5455c4fe0fe89d583b736a312e.png)

## Setting Variables

Now we’ll get to using our variables, which is how you enable the parts of the model that are relevant to your use-case.

1. Define the location of your source data within your `vars` block where your raw events are being loaded into. Make sure to update these with your actual table names!

   ```yaml
   vars:
     snowplow_unified:
       snowplow__atomic_schema: schema_with_snowplow_events
       snowplow__database: database_with_snowplow_events
   ```

> **Info:** Please note that your `target.database` is NULL if using Databricks. In Databricks, schemas and databases are used interchangeably and in the dbt implementation of Databricks therefore we always use the schema value, so adjust your `snowplow__atomic_schema` value if you need to.
>
> Add the following variable to your dbt project's `dbt_project.yml` file
>
> ```yml
> vars:
>   snowplow_unified:
>     snowplow__databricks_catalog: 'hive_metastore'
> ```
>
> Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to 'hive\_metastore') or the same value as your `snowplow__atomic_schema` (unless changed it should be 'atomic'). This is needed to handle the database property within `models/base/src_base.yml`.
>
> A more detailed explanation for how to set up your Databricks configuration properly can be found in [Unity Catalog support](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/#unity-catalog-support).

2. Unified Digital assumes you are modeling both web and mobile events and expects certain fields to exist based on this. If you are only tracking and modeling e.g. web data, you can disable the other as below:

```yml
vars:
  snowplow_unified:
    snowplow__enable_mobile: false
    snowplow__enable_web: true
```

3. Enable [entities](/docs/fundamentals/entities/) to make sure that they're processed within the package - this means they will be un-nested from the atomic columns and made available in the derived tables. Make sure to only enable the ones you need.

   ```yaml
   vars:
     snowplow_unified:
       snowplow__enable_iab: true
       snowplow__enable_ua: true
       snowplow__enable_yauaa: true
       snowplow__enable_browser_context: true
       snowplow__enable_mobile_context: true
       snowplow__enable_geolocation_context: true
       snowplow__enable_application_context: true
       snowplow__enable_screen_context: true
       snowplow__enable_deep_link_context: true
       snowplow__enable_consent: true
       snowplow__enable_cwv: true
       snowplow__enable_app_errors: true
       snowplow__enable_screen_summary_context: true
   ```

4. Set up some initial conditions for our app. So we need to go in here and we need to just set the start date to be whenever you've started tracking your data from:

   ```yaml
   vars:
     snowplow_unified:
       snowplow__start_date: 'yyyy-mm-dd'
   ```

5. Optimize your data processing

There are ways how you can deal with [high volume optimizations](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-custom-models/high-volume-optimizations/) at a later stage, if needed, but you can do a lot upfront by selecting carefully which variable to use for `snowplow__session_timestamp`, which helps identify the timestamp column used for sessionization. This timestamp column should ideally be set to the column your event table is partitioned on. It is defaulted to `collector_tstamp` but depending on your loader it can be the `load_tstamp` as the sensible value to use:

```yml
vars:
  snowplow_unified:
    snowplow__session_timestamp: 'load_tstamp'
```

> **Info:** Verify which column your events table is partitioned on. It will likely be partitioned on `collector_tstamp` or `derived_tstamp`. If it is partitioned on `collector_tstamp` you should set `snowplow__derived_tstamp_partitioned` to `false`. This will ensure only the `collector_tstamp` column is used for partition pruning when querying the events table:
>
> ```yml
> vars:
>   snowplow_unified:
>     snowplow__derived_tstamp_partitioned: false
> ```

6. Configure more vars as necessary but theoretically this is all you need just to get started. If you are unsure whether the default values set are good enough in your case or you would already like to maximize the potential of your models, you can dive deeper into the meaning behind our variables on our [Config](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/) page. It includes a [Config Generator](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/unified/#config-generator) to help you create all your variable configurations, if necessary.

7. Filter your data set

You can specify both `start_date` at which to start processing events and the `app_id`'s to filter for. By default the `start_date` is set to `2020-01-01` and all `app_id`'s are selected. To change this please add the following to your `dbt_project.yml` file:

```yml
vars:
  snowplow_unified:
    snowplow__start_date: 'yyyy-mm-dd'
    snowplow__app_id: ['my_app_1','my_app_2']
```

Below we list a few more that might be of interest depending on your setup or modelling needs:

- Enable extras The package comes with additional modules and functionality that you can enable, for more information see the [consent tracking](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/consent-module/), [conversions](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/conversions/), and [core web vitals](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/core-web-vitals-module/) documentation.

- adjust page ping variables, if needed The Unified Digital Model processes page ping events to calculate web page engagement times. If your [tracker configuration](/docs/sources/web-trackers/tracking-events/activity-page-pings/) for `min_visit_length` (default 5) and `heartbeat` (default 10) differs from the defaults provided in this package, you can override by adding to your `dbt_project.yml`:

```yml
vars:
  snowplow_unified:
    snowplow__min_visit_length: 5 # Default value
    snowplow__heartbeat: 10 # Default value
```

## Adding the `selectors.yml` file

Within the packages we have provided a suite of suggested selectors to run and test the models within the package together with the Unified Digital Model. This leverages dbt's [selector flag](https://docs.getdbt.com/reference/node-selection/syntax). You can find out more about each selector in the [Model selection](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/model-selection/) documentation.

These are defined in the `selectors.yml` file ([source](https://github.com/snowplow/dbt-snowplow-unified/blob/main/selectors.yml)) within the package, however in order to use these selections you will need to copy this file into your own dbt project directory. This is a top-level file and therefore should sit alongside your `dbt_project.yml` file. If you are using multiple packages in your project you will need to combine the contents of these into a single file.

## Running the package

Run dbt\_seed to make sure you see some data, so we have some seeds in our packages, run that in there, and then run the actual model.

```yaml
dbt seed --select snowplow_unified --full-refresh
dbt run --selector snowplow_unified
```

---

# Run the Unified Digital dbt package using Snowplow Console
> Deploy the Unified Digital dbt package through Snowplow Console's fully managed service. Configure web and mobile data processing, enrichments, and scheduling through the Console interface.
> Source: https://docs.snowplow.io/tutorials/unified-digital/setting-up-via-console/

Once you are happy with how your local configuration is working, you will want to schedule it to run regularly. Snowplow provides a fully managed service for running data models securely.

To get started, follow these steps:

1. Commit the local configuration you built previously to a Git repository.

2. Create a Git connection to your repository. In Console, navigate to **Destinations > Connections > Set up connection > Git connection**

3. Create a warehouse connection. In Console, navigate to **Destinations > Connections > Set up connection > Data modeling connection**

4. Create a model project. In Console, select **Modeling > Model projects > Create model project**

5. Add a run configuration with a schedule to your model project. Click **Add run configuration** and **Add schedule**. Pick your preferred schedule, e.g. daily at 00:00 AM.

For a full reference on running data models via Snowplow Console, see [Run data models from Console](/docs/modeling-your-data/running-data-models-via-console/).

---

# View the Unified Digital dbt package output data
> Query and analyze the Unified Digital dbt package output tables including views, sessions, and users. Explore aggregated metrics, engagement time calculations, and user interaction data.
> Source: https://docs.snowplow.io/tutorials/unified-digital/viewing-output-data/

The output data can now be reviewed, focusing on the `views`, `sessions`, and `users` tables. These tables contain detailed records of key metrics and interactions.

```yaml
select * from dbt_ryan_derived.snowplow_unified_views;
```

This query retrieves one row per view, detailing when each view started and ended, along with extensive information on where the event occurred.

```yaml
select * from dbt_ryan_derived.snowplow_unified_sessions;
```

The sessions table includes the engagement time in seconds, calculated using pings in comparison to absolute time. It also contains additional data, such as counts of page views and events.

```yaml
select * from dbt_ryan_derived.snowplow_unified_users;
```

The users table provides one row per user identifier, offering detailed information about each user.