# Load Snowplow data to Databricks
> Send Snowplow data to Databricks for analytics and data processing with Delta Lake tables, schema evolution, and lakehouse architecture support.
> Source: https://docs.snowplow.io/docs/destinations/warehouses-lakes/databricks/
> Documentation index: https://docs.snowplow.io/llms.txt

[Snowplow supported](/docs/destinations/#destinations-catalog "Built and supported by Snowplow") · Warehouse or lake loader · Streaming · Batch · AWS, Azure, GCP

> **Info: Cloud availability**
>
> The Databricks integration is available for Snowplow pipelines running on **AWS**, **Azure** and **GCP**.

The Snowplow Databricks integration allows you to load enriched event data (as well as [failed events](/docs/fundamentals/failed-events/)) into your Databricks environment for analytics, data modeling, and more.

Depending on the cloud provider for your Snowplow pipeline, there are different options for this integration:

| Integration                                                                                                         | AWS | Azure | GCP | Failed events support |
| ------------------------------------------------------------------------------------------------------------------- | --- | ----- | --- | --------------------- |
| Direct, batch-based ([RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/))                | ✅   | ❌     | ❌   | ❌                     |
| Via Delta Lake ([Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/))                            | ❌¹  | ✅²    | ✅²  | ✅                     |
| Via Iceberg ([Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/))                               | ✅²  | ❌     | ❌³  | ✅                     |
| Streaming / Lakeflow ([Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/)) | ✅   | ✅     | ✅   | ✅                     |

_¹ Delta+Databricks combination is currently not supported for AWS pipelines. The loader uses DynamoDB tables for mutually exclusive writes to S3, a feature of Delta. Databricks, however, does not support this (as of September 2025). This means that it’s not possible to alter the data via Databricks (e.g. to run `OPTIMIZE` or to delete PII)._

_² The lake must be in the same cloud as the pipeline._

_³ Iceberg+Databricks combination is currently not supported for GCP pipelines._

## What you will need

Connecting to a destination always involves configuring cloud resources and granting permissions. It's a good idea to make sure you have sufficient priviliges before you begin the setup process.

> **Tip:** The list below is just a heads up. The Snowplow Console will guide you through the exact steps to set up the integration.

Keep in mind that you will need to be able to do a few things.

**Batch-based (AWS):**

- Provide a Databricks cluster along with its URL

- Specify the Unity catalog name and schema name

- Create an access token with the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `CAN USE` on the SQL warehouse

**Via Delta Lake (Azure, GCP):**

See [Delta Lake](/docs/destinations/warehouses-lakes/delta/).

**Via Iceberg (AWS):**

See [Iceberg](/docs/destinations/warehouses-lakes/iceberg/).

**Streaming:**

- Create an S3 or GCS bucket or ADLS storage container, located in the same cloud and region as your Databricks instance

- Create a storage credential to allow Databricks to access the bucket or container

- Create an external location and a volume within Databricks pointing to the above

- Provide a Databricks SQL warehouse URL, Unity catalog name and schema name

- Create a service principal and grant the following permissions:

  - `USE CATALOG` on the catalog
  - `USE SCHEMA` and `CREATE TABLE` on the schema
  - `READ VOLUME` and `WRITE VOLUME` on the volume
  - `CAN USE` on the SQL warehouse (for testing the connection and monitoring, e.g. as part of the [Data Quality Dashboard](/docs/monitoring/#data-quality-dashboard))

Note that Lakeflow features require a Premium Databricks account. You might also need Databricks metastore admin privileges for some of the steps.

***

## Getting started

You can add a Databricks destination through the Snowplow Console.

**Batch-based (AWS):**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Databricks**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

**Via Delta Lake (Azure, GCP):**

Follow the instructions for [Delta Lake](/docs/destinations/warehouses-lakes/delta/#getting-started).

Then create an external table in Databricks pointing to the Delta Lake location.

**Via Iceberg (AWS):**

Follow the instructions for [Iceberg](/docs/destinations/warehouses-lakes/iceberg/#getting-started).

**Streaming:**

(For self-hosted customers, please refer to the [Loader API reference](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) instead.)

### Step 1: Create a connection

1. In Console, navigate to **Settings** > **Connections**
2. Select **Set up connection**
3. Choose **Event loading**, then **Databricks Streaming**
4. Follow the steps to provide all the necessary values
5. Click **Complete setup** to create the connection

### Step 2: Create a loader

1. In Console, navigate to **Destinations** > **Warehouses and lakes**. Click **Add destination** and select **Databricks**
2. **Select a pipeline**: choose the pipeline where you want to deploy the loader.
3. **Select your connection**: choose the connection you configured in step 1.
4. **Select the type of events**: enriched events or failed events
5. Click **Continue** to deploy the loader

You can review active destinations and loaders by navigating to **Destinations** > **Warehouses and lakes**.

Once the loader is up and running, click on the “...” button in the **Loaders** table and select **Databricks setup instructions**. Follow the outlined steps to create a Lakeflow pipeline within Databricks.

***

## How loading works

The Snowplow data loading process is engineered for large volumes of data. In addition, our loader applications ensure the best representation of Snowplow events. That includes automatically adjusting the tables to account for your custom data, whether it's new event types or new fields.

**Batch-based (AWS):**

For more details on the loading flow, see the [RDB Loader](/docs/api-reference/loaders-storage-targets/snowplow-rdb-loader/) reference page, where you will find additional information and diagrams.

**Via Delta Lake (Azure, GCP):**

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

**Via Iceberg (AWS):**

For more details on the loading flow, see the [Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/) reference page, where you will find additional information and diagrams.

**Streaming:**

For more details on the loading flow, see the [Databricks Streaming Loader](/docs/api-reference/loaders-storage-targets/databricks-streaming-loader/) reference page, where you will find additional information and diagrams.

***

## Snowplow data format in Databricks

All events are loaded into a single table (`events`).

There are dedicated columns for [atomic fields](/docs/fundamentals/canonical-event/), such as `app_id`, `user_id` and so on:

| app\_id | collector\_tstamp       | ... | event\_id                            | ... | user\_id                             | ... |
| ------- | ----------------------- | --- | ------------------------------------ | --- | ------------------------------------ | --- |
| website | 2025-05-06 12:30:05.123 | ... | c6ef3124-b53a-4b13-a233-0088f79dcbcb | ... | c94f860b-1266-4dad-ae57-3a36a414a521 | ... |

Snowplow data also includes customizable [self-describing events](/docs/fundamentals/events/#self-describing-events) and [entities](/docs/fundamentals/entities/). These use [schemas](/docs/fundamentals/schemas/) to define which fields should be present, and of what type (e.g. string, number).

For self-describing events and entities, there are additional columns, like so:

| app\_id | ... | unstruct\_event\_com\_acme\_button\_press\_1            | contexts\_com\_acme\_product\_1                                  |
| ------- | --- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| website | ... | data for your custom `button_press` event (as `STRUCT`) | data for your custom `product` entities (as `ARRAY` of `STRUCT`) |

Note:

- "unstruct\[ured] event" and "context" are the legacy terms for self-describing events and entities, respectively
- the `_1` suffix represents the major version of the schema (e.g. `1-x-y`)

You can learn more [in the API reference section](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/).

> **Tip:** Check this [guide on querying](/docs/destinations/warehouses-lakes/querying-data/?warehouse=databricks) Snowplow data.
