# Open Table Format Lake Loader
> Load Snowplow events to data lakes using Delta or Iceberg table formats on S3, GCS, or Azure ADLS Gen2.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/lake-loader/
> Documentation index: https://docs.snowplow.io/llms.txt

The Lake Loader is an application that loads Snowplow events to a cloud storage bucket using Open Table Formats.

> **Info: Open Table Formats**
>
> The Lake Loader supports the two major Open Table Formats: [Delta](https://delta.io/) and [Iceberg](https://iceberg.apache.org/).
>
> For Iceberg tables, the loader supports [AWS Glue](https://docs.aws.amazon.com/glue/) and [Iceberg REST](https://iceberg.apache.org/rest-catalog-spec/) as catalogs. The REST catalog integration has been tested with Snowflake Open Catalog (Apache Polaris), Snowflake Horizon Catalog, and Databricks Unity Catalog.

The cloud you deploy the loader in determines the stream it reads from. Since 0.15.0, the loader can write to S3, GCS, or ADLS Gen2 from any cloud.

**AWS:**

On AWS the Lake Loader continually pulls events from Kinesis and writes to S3.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kinesis stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["S3"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any AWS VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-aws:0.15.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-aws:0.15.0 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**GCP:**

On GCP the Lake Loader continually pulls events from Pub/Sub and writes to GCS.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Pub/Sub stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["GCS"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any GCP VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-gcp:0.15.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-gcp:0.15.0 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

**Azure:**

On Azure the Lake Loader continually pulls events from Kafka and writes to ADLS Gen 2.

```mermaid
flowchart LR
stream[["<b>Enriched Events</b>
(Kafka stream)"]]
loader{{"<b>Lake Loader</b>"}}
subgraph bucket ["ADLS Gen 2"]
  table[("<b>Events table</b>")]
end
stream-->loader-->bucket
```

The Lake Loader is published as a Docker image which you can run on any Azure VM. You do not need a Spark cluster to run this loader.

```bash
docker pull snowplow/lake-loader-azure:0.15.0
```

To run the loader, mount your config files into the docker image, and then provide the file paths on the command line:

```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/lake-loader-azure:0.15.0 \
--config=/myconfig/loader.hocon \
--iglu-config=/myconfig/iglu.hocon
```

For some output formats, you need to pull a slightly different tag to get a compatible docker image. The [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) page explains when this is needed.

***

## Configuring the loader

The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.

The simplest possible config file just needs a description of your pipeline inputs and outputs:

**AWS:**

```hocon
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.aws.minimal.hocon&#xA;>)

**GCP:**

```hocon
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.gcp.minimal.hocon&#xA;>)

**Azure:**

```hocon
loading...
```

[View on GitHub](<https://github.com/snowplow-incubator/snowplow-lake-loader/blob/main/config/config.azure.minimal.hocon&#xA;>)

***

See the [configuration reference](/docs/api-reference/loaders-storage-targets/lake-loader/configuration-reference/) for all possible configuration parameters.

### Windowing

"Windowing" is an important config setting, which controls how often the Lake Loader commits a batch of events to the data lake. If you adjust this config setting, you should be aware that data lake queries are most efficient when the size of the parquet files in the lake are relatively large.

- If you set this to a **low** value, the loader will write events to the lake more frequently, reducing latency. However, the output parquet files will be smaller, which will make querying the data less efficient.
- Conversely, if you set this to a **high** value, the loader will generate bigger output parquet files, which are efficient for queries — at the cost of events arriving to the lake with more delay.

The default setting is `5 minutes`. For moderate to high volumes, this value strikes a nice balance between the need for large output parquet files and the need for reasonably low latency data.

```hocon
windowing: 5 minutes
```

If you tune this setting correctly, then your lake can support efficient analytic queries without the need to run an `OPTIMIZE` job on the files.

### Iglu

The Lake Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.

## Monitor the loader

The Lake Loader reports metrics covering event counts, latency, the state of the table after each commit, and the disk and memory used by its internal Spark instance. It can send them to StatsD, to Prometheus, or to both.

See [Monitoring the Lake Loader](/docs/api-reference/loaders-storage-targets/lake-loader/monitoring/) for the full list of metrics and for how to configure each protocol.

**Telemetry notice**

By default, Snowplow collects telemetry data for Lake Loader (since version 0.1.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).

This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).

If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.

If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.

See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.
