# BigQuery Streaming Loader
> Stream Snowplow events to BigQuery from Kinesis or Pub/Sub with real-time loading and schema evolution.
> Source: https://docs.snowplow.io/docs/api-reference/loaders-storage-targets/bigquery-loader/
> Documentation index: https://docs.snowplow.io/llms.txt
The BigQuery Streaming Loader is an application that loads Snowplow events to BigQuery.
**AWS:**
On AWS, the BigQuery Streaming Loader continually pulls events from Kinesis and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).
```mermaid
flowchart LR
stream[["Enriched Events
(Kinesis stream)"]]
loader{{"BigQuery Loader"}}
subgraph bigquery [BigQuery]
table[("Events table")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```
The BigQuery Loader is published as a Docker image which you can run on any AWS VM.
```bash
docker pull snowplow/bigquery-loader-kinesis:2.2.2
```
To run the loader, mount your config file into the docker image, and then provide the file path on the command line.
```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/bigquery-loader-kinesis:2.2.2 \
--config=/myconfig/loader.hocon \
--iglu-config /myconfig/iglu.hocon
```
Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.
**GCP:**
On GCP, the BigQuery Streaming Loader continually pulls events from Pub/Sub and writes to BigQuery using the [BigQuery Storage API](https://cloud.google.com/bigquery/docs/write-api).
```mermaid
flowchart LR
stream[["Enriched Events
(Pub/Sub stream)"]]
loader{{"BigQuery Loader"}}
subgraph bigquery [BigQuery]
table[("Events table")]
end
stream-->loader-->|BigQuery Storage API|bigquery
```
The BigQuery Loader is published as a Docker image which you can run on any GCP VM.
```bash
docker pull snowplow/bigquery-loader-pubsub:2.2.2
```
To run the loader, mount your config file into the docker image, and then provide the file path on the command line.
```bash
docker run \
--mount=type=bind,source=/path/to/myconfig,destination=/myconfig \
snowplow/bigquery-loader-pubsub:2.2.2 \
--config=/myconfig/loader.hocon \
--iglu-config /myconfig/iglu.hocon
```
Where `loader.hocon` is loader's [configuration file](/docs/api-reference/loaders-storage-targets/bigquery-loader/#configuring-the-loader) and `iglu.hocon` is [iglu resolver](/docs/api-reference/iglu/iglu-resolver/) configuration.
***
> **Tip: Schemas in BigQuery**
>
> For more information on how events are stored in BigQuery, check the [mapping between Snowplow schemas and the corresponding BigQuery column types](/docs/api-reference/loaders-storage-targets/schemas-in-warehouse/?warehouse=bigquery).
## Configuring the loader
The loader config file is in HOCON format, and it allows configuring many different properties of how the loader runs.
The simplest possible config file just needs a description of your pipeline inputs and outputs:
**AWS:**
```hocon
loading...
```
[View on GitHub]()
**GCP:**
```hocon
loading...
```
[View on GitHub]()
***
See the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for all possible configuration parameters.
### Iglu
The BigQuery Loader requires an [Iglu resolver file](/docs/api-reference/iglu/iglu-resolver/) which describes the Iglu repositories that host your schemas. This should be the same Iglu configuration file that you used in the Enrichment process.
## Metrics
The BigQuery Loader can be configured to send the following custom metrics to a [StatsD](https://www.datadoghq.com/statsd-monitoring/) receiver:
| Metric | Definition |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events_good` | A count of events that are successfully written to BigQuery. |
| `events_bad` | A count of failed events that could not be loaded, and were instead sent to the bad output stream. |
| `latency_millis` | The time in milliseconds from when events are written to the source stream of events (i.e. by Enrich) until when they are read by the loader. |
| `e2e_latency_millis` | The end-to-end latency of the snowplow pipeline. The time in milliseconds from when an event was received by the collector, until it is written into BigQuery. |
See the `monitoring.metrics.statsd` options in the [configuration reference](/docs/api-reference/loaders-storage-targets/bigquery-loader/configuration-reference/) for how to configure the StatsD receiver.
**Telemetry notice**
By default, Snowplow collects telemetry data for BigQuery Loader (since version 2.0.0). Telemetry allows us to understand how our applications are used and helps us build a better product for our users (including you!).
This data is anonymous and minimal, and since our code is open source, you can inspect [what’s collected](https://raw.githubusercontent.com/snowplow/iglu-central/master/schemas/com.snowplowanalytics.oss/oss_context/jsonschema/1-0-1).
If you wish to help us further, you can optionally provide your email (or just a UUID) in the `telemetry.userProvidedId` configuration setting.
If you wish to disable telemetry, you can do so by setting `telemetry.disable` to `true`.
See our [telemetry principles](/docs/get-started/self-hosted/telemetry/) for more information.