Running the models on data lakehouses
Running the models on data lakes or lakehouses (using external tables in a warehouse to read directly from a lake) is currently in Early Release state and is not fully supported. Certain features may not work as expected and errors are more likely to occur. Please use this approach at your own risk and raise any issues you find with us.
If you are using the lake loaders to load your data into a lake storage option, it may be possible to use our data models. In general in this section of the docs we are not going to detail which warehouses support which file formats, or how to set up the respective tables in each warehouse - please see the docs for your appropriate warehouse to see what file formats they support.
Databricks
At time of writing, delta is the preferred file format for Databricks external tables. If you create an external table from this lake format in Databricks, you should be able to run the models without any further changes required by simply pointing the model at this table.
Snowflake
At time of writing, Iceberg is the preferred file format for Snowflake iceberg tables. If you wish to use our models with this, currently only the Unified Digital package supports this, by setting the snowplow__snowflake_lakeloader variable to true.
Note that compared to the other loaders for Snowflake, that field names in Self-describing events and Entities are converted to snake_case format (the other loaders retain the format used in the schema, often camelCase). You will need to adjust other variables and inputs accordingly compared to what you may find in the docs.
Spark
Currently using spark directly as a compute engine is not supported for our packages.
Redshift (spectrum)
Currently using Redshift Spectrum tables is not supported for our packages due to limitations with the platform.
BigQuery on GCS
Currently using GCS/BigQuery external tables is not tested but may work, please let us know your experience if you try this.