Operation
There are many ways to operate and run our packages, including backfilling or a-sync running of specific tables. The following pages contain guides for how to do this.
Required Privilegesโ
In addition to the standard privileges required by dbt, our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have create schema
privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:
<profile_schema>_derived
<profile_schema>_scratch
<profile_schema>_snowplow_manifest
Alternatively, you can override the output schemas our models write to, see the relevant package configuration page for how to do this.
- Snowflake
- BigQuery
- Databricks
- Redshift
- Postgres
grant create schema on database <database_name> to role <role_name>;
--alternatively
create schema <profile_schema>_derived;
create schema <profile_schema>_scratch;
create schema <profile_schema>_manifest;
grant usage on schema <profile_schema>_derived to role <role_name>;
grant usage on schema <profile_schema>_scratch to role <role_name>;
grant usage on schema <profile_schema>_manifest to role <role_name>;
For more information, please refer to the Official Guide on setting up permissions.
Please refer to the Official Guide on setting up permissions.
-- user with "use catalog" privilege on the catalog
grant create schema on catalog <catalog_name> to <principal_name>
--alternatively
create schema <profile_schema>_derived;
create schema <profile_schema>_scratch;
create schema <profile_schema>_manifest;
grant usage on schema <profile_schema>_derived to <user_name>;
grant usage on schema <profile_schema>_scratch to <user_name>;
grant usage on schema <profile_schema>_manifest to <user_name>;
For more options (e.g.: granting to service principal, or group instead of users), please refer to the Official Guide on setting up permissions.
-- someone with superuser access
create schema authorization <user_name>;
--alternatively
create schema <profile_schema>_derived;
create schema <profile_schema>_scratch;
create schema <profile_schema>_manifest;
grant usage on schema <profile_schema>_derived to <user_name>;
grant usage on schema <profile_schema>_scratch to <user_name>;
grant usage on schema <profile_schema>_manifest to <user_name>;
For more options (e.g.: granting to role, or group instead of users), please refer to the Official Guide on setting up permissions.
-- someone with superuser access
create schema authorization <user_name>;
--alternatively
create schema <profile_schema>_derived;
create schema <profile_schema>_scratch;
create schema <profile_schema>_manifest;
grant usage on schema <profile_schema>_derived to <user_name>;
grant usage on schema <profile_schema>_scratch to <user_name>;
grant usage on schema <profile_schema>_manifest to <user_name>;
For more information, please refer to the Official Guide on setting up permissions.
๐๏ธ Backfilling
Steps for backfilling the data models
๐๏ธ Debugging data modeling issues
Guide for debugging data modeling issues
๐๏ธ Full or Partial Refreshes
Complete refresh of Snowplow package
๐๏ธ Model Selection
YAML Selectors
๐๏ธ Disabling Models
How to disable a model in our packages
๐๏ธ Tests
Tests are a useful feature in dbt to ensure that your data meets some expectations of it, such as certain fields should not be null. We provide a suite of tests with our packages to help test the output tables for any issues, however you could and should add your own as no one knows the required quality checks of your data better than you.
๐๏ธ Running on Data Lakes
How to run our models on lakehouses
๐๏ธ Asynchronous Runs
You may wish to run the modules asynchronously, for instance run the views module hourly but the sessions and users modules daily. You would assume this could be achieved using e.g.: