Operation

There are many ways to operate and run our packages, including backfilling or a-sync running of specific tables. The following pages contain guides for how to do this.

Required Privileges

In addition to the standard privileges required by dbt, our packages by default write to additional schemas beyond just your profile schema. If your connected user does not have create schema privileges, you will need to ensure that the following schemas exist in your warehouse and the user can create tables in them:

<profile_schema>_derived
<profile_schema>_scratch
<profile_schema>_snowplow_manifest

Alternatively, you can override the output schemas our models write to, see the relevant package configuration page for how to do this.

Snowflake
BigQuery
Databricks
Redshift
Postgres

  grant create schema on database <database_name> to role <role_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to role <role_name>;
   grant usage on schema <profile_schema>_scratch to role <role_name>;
   grant usage on schema <profile_schema>_manifest to role <role_name>;

For more information, please refer to the Official Guide on setting up permissions.

Please refer to the Official Guide on setting up permissions.

   -- user with "use catalog" privilege on the catalog
  grant create schema on catalog <catalog_name> to <principal_name>

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;

For more options (e.g.: granting to service principal, or group instead of users), please refer to the Official Guide on setting up permissions.

   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;

For more options (e.g.: granting to role, or group instead of users), please refer to the Official Guide on setting up permissions.

   -- someone with superuser access
   create schema authorization <user_name>;

   --alternatively
   create schema <profile_schema>_derived;
   create schema <profile_schema>_scratch;
   create schema <profile_schema>_manifest;
   grant usage on schema <profile_schema>_derived to <user_name>;
   grant usage on schema <profile_schema>_scratch to <user_name>;
   grant usage on schema <profile_schema>_manifest to <user_name>;

For more information, please refer to the Official Guide on setting up permissions.

📄️ Backfilling

Steps for backfilling the data models

📄️ Debugging data modeling issues

Guide for debugging data modeling issues

📄️ Full or Partial Refreshes

Complete refresh of Snowplow package

📄️ Model Selection

YAML Selectors

📄️ Disabling Models

How to disable a model in our packages

📄️ Tests

Tests are a useful feature in dbt to ensure that your data meets some expectations of it, such as certain fields should not be null. We provide a suite of tests with our packages to help test the output tables for any issues, however you could and should add your own as no one knows the required quality checks of your data better than you.

📄️ Running on Data Lakes

How to run our models on lakehouses

📄️ Asynchronous Runs

You may wish to run the modules asynchronously, for instance run the views module hourly but the sessions and users modules daily. You would assume this could be achieved using e.g.:

Required Privileges​