Adding custom schemas

By default, the extension is able to see Iglu schemas for events and entities generated by the standard Snowplow tracker and plugin APIs, as they're publicly available on the public Iglu Central registry. The Iglu Central registry is a default configuration in all Snowplow Pipelines as a common base, so events validated in the Inspector against these schemas should also validate correctly in your pipeline.

For custom schemas that aren't publicly available in Iglu Central, you need to configure the extension to access your registries so it can access the schemas to perform validation.

If you don't log in or configure registries, the extension won't recognize or be able to validate any custom events/entities, though they will still display.

The Snowplow Inspector extension displaying a Self Describing Event called "example_event". It contains a property "example_field_1" with the value "abc". The event's schema displays as Unrecognized in any registries.

Snowplow customers with access to the Snowplow Console can log in using the button in the bottom-left of the extension, which will automatically discover schemas defined for any data structures set up in your development or production Iglu Server environments.

When installed, the extension also creates a local Iglu registry, which lets you iterate on schemas without having to deploy them to a external registry. You can create any number of additional local registries to help organize your work.

You can also configure the extension to access other private schema repositories (such as Iglu Servers or static repositories).

Manage schemas and registries#

To access the full list of registries configured and find which schemas are accessible, choose ... > Manage Schemas.

Screenshot of the Snowplow Inspector's "More Options" menu open and the "Manage Schemas" option selected.

The extension will connect to your registries and request a listing of all available schemas.

Screenshot of the Snowplow Inspector's "Manage Schemas" page with the default Local and Iglu Central registries configured.

At the top is a search for filtering the shown schemas, controls for managing registries and local schemas, and then a list of your configured registries.

Underneath is a tree listing of all schemas configured, grouped by vendor, name, format, and version. Each version has a list of the registries that contained it.

Clicking a registry will toggle selecting the registry. The schema list below will filter to only show schemas within selected registries. If there are no registries selected, schemas from all registries will display. Any search terms will also only search schemas from the selected registries.

Using the Registries... button you can:

Add more custom registries
Import a list of registries from an existing Iglu Resolver configuration

If you have some registries selected, this button will also allow you to:

Edit existing registries you've previously configured
Remove configured registries so they aren't accessible to the extension any more

Selecting a local registry will activate the Schemas... button and allow you to manage the schemas it contains.

You can clear the search and registry selections to see all schemas again using the Clear Filters button.

Add custom registries#

You can add or edit different types of registry:

Local
Data structures API
Iglu Server
Static

Configuring a registry requires a name and one of these types. You may also optionally include a priority, and list of vendor prefixes. Each type of registry may require additional configuration like API keys to function.

Once configured, schemas from the registry load in the Manage Schemas view, and events using those schemas get validated against them.

Once the extension can query your private Iglu repositories it will be able to tell you in real time whether your events and entities are failing validation from entirely within the browser.

Here's an example of an event passing validation:

The Snowplow Inspector extension displaying a Self Describing Event called "example_event". The event payload contains a "example_field_1" property with the value "abc". The extension says the event is valid against a schema from the Local Registry, signalled by a checkmark.

Here's an example of an event failing validation. The failure reason is that the schema defines example_field_1 as "string" type, meaning it can't be null.

The Snowplow Inspector extension displaying a Self Describing Event called "example_event". The event payload contains a "example_field_1" property with a null value. The extension says the event is invalid against a schema from the Local Registry, signalled by a red cross.

Hovering over the registry displays text describing which aspect of the payload failed the validation, which is also available in the Errors tab. Find the Errors tab next to the Data, JSON, and Schema tabs within the event details pane. Clicking the registry copies this error to your clipboard.

Local#

Local registries are often used for testing changes of schemas still in development, or quickly including a schema when you don't have the full access required to configure a registry. The extension UI lets you manage the schemas, which it stores on your local machine.

The only required configuration is a name, and the schemas themselves.

You manage the schemas by selecting the registry and using the Schemas... button. You can manually edit the JSON in the extension, or bulk import JSON files from a directory on your machine.

Data structures API#

Data structures API registries use the data structures API included in Console.

Most users can just log in to Console via the extension and it will import registries for any Snowplow Console organizations your account has access to.

If you need to provide access for someone that doesn't have a Console account, they can manually configure one using a Console API key.

In order to function, the extension requires:

Organization ID: find this under Settings > Manage organization in Console
API Key ID: go to Settings > Manage organization in Console to manage your API keys
API Key: go to Settings > Manage organization in Console to manage your API keys

Iglu Server#

Iglu Server is a more full-featured dedicated service for hosting server that's more flexible than static registries.

You may need to add one of these registries if you are accessing the registry built into Snowplow Mini.

To authenticate with your server, the extension will require:

Iglu API endpoint: this is the base URL the extension will use when contacting the API. If you include a path component, the API request will be api/* relative to this path; you may need to add or remove trailing slashes if the API isn't hosted at the root.
Iglu API key: see API keys and the authentication service for instructions on generating an Iglu API key.

When logged into Console, these details should be available in Manage organization > API keys for utilities.

Static#

Most other registries, such as those hosted as websites or via S3 or GCS buckets will use this static type.

If you can access the schemas directly via a browser, this is probably the right choice.

Static registries managed with igluctl version 8 or newer will include a manifest file listing all the schemas in a registry, e.g. the Iglu Central manifest.

If the registry doesn't include a manifest listing the contained schemas, the extension will attempt to request all referenced schemas from static registries to see if it's included. In this case the registries will also not have their schemas listed in Manage Schemas - unless they're referenced by tracked events since you last opened the extension.

To connect to your static registry, the extension will need a base URI to use when requesting schemas. The extension will request schemas relative to this base, expecting a schemas/{vendor}/{name}/{format/{version} format.

Optionally you can also provide a manifest URI to use for the schema manifest file that lists the schemas contained in the registry. If not provided, defaults to schemas, to match the igluctl default. The manifest file is also known as a "schema list", as described in the release notes.