For AI agents: a Markdown version of this page is at https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/upgrade-guides/0-8-0-upgrade-guide.md. The index of all documentation pages is at https://docs.snowplow.io/llms.txt, and the index for this section is at https://docs.snowplow.io/docs/cdi/llms.txt.
Event recovery 0.8.0 upgrade guide
Version 0.8.0 makes the Flink job the only way to recover failed events on AWS, and moves it to a current EMR release. This guide covers what you need to change if you run recoveries with the Spark job, or with the Flink job from version 0.7.x or earlier.
Recovery configurations don't change. Configurations that work with 0.7.x work with 0.8.0.
Breaking changes
Version 0.8.0 changes how you run recoveries on AWS, and changes the license.
New license
Since version 0.8.0, event recovery uses the Snowplow Limited Use License (FAQ).
Spark job removed
The Spark job was deprecated in 0.6.0, and 0.8.0 removes it. Version 0.7.2 is the last release that includes it. To run recoveries on AWS, use the Flink job. See Migrate from the Spark job.
New EMR release for the Flink job
The Flink job runs on Flink 1.20, and needs EMR 7.13 with Java 11. The cluster configuration from 0.7.x doesn't work with 0.8.0. See Update your EMR configuration.
CloudWatch metrics removed
The 0.7.x bootstrap action installed a CloudWatch agent that published the job's metrics to the snowplow/event-recovery namespace. Version 0.8.0 has no bootstrap action, so these metrics are no longer published. Any dashboards or alarms based on them stop receiving data. See Monitoring for the summary and metrics that 0.8.0 provides.
Checkpointing no longer used
The 0.7.x templates turned on checkpointing with the interval variable. Checkpointing is not supported in 0.8.0, and the templates no longer use it. If a run fails, run it again from the start. See Understand delivery and reruns.
Migrate from the Spark job
Use the 0.8.0 dataflow-runner templates flink-cluster.json.tmpl and flink-playbook.json.tmpl in place of spark-cluster.json.tmpl and spark-playbook.json.tmpl. The templates take the same variables as the Spark templates, plus assetsBucket. See Run the job on EMR for the full command.
If you run the job with your own command line, rather than the templates, the Flink job accepts the Spark job's argument names --failedOutput, --unrecoverableOutput, --batchSize, and --directoryOutput. You need to replace spark-submit with flink run -m yarn-cluster -sae, and run it on a cluster that meets the cluster requirements.
Update any monitoring that relies on the Spark job's event-recovery.driver.summary metrics. See Monitoring for what the Flink job reports.
Update your EMR configuration
If you run the Flink job with the dataflow-runner templates, download the 0.8.0 templates and make these changes to your run-transient command:
- Add the
assetsBucketvariable, set to the Snowplow hosted assets bucket for your region, for examplesnowplow-hosted-assets-eu-central-1. In 0.7.x, the playbook copied the jar from your ownbucket. - Remove the
intervalvariable - Set
versionto0.8.0
If you maintain your own cluster configuration, compare it with the templates. The main differences from 0.7.x are:
| Setting | 0.7.x | 0.8.0 |
|---|---|---|
| EMR release | 6.9.0 | 7.13.0 |
| Instance types | m4.large | m5.xlarge |
| Java | Java 11 | Java 11, set explicitly for Flink in flink-conf |
| Bootstrap action | Installs the S3 file system plugin and the CloudWatch agent | None |
flink run options | -m yarn-cluster | -m yarn-cluster -sae |
client.heartbeat.timeout | Not set | 3650d |
| Checkpointing | --checkpoint-interval and --checkpoint-dir | Not used |
Make sure you remove the 0.7.x bootstrap action. EMR 7.13's Flink already includes the S3 file system plugin that the bootstrap action installed, and with a second copy the job fails to start with Delegation token provider with service name s3-hadoop has multiple implementations.
See the cluster requirements for the full list.
Check permissions and inputs
Whether you're migrating from the Spark job or upgrading the Flink job, check these two things before your first 0.8.0 run.
Check IAM permissions
The Flink job needs kinesis:PutRecords on the output stream. If your cluster uses EMR_EC2_DefaultRole, it doesn't have this permission, and the job retries indefinitely without making progress. See Grant permissions for the full list.
Check input paths
The Flink job doesn't expand globs in --input. If your input path ends with *, replace it with the directory, for example s3a://bucket/path/ instead of s3a://bucket/path/*. The job reads directories recursively.
Other changes
These changes don't need any action for most users:
- Command line tool: the
runcommand needs Java 21 or later. The--enrichmentsoption only accepts enrichments that don't need external resources, and fails with an error for any others. - Beam job: the Docker image runs on Java 11. The job's arguments haven't changed.
- File output for the Flink job: the Flink job can write recovered events to files with
--directory-output, as the Spark job could with--directoryOutput. See Test a configuration with a dry run.