For AI agents: a Markdown version of this page is at https://docs.snowplow.io/docs/monitoring/recovering-failed-events/manual/previous-versions.md. The index of all documentation pages is at https://docs.snowplow.io/llms.txt, and the index for this section is at https://docs.snowplow.io/docs/cdi/llms.txt.
Run event recovery 0.7.x and earlier on AWS
This page describes how to recover failed events on AWS with version 0.7.2 and earlier, using either the Spark job or the Flink job. It's kept for reference. Neither setup is supported.
Upgrade to the current Flink job. See the 0.8.0 upgrade guide.
Version 0.7.2 is the last release that includes the Spark job. The Spark job was deprecated in 0.6.0 because its Kinesis integration isn't as reliable as the Flink job's.
Run the Spark job
The Spark job reads failed events from S3, and writes the recovered events to Kinesis and the failed and unrecoverable events to S3. It runs on a transient EMR cluster, using dataflow-runner and the 0.7.2 Spark templates spark-cluster.json.tmpl and spark-playbook.json.tmpl.
Download both templates and run:
dataflow-runner run-transient \
--emr-playbook spark-playbook.json.tmpl \
--emr-config spark-cluster.json.tmpl \
--vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY
Where:
BUCKET_INPUT: the S3 bucket that holds the failed events to recoverBUCKET_INPUT_DIRECTORY: the directory inBUCKET_INPUTto read failed events fromAWS_REGION: the region to run the cluster in, for exampleeu-central-1AWS_SUBNET: the subnet to run the cluster inAWS_IAM_ROLE: the IAM role for the cluster nodesAWS_KEYPAIR: the EC2 key pair for the cluster nodesJOB_OWNER: a tag that marks the owner of the jobRECOVERY_VERSION: the application version, for example0.7.2RECOVERY_CONFIG: the Base64-encoded recovery configuration, as described in ConfigurationIGLU_RESOLVER: the Base64-encoded Iglu resolver configuration, as described in Run event recovery jobsKINESIS_OUTPUT: the Kinesis stream to write recovered events to
The playbook copies the job jar from s3://<bucket>/3-enrich/snowplow-event-recovery/. Copy snowplow-event-recovery-spark-0.7.2.jar there from the Snowplow hosted assets bucket for your region, or from the 0.7.2 GitHub release.
Monitor the Spark job
The Spark job reports summary counts through Spark's metrics system:
event-recovery.driver.summary.Recoveredevent-recovery.driver.summary.Failedevent-recovery.driver.summary.Unrecoverable
To choose where the metrics go, set the spark-metrics classification in the EMR cluster configuration, using the properties in Spark's metrics template.
For example, to expose the metrics over HTTP at http://${SPARK_HOST}:4040/metrics/json:
{
"classification": "spark-metrics",
"properties": {
"spark.metrics.namespace": "event-recovery",
"*.sink.servlet.class": "org.apache.spark.metrics.sink.MetricsServlet",
"*.sink.servlet.period": "1",
"*.sink.servlet.unit": "seconds",
"*.sink.servlet.path": "/metrics/json",
"master.sink.servlet.path": "/metrics/master/json",
"applications.sink.servlet.path": "metrics/applications/json",
"*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"
}
}
To print the metrics to the console log:
{
"classification": "spark-metrics",
"properties": {
"spark.metrics.namespace": "$name",
"*.sink.console.class": "org.apache.spark.metrics.sink.ConsoleSink",
"*.source.metrics.class": "org.apache.spark.metrics.source.Metrics"
}
}
Run the Flink job (0.7.x)
The 0.7.x Flink job runs on EMR 6.9 with Flink 1.15, using dataflow-runner and the 0.7.2 Flink templates flink-cluster.json.tmpl and flink-playbook.json.tmpl.
Download both templates and run:
dataflow-runner run-transient \
--emr-playbook flink-playbook.json.tmpl \
--emr-config flink-cluster.json.tmpl \
--vars bucket,$BUCKET_INPUT,region,$AWS_REGION,subnet,$AWS_SUBNET,role,$AWS_IAM_ROLE,keypair,$AWS_KEYPAIR,client,$JOB_OWNER,version,$RECOVERY_VERSION,config,$RECOVERY_CONFIG,resolver,$IGLU_RESOLVER,output,$KINESIS_OUTPUT,inputdir,$BUCKET_INPUT_DIRECTORY,interval,$INTERVAL
The variables are the same as for the Spark job, plus INTERVAL, the checkpointing interval. Start with 10 minutes and tune it to your job.
The cluster template runs a bootstrap action from s3://<bucket>/3-enrich/snowplow-event-recovery/snowplow-event-recovery-flink-<version>-bootstrap.sh, and the playbook copies the job jar from the same directory. Copy both there before you run the job. The bootstrap action installs the Flink S3 file system plugin and a CloudWatch agent.
The CloudWatch agent publishes the job's metrics to the snowplow/event-recovery namespace, including:
taskmanager_container_XXXX_XXX__Process_0_events_recovered: the number of recovered eventstaskmanager_container_XXXX_XXX__Process_0_events_failed: the number of failed recovery attemptstaskmanager_container_XXXX_XXX__Process_0_events_unrecoverable: the number of unrecoverable eventstaskmanager_container_XXXX_XXX__Process_0_numRecordsIn: the total number of events read
Flink's numRecordsOut metric for the Kinesis sink doesn't reflect the number of records written to the stream.