Setting up alerts in Digdag for slow or delayed workflow
You can set up a failure alert task _error in Digdag to alert you when a workflow fails. But sometimes you want to get an alert even if the task runs successfully but takes more time than expected. For this, you can use sla feature.
You can set up alerts based on time or duration. Below is an example where you get a notification if the whole workflow takes more than 59s. It doesn't fail the workflow but sends the alert so that you can take appropriate action.
timezone: "UTC+05:30"
sla:
# triggers this task after 59 seconds
# SLA duration option needs to be HH:MM or HH:MM:SS format
duration: 00:00:59
alert: true
fail: false
+notice:
sh>: echo "********************Late"
+steps:
+step1:
sh>: echo "********************hello world local"
+step2:
wait>: 120s
+step3:
sh>: echo "******************** Complete"
_error:
sh>: echo "******************** Fail"
On running output looks like this
Let's say you want to fail if the job takes too much time. Then you can enable the fail flag too. This won't stop the workflow but will also trigger the _error task.
timezone: "UTC+05:30"
sla:
# triggers this task after 59 seconds
# SLA duration option needs to be HH:MM or HH:MM:SS format
duration: 00:00:59
alert: true
fail: true
+notice:
sh>: echo "********************Late"
+steps:
+step1:
sh>: echo "********************hello world local"
+step2:
wait>: 120s
+step3:
sh>: echo "******************** Complete"
_error:
sh>: echo "******************** Fail"
output looks like this
You can use the SLA feature creatively to perform tasks like cleaning up, alerting support teams, pausing some other jobs, etc.
2 Responses
[…] Previous Post Next Post → Setting up alerts in Digdag for slow or delayed workflow https://thejeshgn.com/2023/05/23/setting-up-alerts-in-digdag-for-slow-or-delayed-workflow/ via @thej Posted on May 24, 2023 by @thej in […]
[…] wrote a blog post on Digdag about setting up SLAs to monitor the workflows. I am still surprised at how many features Digdag supports and […]