Setting up alerts in Digdag for slow or delayed workflow

You can set up a failure alert task _error in Digdag to alert you when a workflow fails. But sometimes you want to get an alert even if the task runs successfully but takes more time than expected. For this, you can use sla feature.

You can set up alerts based on time or duration. Below is an example where you get a notification if the whole workflow takes more than 59s. It doesn't fail the workflow but sends the alert so that you can take appropriate action.

timezone: "UTC+05:30"

sla:
  # triggers this task after 59 seconds
  # SLA duration option needs to be HH:MM or HH:MM:SS format
  duration: 00:00:59
  alert: true
  fail: false
  +notice:
    sh>: echo "********************Late"

+steps:
  +step1:
    sh>: echo "********************hello world local"

  +step2:
    wait>: 120s

  +step3:
    sh>: echo "******************** Complete"

_error:
  sh>: echo "******************** Fail"

On running output looks like this

Let's say you want to fail if the job takes too much time. Then you can enable the fail flag too. This won't stop the workflow but will also trigger the _error task.

timezone: "UTC+05:30"

sla:
  # triggers this task after 59 seconds
  # SLA duration option needs to be HH:MM or HH:MM:SS format
  duration: 00:00:59
  alert: true
  fail: true
  +notice:
    sh>: echo "********************Late"

+steps:
  +step1:
    sh>: echo "********************hello world local"

  +step2:
    wait>: 120s

  +step3:
    sh>: echo "******************** Complete"

_error:
  sh>: echo "******************** Fail"

output looks like this

You can use the SLA feature creatively to perform tasks like cleaning up, alerting support teams, pausing some other jobs, etc.


You can read this blog using RSS Feed. But if you are the person who loves getting emails, then you can join my readers by signing up.

Join 2,223 other subscribers

2 Responses

  1. May 24, 2023

    […] Previous Post Next Post → Setting up alerts in Digdag for slow or delayed workflow https://thejeshgn.com/2023/05/23/setting-up-alerts-in-digdag-for-slow-or-delayed-workflow/ via @thej Posted on May 24, 2023 by @thej in […]

  2. May 26, 2023

    […] wrote a blog post on Digdag about setting up SLAs to monitor the workflows. I am still surprised at how many features Digdag supports and […]