Web-Powered Workflows: Fetching and Running Digdag Workflows with Callbacks

In Digdag, the workflows are defined in a YAML file with an extension ".dig". Usually, as a developer, you write these workflow YAMLs. Workflows typically have a bunch of tasks defined, which get executed. Developers also write these. But there are other ways of adding tasks to workflows dynamically if that makes sense for your project. For example, you can use Digdag Python API to add sub-tasks.

Yet another way is to download the ".dig" file from a remote HTTP server and add it as a subtask. And then execute that. This can be done using http_call> operator.

This is useful if a web service or an app generates the ".dig" workflow file with a customized flow and embedded parameters based on the conditions of the web-app. This allows the workflow logic to be managed by the web app rather than Digdag. This setup is very useful in many scenarios.

Here is the main dig, that makes an http_call to get a workflow from the service. I am using many dummy header variables to authenticate as examples; you can use the ones that fit your service.

+main_task:
    http_call>: https://example.com/services/workflow/
    method: POST
    content_format: json
    content:
        workflow: "test_call.dig"
        session_id: "${session_uuid}"
        time: ${session_time}
    headers:
        - x-api-key: "dummy-my-api-key"
        - x-username: "dummy-my-user-name"
        - Authorization: "Bearer dummyvaluewelldone"
    timeout: 5

Here is the dig served by the above url. The child workflow also calls the workflow status endpoint to report error or report completion at the end. I use the workflow name and session_id to identify and map in the callback. It works very well.

+child_task:
    sh>: echo "all the work webservice wants me to do" 

+done:
    http>: https://webhook.site/a63e0414-b872-4889-8188-22799866c2f9
    method: POST
    content_format: json
    content:
        workflow: "test_call.dig"
        session_id: "${session_uuid}"
        status: "ok"
        time: ${session_time}
    headers:
        - x-api-key: "dummy-my-api-key"
        - x-username: "dummy-my-user-name"
        - Authorization: "Bearer dummyvaluewelldone"    

_error:
    http>: https://webhook.site/a63e0414-b872-4889-8188-22799866c2f9
    method: POST
    content_format: json
    content:
        workflow: "test_call.dig"
        session_id: "${session_uuid}"
        status: "failed"
        time: ${session_time}
    headers:
        - x-api-key: "dummy-my-api-key"
        - x-username: "dummy-my-user-name"
        - Authorization: "Bearer dummyvaluewelldone"    

Given this is very much possible and is very stable. Where would you use it? In what use cases does this flow fit?