Managing Secrets in Digdag

Its obvious that we need to use secrets or passwords in Digdag whether to connect to a database or to an external API. It's not a good idea to embed secrets in code or in .dig file. They usually get checked into repositories which is not a good security practice. It's also difficult to deploy to various environments where secrets change. Digdag gives an ability to manage secrets. Its simple and easy to use.

Managing Secrets in Digdag

Setup your Digdag server

Secrets are encrypted when they are stored on Digdag server. A secret encryption key must be provided for this encryption to happen. The encryption key must be a valid 128-bit AES key, base64 encoded. Use the following to generate yours.

echo -n 'any16bytesphrase' | openssl base64
YW55MTZieXRlc3BocmFzZQ==

Once you generate add the following to your server.properties along with other server config properties.

digdag.secret-encryption-key = YW55MTZieXRlc3BocmFzZQ==

Then start the server.

digdag server --config server.properties

Configure secret in your dig file

You can access the any under this digdag project using the format ${secret:key}. Here I am exposing the secret as environment (_env) variable which can be used inside your workflow.

timezone: UTC

_export:
  py: 
    python: /root/.virtualenvs/hopcoms/bin/python

schedule:
  daily>: 05:00:00

+daily_data_scraper:
  py>: hopcoms_daily.HopcomsDaily.daily
  _env:
    hopcoms_db_full_url: ${secret:hopcoms_db_full_url}

Use the secret in your workflow

Since in the above step we are setting the hopcoms_db_full_url as an environment variable. Its very straight forward to pick it up in python or any other language.

#part of my workflow showing how I use
import couchdb
from BeautifulSoup import BeautifulSoup
import digdag
import os

class HopcomsDaily(object):
	def daily(self):
		#READ THESE FROM CONFIG
		db_full_url = str(os.environ.get("hopcoms_db_full_url"))
		couch = couchdb.Server(db_full_url)

Add secrets to your digdag server

For the Digdag to pick up the secret to be used by a workflow, we need to set them in the Digdag server. Secrets are stored at the project level. So you need to send the project name along with the setting. A simple setting is straight forward

#example setting
digdag secrets --project <project> --set <secret_key> --config server.properties

#setting hopcoms_db_full_url
digdag secrets --project hopcoms --set hopcoms_db_full_url --config server.properties

#list the secrets
digdag secrets --project hopcoms --config server.properties

Add secrets to your digdag local

When you are running locally using digdag run . It will use the local secrets management. Hence while setting send the flag --local. But usage inside your workflow will remain the same.

#example
digdag secrets --project <project> --set <secret_key> --local

#setting hopcoms_db_full_url
digdag secrets --project hopcoms --set hopcoms_db_full_url --local

#list the secrets
digdag secrets --project hopcoms --local

Errors

If your server throws error: Failed to set project secret: Internal Server Error (web application) when you try to set a secret. It could be because digdag.secret-encryption-key is not set. Update that and restart your Digdag server. Try to set the secret now.