Configuration

SwitchTender is one binary with subcommands. This page lists every command, flag, and environment variable.

Licensed features

Almost everything on this page runs on Community, which needs no license. Seven features are licensed. There is no cap on how many approval policies an install holds. A flag or request that turns on a licensed feature is refused outright rather than quietly ignored, so an install never believes it has a control it does not have.

license status prints the tier this install runs and when a license lapses. Tiers and prices are at https://switchtender.com/pricing.

Environment variables

Every command that takes --db defaults to switchtender.db in the current directory. Only serve, init, demo, restore, and an import create a database there. The commands that work on an existing install, such as token, user, examples, backup, and the audit commands, refuse a SQLite path that does not exist and name the path they looked at. Opening it would create an empty database the server never reads, and the command would report success. An import that starts a new database says so, with the --db a server needs to read it.

init

Bootstraps a new deployment. It creates the database and the first admin account, writes an environment file, and optionally a systemd unit. Run it once, then start serve. Running it again, with --force to rewrite the environment file or after deleting it, leaves an admin account that already exists as it is, password included.

The admin password comes from SWITCHTENDER_ADMIN_PASSWORD, or is generated and printed once when that variable is unset.

serve

Runs the HTTP API, the in-process executor, the scheduler, the retention sweeper, and the web UI.

Retention windows accept a whole number of days with a d suffix, such as 30d, or Go duration syntax such as 720h.

Approval binding needs the producer identity

A server that loads the install's producer identity derives an approval binding key from it, held in memory and never written anywhere. It seals every run it creates or releases under that key and checks the seal before the run executes, so a run whose spec, approval fields, decision, or who decided was changed in the database after it was approved is refused rather than run. Every server of one install must load the same identity, from the identity directory or from SWITCHTENDER_AUDIT_KEY, since a run sealed under one key is refused under another. A relay control node checks each run before it hands it to a relay worker. A worker that shares the database holds no key, so the runs it claims are held only to the digests stored on them.

A server with no identity on a chain no install has signed holds no key. It still compares the digests stored on each run, and its startup warning says that approval binding is row-checked only, which a change written into the database together with matching digests passes. A server that finds the chain bound to an install whose key it does not hold refuses to start, as the backup page explains.

A server that loads an identity refuses to start while work an earlier release left carries no seal and the audit chain holds no adoption record, and it names the runs. Run adopt once to seal that work, then start the server. A server that loads an identity and finds nothing to adopt records the adoption itself at its first start, trusting no row. A start that finds sealed work and no adoption record is refused, since the record was removed.

AI providers

The advisory AI features run against one provider, chosen with --ai-provider: ollama for a local model, or anthropic and openai for a cloud model with SWITCHTENDER_AI_KEY. The provider is off until set, and no feature ever executes anything the provider suggests.

Cloud models see automation content: commands, playbook names, failed-run logs, and host drift. Because that content is security-adjacent, a model with strict safety classifiers can decline a benign request as a false positive. When the Anthropic model is a Fable or Mythos model, SwitchTender opts into server-side fallbacks, so a declined request is retried on claude-opus-4-8 in the same call and the feature keeps working. A Fable model also requires that the account keep 30-day data retention, or the API rejects every request.

desktop

Runs SwitchTender as a local desktop application. It serves on a private loopback port, stores its data in a per-user directory, and opens the web UI in the default browser. It takes no flags. Set SWITCHTENDER_DESKTOP_NO_BROWSER to skip opening a browser. See Desktop for packaging a macOS app or a Windows installer.

worker

Leases pending runs from the shared store and executes them. Point it and a server at the same database and they compete for work. Or start it with --server and it leases runs from the control node over the mesh relay, with no database access of its own.

worker key

switchtender worker key new --out FILE generates a delivery key for a relay worker pool. It writes the private key to --out, mode 0600, and never over an existing file, then prints the public key and its key id as one JSON line:

{"key_id":"f4e8a76a457da19cf3742de37986d439","delivery_key":"x25519:x9f9/kkB6Jq3Wqht/7TWzDGUeTRfbABrj1CO+8/6hHc=","file":"/etc/switchtender/delivery.key"}

switchtender worker key public FILE prints the same line for a key file that already exists, and refuses one another account can read. The file is a standard PKCS #8 X25519 private key, so openssl genpkey -algorithm X25519 -out FILE makes an equivalent one, and openssl pkey -in FILE -pubout -outform DER | tail -c 32 | base64 prints the base64 that follows x25519:.

ansible

Installs, lists, and removes the managed Ansible runtime: a Python virtual environment holding one pinned ansible-core release, installed with pip in hash-checking mode from a lock this binary carries. Runs use it when no --ansible-bin is configured. A python3 the release supports must be installed.

• ansible install [--version <release>] [--python <path>] [--wheels <dir>] [--index-url <url>] installs a release and makes it the one runs use. --version takes a release such as 2.21.4 or a minor version such as 2.21, and defaults to the newest supported release. --python names the interpreter to build with, and defaults to python3, then python3.N, on PATH. --wheels installs offline from a directory of wheels, relative to where the command runs, checked against the same hashes. --index-url downloads from a mirror instead of PyPI. pip's own environment variables and configuration files are not read. Installing an installed release verifies it and makes it the one runs use, reinstalling nothing.
• ansible list prints the installed releases and which one is current.
• ansible remove [--version <release>] removes one release, or every release and the directory when nothing else is in it.
• ansible lock [--version <release>] prints the requirements lock a release installs from.

Every ansible subcommand takes --dir for the runtime directory, which defaults to SWITCHTENDER_ANSIBLE_RUNTIME_DIR, then ansible/ in the data directory of --db, and --pretty for indented JSON.

token

Manages API tokens. A public bind on an empty database mints an initial admin token at startup.

• token new --name <label> [--user <username>] [--ttl <duration>] [--agent] mints a token, printed once. A zero TTL never expires.
• --user binds the token to an account, and the token carries that account's role. A token minted without --user is unscoped and acts as admin, so bind every token you hand to a person, a service, or an AI agent. See running an agent.
• --agent marks the token as held by an AI agent rather than a person. It caps the token at operator whatever role its account holds, so an agent can launch and propose work but can never manage identity, access, or secrets, and can never approve its own held run. Every action it takes is recorded in the chain as actor_type: agent with the account it acts for beside it, which is what actor_kind: agent policy rules match on. Every run such a token asks for waits for a person's approval unless a rule with effect: exempt covers it, as Agent runs are held by default describes. It requires --user, so the chain always records the human the agent acts for. Without one the command refuses. Mint every agent token with it: a token without --agent is indistinguishable from a person's in the record, and gets no agent hold.
• token list lists tokens without their secrets.
• token revoke <id> deletes a token.

All token subcommands take --db and the global --pretty flag for indented JSON.

user

Manages accounts with roles: admin, operator, and viewer.

• user new <username> --role <role> creates an account. The password comes from SWITCHTENDER_PASSWORD or a prompt, never an argument.
• user list lists accounts.
• user delete <id> deletes an account. Its tokens stop working.

All user subcommands take --db.

license

Shows or installs this install's license. No license is Community, which is complete in itself. What each tier covers is listed at https://switchtender.com/pricing.

• license status shows the tier this install runs and when a license lapses.
• license install <file> verifies a license file and installs it beside the database.

A license is read from the file SWITCHTENDER_LICENSE names, or from switchtender-license.json in the same directory as the database. A license this install cannot parse or that has lapsed reads as Community rather than failing the server, so an expiry never takes an install down.

assess

Reports what an automation export holds and what governing it would change, without writing anything and without a database. It reads an export the same way import does, then grades every template it found through the same risk and reversibility graders the product uses at run time, so a number in an assessment cannot disagree with what a run would later say.

switchtender assess awx awx-export.json
switchtender assess chef chef-nodes.json

Formats are awx, semaphore, chef, and puppet. An AWX-format export also covers Ansible Automation Platform, Tower, and Ascender.

An export saved as UTF-8 with a byte order mark, or as UTF-16 the way Windows PowerShell writes a file by default, is read the same as plain UTF-8, here, by import, and by the assessment page.

The output has three parts: what the import would create, what does not survive the move as it was, and what changes about how it is governed. Everything that does not come across is listed, one line each. The third part names the templates that cannot be undone, the ones carrying a destructive signal, the credentials more than one template shares, and the templates targeting no stored inventory.

Grades are a floor rather than a measurement. An Ansible template keeps its work in a playbook, and at assessment time that playbook is in a repository nothing has fetched, so reading one can only ever raise a grade. The report says how many templates that applies to rather than leaving the numbers looking more settled than they are.

import

Migrates from AWX, Semaphore, Chef, Puppet, Rundeck, Jenkins, or cron. Which objects each one carries across is in what each source brings over.

• import awx <export.json> [--awx-template-ids <list.json>] [--apply] brings projects, inventories static and dynamic, credential shells, job templates and workflows, surveys, and schedules. --awx-template-ids takes AWX's job template list, GET /api/v2/job_templates/ saved as JSON, repeatable for each page. A job template that accepted provisioning callbacks is then bound to its AWX id, so it also answers at its old AWX callback address. An export that carries each job template's id needs no list. See the AWX-compatible callback address.
• import semaphore <export.json> [--apply] brings the same kinds from a Semaphore export, apart from the dynamic inventory sources AWX alone carries.
• import chef <nodes.json> [--apply] brings one inventory of the fleet: every node a host, grouped by its chef_environment and by every role[...] in its run list, carrying the ohai facts that identify a machine, with ipaddress also set as ansible_host. Accepts an array of node documents, a single node, or an object keyed by node name. Cookbooks and recipes are not imported and the recipes seen are named in the report.
• import puppet <nodes.json|facts.json|nodes.txt> [--apply] brings one inventory of the fleet, grouped by environment, from a PuppetDB nodes query, a PuppetDB facts query, or the plain certname list puppet node list prints. Deactivated and expired nodes are left out and counted. Manifests and modules are not imported.
• import rundeck <jobs.yaml|project-archive.zip> [--inventory <name>] [--apply] brings templates, surveys, and schedules from either a job export or a project archive, told apart by content. An archive brings one project as well, but only when its source control configuration names a repository this can reach. Neither artifact carries a node definition, so no inventory is imported from either and --inventory names the hosts its jobs target.
• import jenkins <JENKINS_HOME|jobs-dir|job-dir|config.xml> [--inventory <name>] [--apply] brings templates, surveys, and schedules from freestyle jobs. Jenkins picks an agent by label, so --inventory names the machines.
• import cron <crontab-file> [--inventory <name>] [--system] [--apply] brings schedules alone, one per crontab line, each carrying its own one-step bash pipeline rather than a template. --system parses the six-field /etc/crontab form, whose user column sits before the command. That step runs on the SwitchTender host, not on the machine the crontab came from.

All seven take --db for the target database. Without --apply the command only reports what it would create. Rundeck, Jenkins, and cron import no credentials, so a job that needed a login needs one built by hand afterward.

audit

Audit trail tools.

• audit bundle emits the chain as a signed LoomSeal bundle. Anyone verifies it offline with the open loomseal verifier, or in a browser, without trusting the server that produced it.
• audit anchor has a public timestamp authority sign the current head, so a chain that has lost its tail no longer reaches its anchor.
• audit receipt <seq:link> redeems a receipt the server issued, proving that entry is still in the chain.
• audit report renders the period's change register as a self-contained HTML evidence report.
• audit run <id> emits one run's evidence dossier as a self-contained HTML document.

Flags on the subcommands:

• audit bundle --limit <n> carries only the newest N entries. The default carries the whole chain, and a chain past 25,000 entries, the most one bundle carries, is refused with the flags that would serve it.
• audit bundle --from-seq <seq> --to-seq <seq> carries the entries from one sequence through another, both included, at most 25,000 of them. It is how an older slice of a long chain is bundled. Either way the whole chain is verified first, and only the slice is held in memory.
• audit anchor --type <kind> picks the anchor: rfc3161 for a signed timestamp, or git or https for one checked by fetching it. --ref is the authority URL for rfc3161, otherwise the URL a verifier fetches. --tree anchors the Merkle root over the whole chain, which is the coordinate a sparse receipt proves membership against.
• audit report --from <when> --to <when> bounds the change register's period. --from defaults to ninety days before --to, and --to defaults to now and is exclusive.

receipt

Writes a signed receipt for one finished run, which a third party verifies offline with verify. A receipt is the chain segment from the request that created the run through the entry recording what it did, signed with this install's key and carrying any anchors that fix its position.

Publish the key fingerprint the command prints so a verifier can pin it.

verify

Verifies a receipt written by receipt. It trusts nothing this server says: it recomputes every chain link from the receipt's own claims, checks the signature covers the exact bytes, and confirms any anchors name an entry the receipt holds. It reads only the file, reaches no database and no network, and does not run the server, so a relying party can check a receipt on a machine that has never seen this install.

switchtender verify run.receipt --pubkey sha256:...

Pass --pubkey with the fingerprint the producer published to tie the result to a key obtained out of band. Without it the receipt is checked against the key it names, which proves it was not altered but not who signed it, and the verdict says so: INTACT, BUT UNIDENTIFIED rather than VERIFIED. A --pubkey given an empty value, which is what a failed key fetch leaves behind, is refused rather than treated as no pin.

An install enrolled with a witness carries the witness's countersignatures in its bundles and receipts. Pass --witness-pubkey with the key the witness publishes, in either the sha256 key id or the raw hex form, so only that witness's countersignatures are counted, and --witness with the statement fetched from the witness at GET /v1/installs/<key id>/attestation, where <key id> is the producer key id, so the file is held to the newest head the witness countersigned. A file that ends before that head, or a statement that holds nothing yet, is reported as not checkable and never reads VERIFIED. An empty --witness-pubkey is refused.

witness

Watches a server's span beat feed from outside it. A chain proves what it holds was not altered, but not that nothing was removed from the end, because the process running the chain also decides what gets written down. A witness on another machine remembers what the feed served, keeps that memory in a signed checkpoint, and raises a finding when a beat goes missing, an already-witnessed beat comes back rewritten, or the head regresses. Run it where the server's operator has no hand.

The server has to be writing beats for there to be anything to watch, and beats are off by default. Start the server with --span-cadence, for example --span-cadence 60s. Against a server with beats off, the witness reports empty_feed, and a --once run exits nonzero.

• witness serve watches many servers from one process and answers auditors with countersigned attestations.
• witness verify-attestation verifies an attestation offline against a pinned witness key.
• witness rotate-key retires the witness signing key and makes a new one current.
• witness enroll enrolls this install with a push witness that countersigns its chain.

witness serve

Runs the witness. Its flags are checked against that subcommand rather than witness itself. With --accept-enrollments it is also a push witness, the one the hosted witness page describes, and --enrollment-policy open runs it for installs on your own network.

--notify-slack takes a Slack incoming webhook URL, is repeatable, and is unset by default. The witness posts every finding there as a message that names its kind, its class (refusal or note), the producer key id of the install it is about or the watched server, and its detail. When the flag is not given, the URL in SWITCHTENDER_WITNESS_NOTIFY_SLACK is used, which keeps the URL out of the process list. Messages are sent in order from a queue of their own, so an install's push is never answered late because Slack is slow. A post that fails is tried once more after two seconds, and a second failure is logged with the URL masked. The findings record stays the durable copy either way.

witness rotate-key

Retires the witness signing key and makes a new one current. Stop the witness first and start it again after, since a running witness keeps signing with the key it loaded.

switchtender witness rotate-key --key-dir /var/lib/switchtender-witness-key \
  --state-dir /var/lib/switchtender-witness

The witness publishes every key it has signed with at /.well-known/loomseal.json and in every enrollment answer, each with its valid_from and, once retired, its retired time. After a rotation the list is signed by the new key and endorsed by every retired key the key directory still holds, so an enrolled install that holds an earlier key accepts the new list on that key's word. An attestation a key made before it retired verifies against the list for as long as anyone keeps it. The witness keeps reading its own memory signed by a retired key, for memory written before that key retired, and signs it with the new key at the next push it adopts. A retried push gets the bytes it was first answered with while the witness still holds the key that answered it.

--key-dir is required and names the directory holding the key, the same one witness serve runs with. --state-dir names the witness state directory and defaults to switchtender-witness. The key directory must not be the state directory, lie inside it, or hold it, because the state directory is copied off the host. The retired key stays in the key directory, readable only by the account that runs the witness. Back up the whole key directory after a rotation, since its key history is part of what the witness publishes. Deleting a retired key's file stops that key endorsing anything again, which is the step after a compromise, and an install that held only that key then pins the new key by hand. The command prints the retired key and the new current key as JSON.

witness enroll

Enrolls this install with a push witness. The server then sends every span beat to the witness with the tree root at that beat and a consistency proof from the root the witness last signed, and stores the countersignatures the witness answers with. It needs a license that includes the hosted witness: Enterprise, or Team with the add-on. Without one the command refuses and sends nothing.

switchtender witness enroll --db /var/lib/switchtender/switchtender.db \
  --witness https://witness.switchtender.com

The command prints the witness key id. Compare it with the id the witness publishes at /.well-known/loomseal.json and with the one the hosted witness page carries before trusting what it countersigns, or pass --witness-key to refuse any other. The enrollment is written to witness-enrollment.json beside the license file, so a backup and a restore carry both or neither. It holds no secret. An install whose key was changed under its install id through SWITCHTENDER_AUDIT_KEY is refused, since the witness binds an install id to the key it was born to. A new install started after its signing key was lost enrolls as a new record under its new key, beside the old install's record, which keeps its last countersigned head.

witness verify-attestation

Verifies one attestation offline, for the relying party. It reads the JSON, recomputes the signature, and prints the verdict with the signer named in both published forms: signed_by is the raw hex public key the attestation carries, and signed_by_key_id is the sha256 key id the witness prints at startup and serves from its API.

switchtender witness verify-attestation attestation.json --pubkey sha256:...

The verdict carries pinned. Without a pin, ok means only that the document is internally consistent, which an attestation a forger signed with their own key also is, so an unpinned verdict carries a note saying so. An empty --pubkey is refused.

Pass --pubkey with the key you pinned out of band. Either form is accepted, since the key id is what the witness tells operators to publish and the hex key is what the document carries. Without it the check proves only that the attestation is internally consistent, which a forger with their own key satisfies trivially.

mcp

Serves the Model Context Protocol over stdio, so an agent can list templates, propose a run, and read what happened. Every tool call is an ordinary authenticated API request carrying the token given here, so it passes the same authorization, the same approval policy, and the same fail-closed audit append as a request from a person. A run it proposes waits for a person's approval unless a rule with effect: exempt covers it, as agent runs are held by default describes. See Agents.

export SWITCHTENDER_MCP_TOKEN=swt_...
switchtender mcp --server https://switchtender.internal

The token is read from SWITCHTENDER_MCP_TOKEN, falling back to SWITCHTENDER_TOKEN. Prefer the environment variable to a flag, whose value is visible in the host's process list. The command refuses to start on an admin token. There is deliberately no approve tool, so an agent cannot release its own work, and no credential, account, token, grant, or policy tool, so it cannot widen its own reach.

demo

Seeds a fresh database with sample data and real runs, then serves it read-only, so a public instance is safe to expose. It needs ansible on the PATH to run the sample playbooks.

Seeding runs real playbooks and takes a couple of minutes, which is a visible gap if a public demo reseeds in place. The two flags split that work in half so it can happen off to the side:

switchtender demo --db next.db --seed-only     # build the next database, serving continues
mv next.db demo.db                             # swap it in
switchtender demo --db demo.db --no-seed       # serves the prepared data in under a second

A host wired this way reseeds without a visible outage, since the running instance keeps answering the whole time the replacement is being built.

examples

Seeds a handful of starter templates, so a first launch works on the spot rather than opening on an empty list. They use the Bash tool and print or read something local, needing no project, inventory, or credential. Delete them once you have your own.

Run it against the same database serve uses. It skips a template whose name is already present, so it is safe to run twice. Takes --db.

backup

Writes an encrypted, portable backup of the control-plane configuration and secrets. The whole file is sealed with the deployment encryption key, so it stays confidential and tamper-evident, and it restores into either the SQLite or the PostgreSQL backend. See Backup and restore.

Run history and the audit chain are not included. The audit chain has its own signed export, through audit bundle.

restore

Reads a backup and upserts its objects into the store by id. It needs the same encryption key the backup was written with, and it never deletes objects absent from the file.

adopt

Seals, once per install, the work a release with no approval binding key left behind, so a server of this release accepts it. Run it after every server and worker of the earlier release has stopped and before this release's serve first starts. Upgrading gives the order.

Adoption reads every run that has not finished, the finished steps a waiting workflow reads forward, and the decisions left in flight, and gives each the seal its kind is held to, from the row as it reads and the decision entries it names. A row it cannot seal, such as one whose decision entry is not on the chain, is listed with the reason and left unsealed, and a server refuses that row when it reaches it. Before the rows, adopt prints the chain's entry count, its newest anchor (type, sequence, and reference) or that it has none, and the install the chain is bound to or that this adoption binds it. Then it prints every row it seals, with the run id, the kind of seal, who asked for the run, who decided it, the decision entry, its spec digest, and its release entry, and a count of decision claims whose decisions are not on the chain yet. Compare the anchor with its external target, which is the git ref, the published head, or the witness, before answering. Adopt asks before it writes. --yes skips the question. Without it, adopt refuses to run where standard input is not a terminal.

--yes is for one run by hand. Never put adopt in an entrypoint, a chart hook, an init container, or a recurring job. A second adopt on the same install is refused and exits non-zero, which a script should expect.

It refuses, and writes nothing, in each of these cases. Each shows that keyed operation already began, so adopting again would seal whatever was written since as though the earlier release had left it.

• The audit chain does not verify from its first entry to its last, or no longer reaches an anchor recorded over it.
• Any run carries a seal, finished runs included.
• The chain holds a RELEASE entry or an adoption record.

An install refused for one of the last two reasons that has no copy to restore runs adopt with --fail-unsealed. It records the adoption with nothing adopted, so no row is trusted, and fails every unfinished run that carries no seal with the reason not adopted. A run that carries a seal is left for the server to check when it reaches it, and one it did not make is refused. A run still held under a lease, or with a decision or a cancel in flight, is listed and left as it is. It still refuses a chain that does not verify and an install whose adoption is already recorded.

Adopt appends one record to the audit chain, at /runs/adopted under an id the install's key fixes, before it writes any seal. A second adopt on the same install is refused and says when and by whom the first one ran. The rows it seals are the rows it showed, as they read when it showed them. A row changed while the list was on screen carries a seal over what was shown, which a server refuses, and a row created meanwhile is not sealed.

Adopt needs the install's producer identity, the same one serve loads, from the identity directory or SWITCHTENDER_AUDIT_KEY. An install with no identity has no key and nothing to adopt, and adopt says so. Adopt binds an unbound chain to the key it runs with and names where that key came from, so against a PostgreSQL DSN run it with the seed the servers will sign with in SWITCHTENDER_AUDIT_KEY. A server whose key names another install refuses to start.

version

Prints the SwitchTender version.

--verify fetches this version's published binary hashes and verifies the running executable.

help and completion

Both are the standard Cobra built-ins. help prints usage for any command, and completion emits a shell completion script for bash, zsh, fish, or PowerShell.

Output flags

JSON goes to stdout compact by default. Four commands take --pretty to indent it. It is not a global flag, so passing it elsewhere is an error rather than a no-op.

Separation between organizations is opt-in

Organizations, teams, and grants exist, but with --strict-grants off an object nobody has granted falls back to the caller's global role. On that default any operator may use any project, inventory, or credential in the install, whichever organization it belongs to. That is deliberate: a small team should not have to grant every object before anything works, and an upgrade should not lock people out of what they were already using.

It does mean an install with several organizations on it is not separated until --strict-grants is on. If you are running work for more than one team, more than one customer, or anything where one group must not reach another's credentials, turn it on and grant deliberately. Objects created before you do carry no grants, so plan to assign them.

A refusal from these rules says so: the body reads forbidden: this object requires a grant you do not hold, which is the same sentence whether the object is delegated elsewhere, ungranted under strict grants, or does not exist. It tells you a grant is what is missing rather than a role or an account, and nothing about the object itself.

What --strict-grants decides is only the default for an object nobody has granted. An object that carries a grant is access-controlled in both modes: writing a grant on it is what declares it so, and a caller who does not hold one is denied whether or not strict grants are on. Every listing answers the same rule the by-id read does, so a run you are refused by name is also absent from the run list, the fleet and drift views, the host pages, and the change log.

One consequence is worth knowing before you write your first grant. Install-wide figures that carry no per-row id are served whole only to a caller nothing is hidden from, because such a total is every other group's volume in one number. On an install with no grants at all that is everybody, which is the common case and is unaffected. It stays unaffected by a grant on a template or a worker queue, since neither can hide a run. What changes it is a grant on a project, an inventory, or a credential that the caller does not hold.

For a caller in that position each surface answers differently, and the difference is worth knowing before you point a dashboard at one:

The empty /metrics body is deliberate: a scraper reads a 5xx as the install being down and pages somebody. It does mean an alert written against a series going to zero will not fire for a scrape token that has become restricted, because the series is gone rather than zero. Scrape with a token that holds everything, or alert on the scrape's own staleness as well as its values.

Directory sign-in and existing accounts

An account records what created it: an administrator, or the directory that provisioned it. A directory identity signs in to an account only when that account is its own, so an identity provider asserting the username of a local administrator is refused rather than handed that administrator's account and role.

This matters most when the username does not come from a stable subject. With --jwt-username-claim email or a SAML username attribute pointing at an email, against an issuer that lets a user set their own address, matching on username alone was account takeover. OIDC has always refused the equivalent when the provider does not vouch for the address.

An account created before this was recorded carries no source, so it cannot be told apart from a local one of the same name. Those are still signed in to, because refusing them would lock out every directory user on upgrade, and each is logged so an administrator can set the source and remove the ambiguity.

Attention thresholds

The overview's Needs attention panel, GET /v1/attention, sorts waiting work into four answers to what is stopping it: a lost worker, no worker for its queue, an approval, or a block behind another run. Five thresholds decide when work shows as blocked and when each answer alerts. An alert is listed by the doctor and told, once per condition, to the server-wide chat channels, webhooks, ntfy, and email, to a run's own targets of those kinds, and to the named targets attached for the attention event, on every kind including PagerDuty. The API reference describes the answer.

Each value is a Go duration such as 90s, 15m, or 4h, or off. Zero is refused rather than read as off. The file states the organization's defaults and may override them for one organization on the install, one queue, or one template, and the most specific wins: a template over a queue, a queue over an organization, an organization over the defaults. Each override replaces only what it states. The default queue is written as "".

defaults:
  alert_no_worker: 10m
  alert_approval: 8h
orgs:
  org_ab12cd34ef56:
    alert_approval: 2h
queues:
  prod:
    alert_no_worker: 2m
    blocked_after: 5m
  "":
    alert_blocked: 30m
templates:
  tpl_0123456789ab:
    alert_approval: 30m

An unknown key, a value that is not a duration, and blocked_after: off are refused when the server starts, so a typo never silently leaves an alert on its default or off.

Confining relay workers to their queues

A relay worker runs in a segment the control node cannot reach, which means the least trusted machine in the estate holds a worker token. With a single --worker-token, that machine may name any queue it likes and lease from it, so a compromised host in a DMZ can take a production run and execute it with production credentials.

--worker-pools binds each token to the queues it may serve. The file stores the SHA-256 of each token, never the token itself, the same way a webhook secret is stored:

workers:
  - name: dmz
    token_sha256: 9f2c...           # sha256 of that pool's bearer token
    queues: [dmz]
  - name: production
    token_sha256: 41ab...
    queues: [prod, canary]

A pool that declares no queues may lease from all of them, which is the single-token shape stated out loud. A pool that declares queues is refused anything else, including the default queue when it names none, so confinement cannot be escaped by omission.

Generate a digest with printf %s "$TOKEN" | shasum -a 256. A malformed file stops the server rather than falling back to no confinement, because an install that believes it is segmented and is not is worse than one that refuses to start.

That confines the lease side. The submit side is confined by granting the queue, which is a grantable object named queue:<name>:

curl -X POST localhost:8080/v1/grants \
  -H "Authorization: Bearer $ST_TOKEN" \
  -d '{"subject": "team_sre", "object": "queue:prod", "access": "use"}'

A queue nobody has granted follows the same rule every other object does: the global role decides, unless --strict-grants is on, in which case an ungranted queue is refused. Granting a queue makes it access-controlled, so only the subjects named may route work to it. The grant is checked wherever a queue is chosen: on a run, on a template, on an inventory, and at launch against the template's own queue.

An install that has not turned strict grants on can gate a queue with a rule instead, since a policy matches on queue:

policies:
  - name: hold anything headed for production
    queue: prod
    require_distinct_approver: true

A policy with queue holds only what is routed to that queue, and a run on a named queue executes only when a worker serving that queue claims it. The control node's own runner takes unqueued runs alone. On an install without relay workers, a queued run therefore waits as pending indefinitely, even after approval. To gate runs the server itself executes, write the policy without queue.

Delivering secrets to relay workers

A relay worker holds no database and no encryption key, so on its own it cannot open a credential, and a run that needs one fails on it. A pool opts in to sealed delivery by registering a delivery key. When one of its workers claims a run, the control node opens that run's secrets and nothing else: its credentials and those of the inventory it targets, the registry login its image is pulled with, its custom credential types' field values, from which the worker renders their files, its secret survey answers, and, for a federated credential, an identity token minted for the run. It seals them to the pool's key, bound to that claim, and sends them with the claim. The worker opens them in memory when the run first needs one, writes them into the run's private directory exactly as a worker with database access does, and removes them when the run ends, whether it succeeded, failed, was canceled, timed out, or never started.

To opt a pool in:

1. Generate its key, on a worker or wherever you then copy the private key from:

   switchtender worker key new --out /etc/switchtender/delivery.key
   

2. Register the printed public key on the pool in the worker pool file every control node reads with --worker-pools:

   workers:
     - name: dmz
       token_sha256: 9f2c...
       queues: [dmz]
       delivery_key: x25519:x9f9/kkB6Jq3Wqht/7TWzDGUeTRfbABrj1CO+8/6hHc=
   

3. Restart each control node so it reads the file, and start every worker of the pool with the private key:

   switchtender worker --server https://switchtender.example.com --queue dmz \
     --delivery-key /etc/switchtender/delivery.key
   

The rules it follows:

• Delivery is off until a pool registers a key. A pool without one is sent nothing, and a run that needs a secret fails on its workers with a message naming the pool and how to register a key, which is what every relay worker did before delivery existed.
• Only a pool bound to explicit queues is sent secrets. A pool's queues are the runs it can claim, and so the runs whose secrets its key unlocks, and a pool with no queues could claim anything. A pool file that gives a key to a pool with no queues, or the same key to two pools, stops the server at startup.
• A worker refuses a key file another account can read, the way ssh refuses an unprotected identity file, and refuses to start with one.
• Credential sources resolve on the control node, so a worker in an isolated segment needs no route to Vault or a cloud secret manager. A dynamic secret minted for the run is revoked when the worker reports the run finished, or within about thirty seconds of its claim ending any other way: a worker that died, a run the janitor interrupted or requeued, or a finish recorded through another replica.
• For a Vault dynamic secret the control node also records a revoke handle in the database: the lease id and the Vault address, sealed with the encryption key, beside the run and the credential that minted it. The secret itself is never stored, and the handle is never logged or returned by the API. Every control node with the encryption key sweeps these records, so a secret is revoked once its claim ends even when the node that minted it was stopped or restarted in between. The revoke uses the credential's token, or VAULT_TOKEN for a credential that names the Vault in VAULT_ADDR, and is refused when the credential now names another Vault. A revoke that fails is retried with backoff until the lease would have expired anyway.
• Two kinds of dynamic source give no handle. A dynamic source a plugin adds, whether compiled in through the SDK or loaded as an external plugin, is revoked only by the control node that minted it, so when that node stops first its secret expires on its own lifetime. AWS STS credentials cannot be revoked early at all and always expire on their own lifetime.
• A federated credential's token is minted on the control node, whose signing key never leaves it, for the mode the worker will execute: a plan for a dry run or for an apply a plan-content rule plans first, and an apply otherwise. A worker about to execute in the other mode refuses the token.
• Anything that goes wrong is refused rather than worked around. A secret that does not open on the control node, a delivery that cannot be recorded in the audit trail, a delivery sealed to a key the worker does not hold, or one that does not open for the claim fails the run with the reason. A run never executes with a secret missing.

The seal is HPKE (RFC 9180) in base mode with DHKEM(X25519, HKDF-SHA256), HKDF-SHA256, and ChaCha20-Poly1305, from the Go standard library. HPKE was chosen over NaCl's anonymous sealed box because it authenticates associated data, so the binding is checked by the decryption itself. Each delivery uses a fresh ephemeral key, and the run id, a hash of the claim's lease, the worker's lease name, the pool, the key id, and a random delivery id are all authenticated. So:

• A worker of another pool holds another key and cannot open the delivery.
• A delivery presented for another run, for a later claim of the same run, or by another worker does not open.
• A worker opens a given delivery once and refuses it if it is presented again.
• A worker opens a delivery only after the control node has accepted its fenced start under the same lease, so a claim answer replayed after its run started or finished elsewhere is never decrypted. The start is refused once the janitor has taken the claim back, and to a worker that never claimed the run.
• Only ciphertext crosses the relay, so a proxy or load balancer that logs relay traffic logs nothing readable without the pool's private key.

The worker holds the opened values only until it has written them where the tool reads them, then drops them. Go cannot overwrite a string in place, so what is dropped is the reference, and the decrypted bytes themselves are zeroed. Disable core dumps on worker hosts, as on any host that runs credentialed work, and see Secrets for how the run's private directory is handled.

Rotating a pool's key

A worker can hold several keys, and a delivery names the key it was sealed to, so a rotation needs no moment where runs fail:

1. Generate the new key and install it on every worker of the pool beside the old one, restarting each with both: --delivery-key old.key --delivery-key new.key.
2. Replace the pool's delivery_key with the new public key in the worker pool file on every control node, and restart them. Deliveries are sealed to the new key from then on, and every worker opens them.
3. Remove the old key from the workers and restart them with the new one alone.

Each control node logs, at startup, the key id every pool registered, and each worker logs the key ids it holds, so the step a fleet has reached shows in the logs. Every delivery's audit record names the key id it was sealed to.

When a key may be compromised, replace the pool's delivery_key and restart the control nodes first, then install the new key on the workers. Until a worker has it, credentialed runs on that pool fail rather than run. Treat the secrets of runs the pool executed as exposed and rotate them at their source, and rotate the pool's worker token too, since whoever read the key file could read the token beside it.

What a relay worker writes into the audit trail

A relay worker runs where the control node cannot see it, so two moments are recorded: the run leaving for that machine, and the outcome coming back. A run whose secrets were delivered records a third, between them, naming which pool and worker received which credential ids and secret answer names, under which key, and never a value:

RELAY  /relay/claim/run_4f21a9                       pool:dmz worker:build-dmz-01
RELAY  /relay/delivered/run_4f21a9/key/f4e8a76a457da19cf3742de37986d439/credentials/cred_1a2b3c,cred_4d5e6f/answers/db_password
                                                     pool:dmz worker:build-dmz-01
RELAY  /relay/finished/run_4f21a9/succeeded          pool:dmz worker:build-dmz-01

Captured output, structured events, per-host and per-task summaries, and heartbeats are not recorded. They are the content and liveness of a run that is already stored on the run itself, they arrive several times a second, and writing each into a hash chain would drown the record it exists to make readable.

The claim and finish records do not fail closed, unlike every mutation through the API. Refusing a worker's report because the audit store is unhealthy does not un-finish the run. It loses the outcome of work that already ran on real hosts, so a failure to record is logged loudly instead. The delivery record is the exception: it is written before anything is sent, and a delivery that cannot be recorded is not sent, so the run fails on the worker with that reason.