Shared schedules (runspec-scheduler)
runspec-scheduler runs the runnables in one venv on a timetable, on the host
itself. Nobody needs a console open. Anyone on the team can see these schedules,
start one now, and say they're handling a failure. Think of it as cron, but
only for runspec runnables, and every fire is recorded.
This page covers how it works, how to set it up, every setting it has, and what
to check when something looks wrong. For the console screens, see
Automation → All schedules.
The same daemon can also take webhooks: another system starts a
runnable with an HTTP POST.
Console schedules or shared schedules?
| Console schedules | Shared schedules | |
|---|---|---|
| Where they live | Your console (runspec_schedules.toml in its config folder) |
The host, in each venv ({venv}/runspec_scheduler.toml) |
| Fire when | Your console is open | Always, whether or not a console is open |
| Who sees them | You | Everyone whose console reaches the host, plus anyone on the host |
| Can run | Runnables on any host, prompts, runbooks, saved runs, evals | Runnables installed in that one venv |
| Credentials | Yours, from your keychain | The venv's schedule-only env file |
Use a console schedule for your own reminders and checks. Use a shared schedule for jobs the team relies on: nightly cleanups, reports and health checks.
Not in root-owned venvs
A venv that belongs to root (for example one holding runspec-linux-root) gets
no shared schedules. Every fire there would run as root, and the console never
becomes root to manage a daemon. Schedule those runnables from your console
instead: the console runs them with sudo, the way it runs them from Forms.
The Schedules tab lists root-owned venvs it left out, and Provision and Update
venv don't offer shared schedules for them.
If you already put runspec-scheduler in a root-owned venv, take it out on the
host, as root:
V=/opt/venvs/<root-venv>
sudo $V/bin/runspec-scheduler install-service --remove --write # stop the daemon and its keeper
sudo $V/bin/runspec-scheduler use daemon # take its lines off root's crontab
sudo $V/bin/pip uninstall -y runspec-scheduler
How a fire runs
Every fire, from either backend and from Run now, runs the same command:
{venv}/bin/runspec exec --label sched:<id> --trigger schedule:<cron|daemon|run-now> \
--lock ~/.cache/runspec-scheduler/<venv>-<hash>/<id>.lock \
--env-file {venv}/.runspec_scheduler_env [--timeout <seconds>] \
<runnable> -- <args>
So each fire:
- is one recorded run in the venv's runs index, with its exit code and the
last part of its stdout and stderr. Console History finds every fire of a
schedule by searching
sched:<id>; - never overlaps itself. If the last fire is still running, the new one is recorded as skipped instead of starting a second copy. Each venv has its own lock folder (runspec-scheduler 0.7.3), so two venvs with the same schedule id don't block each other;
- loads the schedule-only env file on top of
.runspec_env(see Credentials); - says what started it. History's Run by column shows cron, daemon or run now.
Two backends: cron or daemon
Each venv picks one backend. A venv never uses both, so a schedule can't fire twice.
cron (default) |
daemon |
|
|---|---|---|
| What fires the schedules | Lines in the owner's crontab (# runspec:sched:<id>) |
One runspec-scheduler run process per venv |
| Extra processes | None | The daemon, plus a keeper |
Cron expressions (0 2 * * *, @daily) |
Yes | Yes |
Intervals (every = "15m") |
No | Yes |
| A time zone other than the host's | No | Yes |
last day of the month (package schedules) |
No | Yes |
| Missed fires (host was down) | Lost, as with plain cron | Recorded as skipped; catch_up runs one on start |
| Default fire timeout | None | 6 hours |
| Picks up a config change | After runspec-scheduler sync (the console runs it for you) |
Within a second or two, by itself |
| Health shown in the console | Per schedule | Per schedule, plus the daemon (running / stale / stopped, keeper, version) |
Start with cron. It has no moving parts and works on any host with a crontab. Move a venv to the daemon when you need intervals, another time zone, catch-up after a reboot, or fire timeouts.
To mix them on one host, use two venvs. For example, keep ops on cron and put
the interval checks in an ops-daemon venv.
Setting it up
From the console
- New venv: Settings → Hosts → Provision venv…. Tick
Shared schedules to install runspec-scheduler, move the venv to the daemon
and set up its keeper. Leave it off and you can still add
runspec-schedulerunder Packages to use the cron backend. See Provision a venv. - Existing venv: Update venv…, then Shared schedules:
- Leave as they are;
- Move to the daemon, with a keeper;
- Back to crontab.
The preview says how many crontab entries would move. See
Update an existing venv.
- Just the package: install runspec-scheduler into the venv from the
Marketplace. The venv stays on the cron backend.
Then add schedules from the Schedules tab: New host schedule. See Adding and editing host schedules.
From a shell
Run these as the venv owner, with the venv's own binaries:
/opt/venvs/ops/bin/pip install runspec-scheduler
/opt/venvs/ops/bin/runspec-scheduler add --id nightly-cleanup --runnable cleanup-tmp --cron '30 2 * * *'
/opt/venvs/ops/bin/runspec-scheduler sync # cron backend: write the crontab lines
/opt/venvs/ops/bin/runspec-scheduler status # check them
To use the daemon instead:
/opt/venvs/ops/bin/pip install 'runspec-scheduler[supervisor]' # [supervisor] only if you'll use that keeper
/opt/venvs/ops/bin/runspec-scheduler use daemon # takes this venv's lines out of the crontab
/opt/venvs/ops/bin/runspec-scheduler install-service # shows which keeper it would set up, changes nothing
/opt/venvs/ops/bin/runspec-scheduler install-service --write # sets it up and starts the daemon
Run it as the owner
The scheduler only runs as the account that owns the venv. Its config,
crontab lines and daemon all belong to that account. For a venv owned by a
service account, use sudo -iu svc-ops /opt/venvs/ops/bin/runspec-scheduler ….
The console does the same for you, without a shell (sudo -n -u <owner>
{venv}/bin/python …).
Keepers: what keeps the daemon running
The daemon is an ordinary process. Something has to start it when the host boots and start it again if it dies. That something is the keeper. A cron-backend venv has no daemon, so it needs no keeper: cron itself is always running.
| Keeper | Needs | Restarts a crashed daemon | Survives a reboot | Root needed |
|---|---|---|---|---|
systemd --user (systemd-user) |
Lingering on for the owner; systemd newer than 219 (not CentOS 7) | Within seconds | Yes | Once, to turn lingering on |
supervisord in the venv (supervisor) |
runspec-scheduler[supervisor] in the venv; a crontab |
Within seconds; supervisord itself within 5 minutes | Yes, through @reboot |
No |
Crontab watchdog (cron) |
A crontab | Within 5 minutes | Yes, through @reboot |
No |
System unit (systemd) |
Root | Within seconds | Yes | Yes (only printed, root installs it) |
install-service --manager auto (the console's keeper picked per host) picks:
- systemd --user, when lingering is on, systemd is newer than 219 and the user's systemd answers;
- else supervisord, when it is installed in the venv;
- else the crontab watchdog.
The result says which keeper it picked and why. Name one with
--manager systemd-user|supervisor|cron.
Nothing changes without --write. Without it, install-service prints the
plan: the files it would write, the crontab lines and commands, and any steps
root has to run.
A venv has one keeper. Installing a different one takes the old one down first, from runspec-scheduler 0.10.4: it removes its crontab lines, disables its systemd --user unit or shuts down its supervisord, and stops the daemon it started. The new keeper then starts the daemon. Before 0.10.4 the old keeper stayed, and the daemon it had started kept the venv lock. The new keeper's daemon then exited at once, and systemd took that as finished.
systemd --user
# ~/.config/systemd/user/runspec-scheduler-<venv>.service (the main lines)
[Service]
ExecStart={venv}/bin/runspec-scheduler run
Restart=on-failure
RestartSec=5
KillMode=process # a restart leaves fires already running to finish
install-service --write writes this unit and runs systemctl --user enable --now.
systemd starts the daemon at boot and restarts it 5 seconds after it fails. Check it
with:
systemctl --user status runspec-scheduler-ops
journalctl --user -u runspec-scheduler-ops # the daemon's log lines
A --user unit normally stops when the account logs out. Lingering keeps it
running with nobody logged in. Root turns it on once per account:
sudo loginctl enable-linger svc-ops
loginctl show-user svc-ops -p Linger # Linger=yes
systemctl --user needs that account's own user manager. A plain
sudo -u svc-ops (which is how the console runs it) doesn't set one up. When
lingering is on, the manager runs all the time and install-service finds it in
/run/user/<uid> by itself. When lingering is off and the account has no login
session, there is no manager, so install-service --manager systemd-user
refuses before writing anything and says to turn lingering on (from 0.10.3).
Before 0.10.3 it wrote the unit, failed to enable it, and left the file behind.
supervisord in the venv
This keeper needs no root and no system packages. It is two parts, each watching the next:
crontab (always running)
└─ every 5 min + at boot: start {venv}/bin/supervisord ← exits at once if already running
└─ supervisord: runs runspec-scheduler run, restarts it when it dies
└─ runspec-scheduler daemon: fires the schedules
- supervisord is pip-installed into the venv by
runspec-scheduler[supervisor]. Its config is in{venv}/supervisor/: supervisord.conf;conf.d/runspec-scheduler.conf, the program block (autorestart=true);- its socket and pid file.
- Two crontab lines start supervisord at boot (
@reboot) and every 5 minutes (*/5 * * * *). supervisord refuses to start a second copy while one is running, so the 5-minute line does nothing unless supervisord itself has died. - Each line has a tag on the line above it, so you can tell what it is:
# runspec:scheduler-keeper:/opt/venvs/ops
@reboot /opt/venvs/ops/bin/supervisord -c /opt/venvs/ops/supervisor/supervisord.conf >/dev/null 2>&1
# runspec:scheduler-keeper:/opt/venvs/ops
*/5 * * * * /opt/venvs/ops/bin/supervisord -c /opt/venvs/ops/supervisor/supervisord.conf >/dev/null 2>&1
Keeper lines carry a different tag from the cron backend's
# runspec:sched:<id> lines, and each tag names its venv. That keeps the
backend's lines and the keeper's lines separate, even with several venvs in
one crontab.
To look at it by hand:
/opt/venvs/ops/bin/supervisorctl -c /opt/venvs/ops/supervisor/supervisord.conf status
/opt/venvs/ops/bin/supervisorctl -c /opt/venvs/ops/supervisor/supervisord.conf restart runspec-scheduler
Its own events go to {venv}/logs/supervisord.log. The daemon's startup errors
and crash tracebacks go to {venv}/logs/scheduler.stdout.log.
--manager supervisor --system prints a program block for the host's own
supervisord instead. An admin adds it to /etc/supervisord.d/ and runs
supervisorctl update.
Crontab watchdog
This is the simplest keeper: two tagged crontab lines that run
runspec-scheduler run --detach, at boot and every 5 minutes.
# runspec:scheduler-keeper:/opt/venvs/ops
@reboot /opt/venvs/ops/bin/runspec-scheduler run --detach >/dev/null 2>&1
# runspec:scheduler-keeper:/opt/venvs/ops
*/5 * * * * /opt/venvs/ops/bin/runspec-scheduler run --detach >/dev/null 2>&1
Only one daemon can run per venv, because it holds a lock on
{venv}/runspec_scheduler.lock. A start while one is running exits straight
away, so the 5-minute line does nothing until the daemon has died. A crashed
daemon is back within 5 minutes. Fires due in that gap are recorded as skipped,
or run once on start for a schedule with catch_up.
Restarting and removing
After pip upgrades runspec-scheduler, the daemon keeps running the old code
until it restarts. status and the console then show restart needed.
runspec-scheduler restart --dry-run # what it would do
runspec-scheduler restart # restart through the keeper, wait for the first heartbeat
restart goes through the keeper:
systemctl --user restart;supervisorctl restart;- for the crontab watchdog (or no keeper), stop and then
run --detach.
Fires already running carry on and are still recorded. Config changes never need a restart. The console's Restart / Start button and Update venv… (after it upgrades packages) run the same command.
Before you delete a venv, remove its keepers, so nothing tries to start a daemon from a folder that is gone:
runspec-scheduler install-service --remove # what it would remove
runspec-scheduler install-service --remove --write # remove every keeper, stop the daemon
The console's Deprovision… does this first. For a cron-backend venv it also
takes the venv's # runspec:sched: lines out of the crontab.
Schedules
A schedule is a [[schedule]] entry in {venv}/runspec_scheduler.toml. You
rarely edit the file by hand: the console's schedule dialog and
runspec-scheduler add write it for you. It looks like this:
[[schedule]]
id = "nightly-cleanup"
runnable = "cleanup-tmp"
enabled = true
timeout = "30m"
catch_up = true
when = { cron = "30 2 * * *", timezone = "Europe/London" }
args = { older-than = "7d" }
[[schedule]]
id = "tmp-usage"
runnable = "filesystem"
command = ["directory-size"]
when = { every = "15m" }
args = { path = "/tmp" }
| Field | What it does | Default |
|---|---|---|
id |
Names the schedule. It is the crontab marker and the History label (sched:<id>). Letters, digits, ., _ and -, up to 64 characters, starting with a letter or digit. |
(required) |
runnable |
A runnable installed in this venv. | (required) |
command |
The subcommand path for a runnable with verbs, e.g. ["directory-size"] or ["admin", "users"]. |
none |
args |
Arguments for the runnable, by name. They are checked against its spec when you save. | none |
when.cron |
Five cron fields (30 2 * * mon-fri) or @hourly, @daily, @midnight, @weekly, @monthly, @yearly. |
one of cron or every |
when.every |
An interval: a number and s, m, h or d (30s, 15m, 1h). Daemon only. |
one of cron or every |
when.timezone |
An IANA zone (Europe/London, UTC) whose clock the schedule keeps. Blank means the venv's zone, else the host's. On cron, only the host's zone works. |
the venv's zone |
enabled |
false pauses it. A paused schedule still runs with Run now. |
true |
timeout |
Stops a fire that runs longer than this: seconds, or 90m, 6h, 1d. 0 means no limit. |
the venv's timeout on the daemon; none on cron |
catch_up |
Daemon only. When the daemon starts after this schedule's fire was due (host down, daemon stopped), run it once straight away. Several missed fires become one run. | false (missed fires are skipped, as with cron) |
run_as, become_method |
Fire as another account (sudo or su). The owner needs passwordless access to that account. |
the venv owner |
Clock changes. When the clocks go back, a schedule at fixed times (no * in
its minute or hour, like 30 1 * * *) fires once, as cron does. One with *
there (like */30 * * * *) keeps firing through both copies of the hour. A time
in the hour the clocks skip fires straight after the jump.
Time zones. A crontab line can only fire on the host's clock. On a cron venv,
add refuses a schedule in another zone, and sync skips it (status says
why; the console shows needs daemon). To run a 09:00 Sydney job on a host
whose clock is UTC, either:
- move the venv to the daemon and set the schedule's zone; or
- set the venv's zone (
settings --timezone Australia/Sydney) so every schedule without its own zone uses it.
Schedules a package suggests
A runnable package can suggest schedules in its own runspec.toml. A venv it is
installed in can then run them without anyone writing the schedule:
[cleanup-tmp.meta.scheduler.nightly] # schedule id: cleanup-tmp.nightly
description = "Remove files older than 7 days from /tmp"
at = "02:30" # or "midnight", "noon"
on = "daily" # or "weekdays", "weekends", "mon-fri", "sun", "mon,wed", "1", "last"
timezone = "local" # "local" (the venv's zone, default), "UTC", or a zone name
spread = "30m" # each host picks its own fixed minute in 02:30-03:00
args = {older-than = "7d"}
- Each suggestion has exactly one of
at(withon),everyorcron. - It can also set
command,timeout,catch_upanddescription. spreadgives each host a fixed offset worked out from its name and the schedule id. That way 70 hosts don't all start at 02:30:00.last(the month's last day) needs the daemon.
The venv's package schedules setting says what happens to them:
| Setting | What happens |
|---|---|
offer (default) |
Listed as offered, off until someone turns one on |
auto |
On as soon as the package is installed |
off |
Ignored |
Turning one on (runspec-scheduler enable --id cleanup-tmp.nightly, or
Turn on in the console) writes only enabled for that id. So when a new
package version moves the time, the venv follows. If you change its time
(add --id cleanup-tmp.nightly …), your time sticks. remove --id drops your
changes and goes back to the package's time. Uninstalling the package drops its
schedules.
The daemon picks up installed, upgraded and removed packages once a minute. On
cron, run sync after pip. The Marketplace and Update venv… run it for you.
Venv settings reference
The [scheduler] table at the top of {venv}/runspec_scheduler.toml holds the
venv's own settings. Change them with runspec-scheduler settings, or in the
console with Venv settings on the Schedules tab (or Settings on a daemon
strip). The daemon picks up changes by itself. On cron, the console runs sync
for you.
[scheduler]
backend = "daemon"
timeout = "6h"
misfire_grace = "5m"
catch_up_window = "1d"
timezone = "Australia/Sydney"
package_schedules = "offer"
| Setting | Console label | What it does | Default |
|---|---|---|---|
backend |
Update venv… → Shared schedules | cron or daemon. Change it with runspec-scheduler use cron or use daemon: use cron writes the crontab lines; use daemon removes them. |
cron |
timeout |
Fire timeout | The longest a daemon fire may run before it is stopped. 0 means no limit. A schedule's own timeout wins. |
6h |
misfire_grace |
Misfire grace | How late a daemon fire may still start, for example after the host was busy or the daemon was restarting. Later than this, it is recorded as skipped. | 5m |
catch_up_window |
Catch-up window | How far back a catch_up schedule looks when the daemon starts. A fire missed longer ago than this isn't run. |
1d |
timezone |
Time zone | The zone for schedules with no zone of their own, and for package schedules that say local. settings --timezone host goes back to the host's zone. |
the host's zone |
package_schedules |
Package schedules | offer, auto or off (see above). |
offer |
Durations are seconds or a number with s, m, h or d.
When a daemon fire times out, runspec exec stops it and records exit code 124.
The daemon then kills anything left in its process group. One caution: a
run_as fire that went through sudo can leave children running as that other
account.
Credentials: the schedule-only env file
Host schedules never carry secrets. Put what the fires need in
{venv}/.runspec_scheduler_env:
# {venv}/.runspec_scheduler_env — loaded only by scheduled fires and webhook runs
JIRA_URL=https://jira.example.com
JIRA_PERSONAL_TOKEN=…
- Every fire loads it on top of
.runspec_env, so a value here wins. This covers cron, daemon and Run now fires, and webhook runs. - Before runspec-scheduler 0.11.0 it was called
runspec_schedules.env. The old name isn't read any more: rename the file (mv runspec_schedules.env .runspec_scheduler_env).statuswarns while the old one is there. - Nothing else in the venv reads it: console runs, the agent and runs from a
shell only see
.runspec_env. A service account's token in this file never stands in for an operator who didn't pick their own credential. - It must be mode
600and owned by the account the fires run as. A file that others can write, or that belongs to another account, is refused: the fire is recorded as failed with exit code 78. A missing file is fine. - Edit it from the console's Env files window (switch to Schedules file). When you pick a runnable in the schedule dialog, it lists the variables the runnable reads and whether this file sets them.
This keeps scheduled credentials away from interactive runs. It is not a hard wall: if scheduled fires and interactive runs use the same account, that account can read both files.
Watching schedules and handling failures
On the host:
runspec-scheduler status # every schedule: next fire, last runs, failures, who is on it
runspec-scheduler watch # the same, redrawn every 5 seconds (Ctrl+C to stop)
runspec-scheduler status --json # what the console reads
runspec-scheduler run-now --id nightly-cleanup --detach # fire it now, in the background
runspec history jobs # each scheduled job's state, and who is handling a failure
In the console, the Schedules tab has four cards: All, This console, Host cron and Host daemon. They cover local and every connected host in the active profile. Each row shows:
- the schedule's state, next fire and last run;
- the venv owner and the account the schedule runs as;
- Run now, plus Edit, Pause and Remove for host schedules.
Expand a row to see recent runs and the last run's output.
Because the schedules are shared, a failing one may be seen by several people. Tick the failing rows, then:
- I'm on it claims them, with an optional note;
- Dealt with… marks them handled, with a note saying what was done;
- Release takes either mark off.
A mark lasts until the schedule next succeeds. It is stored on the host, so everyone sees it. A claim with no update for 4 hours counts as a problem again. See Who is handling a failure.
Files on the host
| File | What it is |
|---|---|
{venv}/runspec_scheduler.toml |
The schedules and the [scheduler] settings |
{venv}/.runspec_scheduler_env |
Credentials for scheduled fires and webhook runs only |
{venv}/hooks.d/*.toml, {venv}/hooks/spool.sqlite |
Webhooks: deployed hooks and the delivery spool |
{venv}/runspec_scheduler.lock |
Held by the running daemon, so only one runs per venv |
{venv}/runspec_scheduler_backups/ |
A copy of the crontab from before each rewrite |
{venv}/logs/scheduler.log |
What the daemon did: starts, reloads, fires, warnings |
{venv}/logs/scheduler_state.json |
The daemon's heartbeat, every 5 seconds; status reads it |
{venv}/logs/runspec-scheduler.log |
A record of each runspec-scheduler command run |
{venv}/logs/scheduler.stdout.log |
The daemon's own output: startup errors and crash tracebacks (supervisord and crontab watchdog keepers) |
{venv}/logs/supervisord.log |
supervisord's events (supervisord keeper only) |
{venv}/supervisor/ |
supervisord's config, socket and pid file (supervisord keeper only) |
~/.config/systemd/user/runspec-scheduler-<venv>.service |
The unit (systemd --user keeper only) |
~/.cache/runspec-scheduler/<venv>-<hash>/<id>.lock |
Each schedule's lock, in the home of the account it runs as |
The four log files rotate at midnight and keep 7 days. Each fire's own output is in the venv's run logs, under the runnable's name. The venv's log cleanup policy covers those.
Troubleshooting
A daemon venv shows scheduler stopped. Nothing is running the daemon.
Press Start in the console, or run runspec-scheduler restart. If it stops
again, its startup error is in {venv}/logs/scheduler.stdout.log (or
journalctl --user -u runspec-scheduler-<venv> under systemd). status lists
the keepers the venv has under keeper. No keeper means nothing will start
the daemon after a reboot: run install-service --write.
A daemon venv shows scheduler stale. The process is there but has stopped
writing heartbeats for more than three intervals, so it is probably stuck.
Restart it, and check scheduler.log for the last thing it did.
restart needed. pip installed a different runspec-scheduler since the
daemon started. Run Restart. If the flag stays after a restart, an older
daemon that the keeper didn't start (for example one the cron watchdog started
before you switched to systemd --user) holds the venv lock, and each new daemon
quits at once. From 0.10.5 Restart stops that daemon first. On older versions
stop it yourself (kill <pid> as the venv owner; status shows the pid) and
restart.
A cron schedule shows not installed or stale line. The crontab doesn't
match the config. Press Sync crontab (or run runspec-scheduler sync).
Legacy line means a scheduler older than 0.3.0 wrote the line, so its fires
aren't recorded. A sync fixes that too.
needs daemon. The schedule wants an interval, another time zone or the
last day, and the venv is on cron. Move the venv to the daemon with
Update venv…, or change the schedule.
A fire shows skipped. Either the previous fire of the same schedule was
still running, or the daemon was down (or later than misfire_grace) at the
time. Expand the row in the console to see which.
A fire failed with exit code 78. .runspec_scheduler_env exists but is
group- or world-writable, or owned by another account. Fix it with
chmod 600 and chown to the account the fires run as.
The console shows can't read for a venv. The console reads each venv's
status as the venv owner with sudo -n -u <owner>. That needs passwordless
sudo from your SSH login to the owner.
Failed to connect to bus: No medium found, or Failed to connect to user
scope bus … $XDG_RUNTIME_DIR not defined. systemctl --user found no user
manager for the account. Turn lingering on (sudo loginctl enable-linger
<owner>) and try again; from the console that is enough. Or pick the supervisord
or crontab-watchdog keeper, which don't need systemd.
Restart says systemctl --user can't connect, but the venv uses another
keeper. A systemd --user install that failed before runspec-scheduler 0.10.3
left its unit file behind, and restart took it for the keeper. Upgrade
runspec-scheduler, which only counts an enabled unit. install-service --remove
--write deletes the stray file, or delete
~/.config/systemd/user/runspec-scheduler-<venv>.service as the owner.
CentOS 7. Its systemd (219) can't run user services reliably, so auto
never picks systemd --user there. Use supervisord or the crontab watchdog.
Scheduled runs count as agent runs in Analytics. Before runspec-scheduler
0.10.1, crontab lines, daemon fires and Run now set RUNSPEC_AGENT=1, so each
fire's record said it was an agent run. Upgrade runspec-scheduler, then restart
the daemon, or on cron press Sync crontab so the lines are rewritten.