CASE · Major · monitoring-gap
VerifiedFirsthand recordA scheduled job failed 13 runs in a row and no one noticed for 5 days
A scheduled translation pipeline had been exiting with code 1 on every run for 13 consecutive runs, and nobody noticed for 5 days.
2 min read
- OBSERVED
- CATEGORY
- monitoring-gap
- BLAST RADIUS
- uptime, data
- DETECTED BY
- manual-log-review
- TAGS
- #launchd #macos #monitoring #silent-failure
CH 1
Cause
launchd does not surface failures on its own. Nothing was reading exit codes, so 'it's probably running' was the only signal anyone had. To make it worse, some jobs are designed to exit 0 silently when their preconditions aren't met, which is indistinguishable from a healthy run by exit code alone.
CH 2
Consequence
For 5 days, the job simply did not run, and nobody knew.
CH 3
Fix
Wrap scheduled jobs with a thin observability layer that records every run's exit code and last output line, and set an IDLE threshold per job based on what 'normal' actually looks like for that job — a job that's supposed to sit idle after finishing needs a different threshold than one that's supposed to run every night.
What happened
Nobody was watching the exit codes. A macOS launchd job that retranslates
research notes had been failing quietly, run after run, for five days
straight — 13 failed attempts, all exit code 1, and not a single alert.
The job wasn't broken in a dramatic way. It just stopped doing its job and nothing said so.
The chaos on the ground
The evidence surfaced in a plain log file: a retranslate.log showing 13
straight exit=1 runs, unnoticed for 5 days. A second, independent case of
the same blind spot was on record around the same time: a backup script had
been failing every night for about two weeks after a path was renamed —
silent for a different reason, exiting 0 (success) whenever its
preconditions weren't met.
Two unrelated jobs, same blind spot: nothing was reading what they reported.
Root cause
Autonomous scheduled agents don't complain on their own. If nothing reads their exit code, "still running" and "dead for a week" look identical from the outside. Some jobs make this worse by design: they exit 0 (success) when a precondition isn't met, which means even watching for non-zero exit codes isn't enough — you have to know what a healthy silence looks like for that specific job.
The fix
A minimal kernel now wraps six of the scheduled jobs in three stages — preflight,
the job itself unmodified, and a record step — and logs exit code plus the
last line of output to a small database. Each job gets its own IDLE
threshold tuned to its actual behavior, not a one-size-fits-all timeout.
Jobs that are supposed to go idle after finishing get IDLE_THRESHOLD=off;
jobs that are supposed to run nightly and might silently no-op get a tight
threshold that flags the silence itself as the anomaly.
Sources
- Firsthand account from the site operator. There is no public write-up to link to.
Pass this record on to your team