Eight failures broke an unattended pipeline without raising errors, the longest for about four months (unverified report)

What happened

The author (krbrr) runs a home-built pipeline that collects articles from five sources, scores them with a local LLM (Ollama), and turns the top ones into improvement-proposal files. Claude Code builds and fixes it, and it runs unattended every day on Python 3.11 and SQLite, scheduled with Windows Task Scheduler and PowerShell scripts. The author only watches Discord notifications and approves decisions. Recounting from fix commits and investigation notes, the author says eight failures that broke the pipeline without raising errors had a duration that could be stated. This is an n=1 record from a personal setup, and the author says only one of the eight is specific to Claude Code.

The chaos on the ground

The eight, with dates as the author gives them:

1. A dependency library was missing from the virtual environment, and collection stopped for about 10 days (May 2026, start date unknown). The Discord notification code loaded the same library and stopped with it. It came to light when the author asked for the latest collection status. A separate KeyError bug in the aggregation step overlapped the same period, so how much of it was caused by the dependency has not been separated out.
2. The daily collection on July 15 was missing for a whole day. The not-run check used 48 hours and the health check ran every 6 hours. At the last check the elapsed time was 46.3 hours, which passed, and the next day's run succeeded before the next check and reset the clock.
3. The collection process died partway on two runs (July 21 and 23). Across 63 daily logs, none ended with a non-zero code, and these two had no end line.
4. A task definition for a daily backup was added to the repository on May 1 with a planned start of May 2, but it was never registered in Windows, so the daily backup did not run once until its first run on September 3, about four months later.
5. Nine items whose evaluation failed sat unattended for up to 799 hours (about 33 days). The structure never retried them the next day, and the diagnostic command printed WARN while notifications covered FAIL only.
6. The weekly quality-gate notification did not fire on August 23, when F1 was 0.526 against a pass line of 0.7.
7. A scheduled task in the Claude desktop app froze on permission prompts in five runs between August 30 and September 27.
8. The model used for the pre-filter in front of the evaluation (qwen2.5:1.5b) disappeared from Ollama on September 21, and the filter passed everything through for 28 days (September 21 to October 18).

How they were found varied: a large review on September 2, a WARN read while handling that review, reading the weekly log the next day. The eighth was found by accident while investigating an unrelated error (a dropped connection while fetching article bodies). The author writes that only what gets found can be counted, and that there are probably silent failures going on right now that nobody can see.

Root cause

The author groups the reasons for the silence into five patterns.

• Fail-open also produces "not noticing" (cases 6 and 8). Instead of stopping, permanent faults flowed through the same way as transient ones. In case 8, "model not found" was handled like a network error, retried three times per item, and in the end the item went to the main evaluation without the pre-filter. The log held about 800 WARNING lines a day, nothing reached Discord, and the diagnostic command only checked whether Ollama was reachable.
• A start record or exit 0 is not proof of work (cases 2, 3, 4, 7). In case 3 the collection command returned exit code 0 even when it failed internally, and if the whole process vanished the script never reached its exit-code branch. In case 7 the liveness check looked only at the task's last start time, and since the task did start, it looked healthy. The watcher itself was also a scheduled task and was stuck in the same way.
• The notification path breaks together with what it monitors (cases 1, 6, 7). In case 6, a wrong variable name in the notification code raised an exception that was swallowed by the catch meant to keep the weekly loop from stopping.
• Warnings sound but do not arrive (cases 5, 7, 8). In case 7 the freshness warning for the output reached Discord every day from about September 12, but its guidance text did not point to the real cause (waiting on a permission prompt), and it went nowhere until the investigation on October 3.
• Threshold against cycle (case 2). A 48-hour threshold on a 24-hour job means a one-day gap always slips through.

On the root cause of case 7, the author goes only as far as a leading hypothesis: when the environment variable CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1 is set, the permission mode is forced to default; the version where behavior changed was not identified. Why the model in case 8 disappeared, why the definition in case 4 was never registered, and when case 1 began are unknown because no records exist.

The fix

The author fixed each case as follows.

• Case 1: made the measurement part swappable for a do-nothing version so the code runs without the library, added a dependency consistency check and an import test at the top of the daily script, and added items to the diagnostic command for unevaluated backlog, consecutive failures and stuck runs.
• Case 2: tightened the not-run threshold to 30 hours.
• Case 3: gave the collection command's exit code a convention of 0 = ok, 1 = failure, 3 = partial failure, with the script only branching on it.
• Case 4: re-registered the task after the September 2 review; the first run succeeded at 05:30 on September 3. Added a check that fails if the newest backup file is older than 48 hours.
• Case 5: a daily job now re-picks up to 50 unevaluated items after the daily collection and notifies only when it cannot.
• Case 6: fixed on August 24. After adding any notification, the author now feeds it a failing input to confirm it really fires.
• Case 7: on October 4 moved the job to Windows Task Scheduler with claude -p --permission-prompts none (operations that need confirmation are refused instead of awaited). In a test, 58 items dated October 3 were processed in 5.5 minutes.
• Case 8: separated permanent errors (model not found) from transient ones; a permanent error logs ERROR once and the rest of that run stops calling the filter. The diagnostic command also checks that the models in use exist, and the author confirmed that setting a nonexistent model name makes the diagnosis FAIL.

The author also made the message printed by fail-open catches (... failed (fail-open)) something to grep for in periodic checks, and now writes the most common cause and the investigation-note path into warning texts.

The lesson the author draws is to judge detection by evidence that the work happened, not by whether settings look right. For anything running unattended, check with your own eyes when the output was last updated, rather than relying on notifications.