Blackbox BlastIncident Log
← All incidents

majormonitoring-gapverifiedfirsthand

A receipt-OCR watcher crashed 11 times on a missing module and nobody was told

A launchd job that OCRs receipt images dropped into a folder and adds them to a household ledger had crashed with `ModuleNotFoundError` 11 times in a row. No alert ever fired, and the images just sat there unprocessed.

Observed
Severity score
5/10
Blast radius
uptime
Tags
#launchd#macos#python#dependencies#silent-failure

Cause

The OCR dependency, pyobjc (Quartz/Vision), was never declared in the requirements file. The Python it had been installed into during development was not the Python the launchd job definition actually called, so the job broke silently the moment the interpreter changed.

Consequence

Automatic receipt import was down for more than a day. The ledger itself was not corrupted; its SHA-256 matched before and after.

Fix

Declare the dependency, install it into the exact interpreter launchd calls, and verify the import works in a stripped-down environment. The job now runs under a monitoring kernel whose preflight tries import Quartz, Vision before doing anything else.

What happened

While building a small kernel to monitor launchd jobs and migrating the existing jobs onto it, the last exit code of each job got checked one by one. The receipt watcher was the only one sitting at exit=1.

The job wakes up when a receipt image lands in a watched folder, runs macOS Vision OCR on it, and appends a line to a household ledger. Its log held the same traceback eleven times over:

ModuleNotFoundError: No module named 'Quartz'

The last failure was the evening before it was discovered. Up to that point, not a single dropped image had been imported.

The chaos on the ground

The nasty part is that this job only fires when a person drops a file in. A job that runs hourly makes noise when it goes quiet. A job that fires rarely looks exactly the same broken as it does on a day with no receipts. It failed eleven times, and all eleven failures went nowhere.

Worse, the development notes said “pyobjc installed.” That was true — for one Python, at one point in time. The launchd job definition called /usr/local/bin/python3.14, and pyobjc was not in it.

Same shape as the translation job that failed 13 runs in a row unnoticed for five days: python3 resolved to something different under launchd than it did in an interactive shell.

Root cause

The real cause was that the dependency was declared nowhere. The requirements file listed only openpyxl and pytest; the pyobjc packages the OCR module needed (Quartz/Vision/Cocoa) were missing. With nothing written down, there was no way to know what went missing when the interpreter changed underneath the job. And a job that dies on a missing dependency keeps dying quietly unless something is watching it.

The fix

  • Added pyobjc-framework-Vision>=12.0 to the requirements file, with a comment explaining why.
  • Installed it with pip install --user into /usr/local/bin/python3.14, the interpreter launchd actually calls, without touching the framework’s own site-packages.
  • Confirmed the import works under a minimal env -i HOME=$HOME environment, closing the gap between launchd and the interactive shell.
  • Moved the job onto the monitoring kernel with FAIL_THRESHOLD=2. For a job that fires this rarely, waiting for three failures would delay detection by days.
  • A preflight step now tries import Quartz, Vision before the job runs. If the modules are missing, the run is recorded as a failure before it ever reaches OCR.

The image that had been stuck in the folder turned out to be a screenshot, and the gate added after an earlier incident correctly rejected it. The ledger’s SHA-256 matched before and after: nothing was corrupted.

Sources

  • Firsthand account from the site operator. There is no public write-up to link to.