← All incidents

majormonitoring-gapverifiedfirsthand

Editing a shell script made the firewall silently block every git push for 3 days

An hourly job that pushes a vault's git mirror started failing to connect within 16 milliseconds, and 9 commits sat unpushed for 3 days.

Observed
Severity score
6/10
Blast radius
uptime
Tags
#little-snitch#macos#git#launchd#firewall#ai-coding

Cause

Little Snitch uses the hash of any shell script in a process's lineage as part of its identity. Editing the script changed that hash, and every connection from that lineage was denied without so much as a dialog. On top of that, the job only attempted a push when there were new changes, so a failed push was never retried.

Consequence

Nine commits piled up unpushed for three days. Meanwhile, the working notes still claimed that a failed push would be retried automatically on the next run.

Fix

Split the push out of the script into its own job that has launchd run the Apple-signed /usr/bin/git directly. The script no longer talks to the network; it just counts unpushed commits and alerts when the count crosses a threshold.

What happened

A scheduled job that mirrors an Obsidian vault into git and pushes it sent a notification: “push failed, needs attention.” The log said:

Failed to connect to github.com port 443 after 16 ms

Sixteen milliseconds. A genuine network drop makes you wait for a timeout. Three days earlier, when a push really had failed on a network blip, it took 151 seconds. This time the connection was refused the instant it was attempted.

The chaos on the ground

That earlier failure had been diagnosed, in a session with an AI coding agent, as a transient network drop, and the working notes recorded that “a failed push gets retried automatically on the next hourly run, so the job is resilient to blips.” That was wrong.

The old implementation hit git diff --cached --quiet and, with nothing new staged, unconditionally ran exit 0. Unless new changes arrived, the push was never attempted again. And that day’s “retry succeeded” had been a manual push from a terminal. Pushes from the script were already being denied by then.

The culprit was Little Snitch:

The identity check detected a modification of the program.
Therefore all of its connections were denied as a precaution.

Little Snitch judges identity by the hash of the shell scripts in a network process’s lineage. The script had been edited at 12:16 three days earlier — the same day it was rewritten to fix an rsync failure. From that moment on, every connection was denied, with no dialog at all.

Three things were tried, and none worked:

  • Deleting the relevant deny rule for github.com.
  • Checking that Silent Mode was off. It already was.
  • Creating an allow rule for the git-remote-http executable.

Granting permission to a child process doesn’t help: the denial triggered by the modified script higher up the lineage takes precedence.

Root cause

Two silences stacked on top of each other. The firewall treated a script edit as tampering and cut off the network without a word. And the job was believed to retry failed pushes when it was built not to. A silent denial plus no retry meant nothing anywhere was set up to say the pushes had stopped.

The fix

  • Push now lives in a separate job where launchd runs /usr/bin/git directly, every 15 minutes. git is Apple-signed and its contents don’t change, so the identity check stays intact. Putting a wrapper script in between brings the problem right back, so the binary must be called directly.
  • The original script handles rsync, secret scanning, and the commit, and makes no network calls. It only counts unpushed commits and alerts when the count crosses a threshold.
  • The lesson: under Little Snitch, don’t run scripts that talk to the network. Every edit silently cuts them off. Let a signed binary do the talking.

Sources

  • Firsthand account from the site operator. There is no public write-up to link to.