Do not start the job: load shedding for scheduled batch work

💥 What happened

On 8 July 2026 a heavy test run started on a host that was already busy. Load reached roughly eight hundred percent on eight cores, and the whole machine, including the production services, hung. Nothing was wrong with the job. It was wrong to start it then.

A second incident on 14 September 2026 had a different cause with the same shape: the disk reached ninety-seven percent because backups had no rotation, and every job that wrote in that window made it worse.

🛑 The rule

A scheduled job does not start when the host is overloaded. It records the skip and exits successfully, and the scheduler starts it on the next tick. Data catches up later, and production keeps breathing. Our wrapper checks two things before running a job: the one-minute load against a limit, six by default on this eight-core host, and the free disk space against a minimum, two gigabytes by default.

If both pass, the wrapper runs the job under an exclusive lock so that two copies cannot overlap, at a lowered CPU priority and with idle disk priority. A second wrapper rule: if the disk check tool itself is broken, the disk check is skipped. A broken tool must not stop every job on the host.

🕵️ A watchdog that never kills

A separate watchdog runs every five minutes and writes its status to a file. It derives a soft and a hard load threshold from the number of cores. Past the hard line it lowers the priority of known batch processes, and it never kills anything. On 3 November 2026, 21:53 UTC it reported 8 cores, a soft threshold of 4.8 and a hard threshold of 6.8, with the last action "ok".

We also moved CPU shares: scheduled jobs together may use at most six of eight cores and carry a low weight, while production services carry a higher weight.

📊 The shed log

The wrapper appends a line each time it declines a job. Since 2026-07-30 the log holds 8,415 lines across 67 different jobs, and the four most frequent jobs account for 4,938 of them. The busiest day was 2026-10-15 with 1,155 declined starts.

Read this as a capacity signal, not as failure. Thousands of declined starts mean that the host is full at the moments when jobs are scheduled, and that the schedule needs spreading or the work needs moving to another machine. They also mean the guard is doing its job: each one was a start that might have hurt production.

🛠️ Practice: wrap your heavy jobs

1. List the scheduled jobs and mark the heavy ones: tests, backfills, optimisers, exports.
2. Write a wrapper that checks load and free disk and exits quietly with a log line when either is out of range.
3. In the same wrapper add an exclusive lock, a lower CPU priority and idle disk priority.
4. Put every heavy job behind the wrapper. Do this by file, not by editing a shared schedule table.
5. Add a watchdog that records load and lowers priority of batch processes, and never kills them.
6. Review the shed log weekly. Move or spread the jobs that are shed most often.

✅ Check yourself

• Start a CPU burner and then a wrapped job. The job is declined and the log shows the load figure.
• Start the same wrapped job twice. The second copy exits at once.
• Fill the disk past your minimum on a test host. Jobs that write are declined.
• You can say which job was declined most often last week.

🚧 When this does NOT work

• Jobs that must run. A job whose deadline matters, such as a backup, should not be shed forever. Add a maximum skip count.
• Persistent overload. If the host is always over the limit, nothing runs. That means you need more capacity, not a lower limit.
• Spiky services. A short spike can decline a job that would have fit. Use a short moving average.
• Priority is not isolation. A low-priority job can still use memory and disk. Limits per group are stronger.
• Silent skipping. If nobody reads the log, you can lose a job for weeks. Alert on a job that has not completed for a day.

📊 Evidence

➡️ Next

Plan what happens when something does break in An AI incident playbook for a small team, and read Recovery after a failed, stuck or uncertain run.

Sources

• An AI incident playbook for a small team: response steps.
• Recovery after a failed, stuck or uncertain run: recovery.
• Costs, budgets and limits: limits.
• Agent shell access: decision forks: shell access choices.