hearth is what I built to schedule every job on my homelab's GPU

Every scheduled job in my homelab runs through a platform I built called hearth, and one piece of it is a gateway that admits one job at a time to whichever machine happens to be running a model. It runs on the Pi, and it exists because nothing else that could do the job fit the hardware. Kubernetes-style orchestration (k3s, Nomad) carries its own memory tax, and the Pi was already sitting at 5 GiB used plus 4.4 GiB of swap before adding a scheduler to the pile. Plain systemd timers per service were the other option, and they don’t do priority ordering, backpressure, or a shared budget across jobs, which is most of what a GPU with one model slot needs.

A queue that survives a reboot for free

hearth’s queue is SQLite in WAL mode, not Redis or Celery. Memory is the scarce resource on this box, not throughput, and a SQLite file on disk means the queue’s state survives a reboot without anything extra. Jobs carry a priority (interactive, user, batch, best-effort), a manifest-driven schedule, retry with exponential backoff, and a dead-letter path for anything that needs a human to look at it. If a job depends on something that’s down, it parks instead of burning retries against a dependency that isn’t coming back soon.

Each job runs as a subprocess inside its own systemd-run scope, and every resource class maps to a systemd slice with its own memory and CPU limits. An OOM kills the one job in that slice, not the host.

Adding a resource class costs two files, not one, a manifest entry and a matching slice unit, and there’s a test that fails loudly if the second file is missing. A job in a class with no slice unit doesn’t error at validation, it dies five seconds into every run with no output and nothing in the journal, so the two-file rule exists to keep that failure mode rare.

The gateway learns durations from the worst recent run, not the typical one

hearth’s scheduler will only start a queued job ahead of a fixed time window if it can promise the job finishes before that window closes, which means it needs an honest estimate of how long the job takes. The first version learned that estimate as the median of the last five runs, on the reasoning that one bad night shouldn’t poison the number.

One of the jobs that runs through hearth is bimodal: about twenty minutes when it finds real work, about five seconds when it doesn’t. Two of its three recorded runs were quiet ones, so the learned median came out to six seconds against a real run that took twenty-one minutes. Under that estimate, the scheduler would have started the job thirty seconds before a workday window it had no chance of clearing. Deleting the two quiet runs fixed the number for twenty minutes, until a fresh quiet run dragged it right back down, which is what proved the fix was against the symptom, not the cause: a bimodal job produces quiet runs indefinitely, and a median can’t tell a normal quiet run from an anomaly.

The estimate is now the maximum of the last five runs, not the median. Verified against the same history: max reads 1277 seconds where median read 641. Under-promising is the dangerous direction here, since a job that starts and gets killed partway wastes the work it did before dying, while over-promising only makes something wait a little longer, which costs nothing. A job with occasional long runs is always scheduled as if it will take the long path, so fewer jobs fit into small gaps in an otherwise empty day. For a scheduler whose entire job is filling a mostly idle day, that’s a fine trade. It wouldn’t be on hardware that’s busy.

The output cap moved out of every caller and into the gateway

Every service calling into the GPU box’s model used to hardcode its own token limit, and every single one of those numbers was wrong. Twenty-nine calls in one day landed on exactly 1024 tokens against prompts of 600 to 900 tokens, some of them for a job scoring zero results out of eighty-one it should have judged, because the model’s answer got cut mid-word before it finished.

No caller can size this correctly, because the two facts that determine a correct cap don’t live where the caller lives. The context window is a property of the model config, and the generation rate is a property of the hardware, and a caller has access to neither, so any number it writes down is a guess. Guesses land low, because a low guess only fails sometimes, and it takes a while to notice.

The fix moved the cap to the one place that has both facts: the gateway derives it from the request’s timeout budget times the model’s measured tokens per second, and fills that number in whenever a caller sends none. One of the models running through hearth is both the slowest measured, at 13.9 tokens a second against a faster model’s 17.5, and the one with the largest context window, so an unbounded generation on that model alone could hold the GPU for close to an hour under a single-slot gateway. Sizing the cap to the same time budget the timeout already enforces costs the same wall clock in the worst case, but comes back with a usable partial answer instead of nothing.

A second GPU doesn’t get routed through a chokepoint built for a different GPU

There’s a second machine on the network with its own GPU, a dedicated render box that runs its own copy of the render pipeline directly against its own REST API, not through hearth’s gateway. The gateway’s entire job is arbitrating time on the GPU box’s single model slot, and the second box was never competing for that slot in the first place, so sending its traffic through anyway would add nothing but latency.

It does share one thing with the GPU box: a short script-writing call that only the GPU box can serve, since the second box has no model of its own. That one call goes through the gateway, under its own named budget, so heavy experimentation on the render side can’t starve the budget an established job already depends on. The rest of that box’s work gets its own resource class rather than sharing the class built to stop the GPU box double-booking itself, because serializing a render job behind the GPU box’s queue would waste GPU time on hardware that was never in contention to begin with. Reachability is checked the same way hearth checks any dependency, a live probe declared once in the job’s manifest, not a guard written into the job itself.