anyesh.me

Scheduling every job on a GPU that can only hold one model

I run this homelab on two small boxes. One never turns off and just serves DNS and background services; the other has a GPU with a memory pool that can hold exactly one loaded model at a time, nothing more. This post covers the queue and gateway that constraint forced me to build, and the network shape wrapped around both boxes.

A Raspberry Pi 5 runs DNS for the network, plus dozens of other small services that don’t need much CPU or memory but can’t go down. A Jetson Orin Nano handles anything that needs a GPU, and its GPU and CPU share one memory pool, small enough to hold one loaded model or one active render job, never both.

One flat network

There’s one LAN. Clients and services aren’t split into VLANs or separated behind an internal firewall; the upstream router is the gateway, and everything else sits behind it as a peer. If one device gets compromised, there’s nothing stopping it from reaching every other device on the LAN. I’ve thought about segmenting it and decided against it: at this size, a mesh of routing rules is more to maintain than the threat justifies. It stays flat.

The Pi carries DNS and everything that has to be up all the time. It’s picked for uptime and low draw, not speed, and I don’t let it run anything that could pin its CPU long enough to take DNS down with it.

One model at a time

Load two things onto the Jetson that it can’t both hold, and it thrashes. So nothing talks to it directly: every job (a chat request, a background research task, a scheduled scrape) lands in a durable queue first, and a single gateway between that queue and the Jetson admits exactly one job at a time, by priority. That gateway is most of the engineering here. Skip it and two jobs landing at once both try to load into memory built for one, and now the box is fighting itself instead of doing either job.

Priority doesn’t preempt. If a background scrape is already loaded and running when a chat request comes in, the chat request doesn’t kill it, it queues behind it and jumps ahead of everything else waiting. The alternative, killing a running job because something more urgent showed up, throws away whatever work was already done and risks leaving the model half-loaded. Waiting for the current job to finish costs a few seconds at most; a job that has to restart from zero costs more than that most of the time.

client devicesphones, laptops, LAN clientsRaspberry Pi 5DNS + dozens of small serviceslow-power, never sleepsjob queuegatewaysingle admission, one job at a timeJetson Orin Nanoone model loaded at a timeRTX 4070 Ti Super boxdedicated render, separate hardwarerender outputDNS queryenqueues jobadmits one at a timeforwards requestrenders directlyshared LLM call only
Everything queues through one gateway before it reaches the Jetson, except the RTX box’s own render work, which never goes near it.

A second machine, with an RTX 4070 Ti Super, does rendering, and it doesn’t go through this gateway for that work. The gateway exists to arbitrate the Jetson’s single model slot, and the RTX box was never competing for that slot to begin with, so routing it through anyway would just add latency it doesn’t need. It does share one thing with the Jetson: a single LLM call both need. For that one call, the two machines are fighting over the same resource, so it queues like everything else.

The price of running it this way

A Pi 5, a Jetson Orin Nano, and a desktop GPU: together roughly the price of a decent laptop.