Cutting GPU inference costs 60–80% with a CPU/GPU split

Leaving model services running on always-on GPUs is the quickest way to torch a cloud bill. The GPU sits there costing money whether it's doing anything or not. On a recent production system I split the stack so that stopped happening.

The setup

A lightweight CPU backend handles all the boring but constant stuff — API traffic, orchestration, request handling. The actual model work runs on GPU services that only spin up when there's something to do. Between the two sit Redis queues and background workers: requests get enqueued on the CPU side, and workers pull them and route to a GPU only when there's real work waiting.

When the queue drains, the GPUs idle down. No idle GPU, no idle bill.

What it got us

Running GPUs on demand instead of around the clock cut infrastructure costs by roughly 60–80%. And because the queue absorbs bursts, the system actually held up better under real traffic than the always-on version did.

The lesson I keep coming back to: most of the time your expensive hardware isn't doing anything. Build the system so it only pays for the moments it is.