GPU economics for self-hosted code models

Learn what GPUs cost for self-hosted code models, how to size hardware, and when cloud or managed options save money.

Running a self-hosted code model means choosing between fast inference and burning money. A single H100 GPU rents for around $2–3 per hour on major clouds; a full 8-GPU node pushes $20 per hour. If your team writes code eight hours a day, five days a week, you're looking at $3,200–$16,000 monthly just for compute—before storage, bandwidth, or the engineering time to tune, monitor, and keep the system fed with requests. Understanding GPU economics isn't optional; it's the difference between a tool that pays for itself and a line item your CFO flags every quarter.

This guide breaks down the cost drivers, trade-offs, and tactics for running self-hosted code models on GPUs, so you can predict spend, size hardware correctly, and decide when cloud-hosted or managed alternatives make more sense.

Why GPUs matter for code models

Code models—whether you're running DeepSeek Coder, CodeLlama, or a fine-tuned variant—are transformer networks with billions of parameters. Inference means multiplying massive weight matrices by input tokens, then decoding output tokens one at a time. CPUs can technically do this work, but a single inference request might take tens of seconds; GPUs parallelize the matrix math and deliver answers in under a second.

Batch size and throughput: A GPU can process multiple requests in parallel. If you send one request at a time, you're underutilizing the hardware; if you queue hundreds, latency climbs. The sweet spot depends on model size, context length, and how often developers actually wait for completions.

Memory bandwidth: Large models (30B+ parameters in FP16) exceed the VRAM of consumer cards. You'll need A100s (40 or 80 GB), H100s (80 GB), or multi-GPU setups with tensor parallelism, which adds network overhead and doubles the number of GPUs you pay for.

Cost structure: hardware, power, and utilization

Instance rental: Cloud providers charge by the GPU-hour. An A100 80 GB instance typically costs $3–$4/hour; an H100 closer to $4–$5/hour. On-premises purchase shifts the model: an H100 retails around $30,000–$40,000, so break-even versus cloud happens after roughly 6,000–10,000 hours (about one to two years of continuous operation).

Power and cooling: A single H100 draws 700 watts; a full rack of eight draws 5.6 kW. At $0.10/kWh, that's $0.56/hour in electricity alone, plus cooling overhead. Data centers budget 1.3–1.5× power for cooling, so real opex climbs to roughly $0.80/hour per 8-GPU node—small compared to cloud rental but significant when you own the hardware.

Utilization: A GPU sitting idle still costs money. If your team works 40 hours a week, that's 24% utilization over a 168-hour week. Cloud rental makes sense at low or spiky usage; ownership pays off only when you can keep GPUs busy or batch requests from many teams.

Model size, quantization, and memory trade-offs

Parameter count: A 7B model in FP16 needs roughly 14 GB VRAM (two bytes per parameter); a 34B model needs 68 GB. Quantization to int8 halves memory (and slightly degrades quality); int4 quarters it but loses more precision. For code, where a wrong token can break syntax, int8 is often the floor.

Context window: Longer contexts (16k, 32k tokens) let the model see entire files but grow the key-value cache quadratically. A 16k-token context for a 34B model can add another 20–30 GB of VRAM during inference. If you frequently autocomplete across large codebases, budget memory accordingly.

Example sizing: A team running a 34B model with 8k context in int8 quantization needs about 40 GB VRAM per request. An A100 80 GB can handle one or two concurrent requests; an H100 80 GB slightly more. To serve ten developers hammering the model simultaneously, you'd need multiple GPUs or a smaller model.

Batching, queuing, and latency

Static batching: Collect requests until you have a full batch (say, 8 or 16), then run inference. Throughput improves, but the first request waits for the batch to fill—unacceptable for interactive autocomplete.

Continuous batching (also called iteration-level batching): Start decoding tokens for request A; when request B arrives mid-decode, fold it into the same forward pass. Libraries like vLLM and TensorRT-LLM implement this, cutting latency and boosting GPU utilization. Expect 2–4× better throughput versus naïve batching.

Queuing theory: If requests arrive faster than the GPU can process them, queue depth grows and latency spikes. Monitor P95 and P99 latency; once they cross acceptable thresholds (typically 500 ms–1 s for autocomplete), you need more GPUs or a smaller model.

When self-hosting pays off

High, steady volume: If twenty engineers each generate fifty completions per hour, eight hours a day, you're making 8,000 requests daily. At that scale, a dedicated GPU node amortizes fixed costs better than pay-per-token APIs.

Data residency: Some organizations cannot send proprietary code to third-party APIs. Self-hosting keeps embeddings, training data, and inference logs on-premises or in a private VPC.

Customization: Fine-tuning on your codebase, tweaking sampling parameters, or swapping models weekly is simpler when you control the stack. Managed services may lock you into specific model versions or update schedules.

When managed or cloud-hosted makes sense

Variable load: Startups and small teams often see spiky usage—heavy during sprints, quiet on weekends. Renting GPUs by the hour or using a hosted API (pay per token) avoids idle capacity charges.

No ML ops team: Running inference servers, monitoring VRAM leaks, updating CUDA drivers, and tuning kernel launches takes engineering time. If your team would rather ship features than manage infrastructure, a platform that abstracts the GPU layer saves headcount. Goatfied's BUILD suite includes cloud coding agents that handle inference, testing, and pull requests without requiring you to provision or tune GPUs—one less stack to manage.

Capital constraints: Buying eight H100s ($240,000–$320,000) ties up cash; cloud rental turns capex into opex and preserves runway for product development.

Measuring and optimizing cost per completion

Track tokens processed per dollar: divide total tokens generated by total GPU cost (rental + power + network). If you're getting 100,000 tokens per dollar on a self-hosted A100 and a managed API charges $0.10 per 1,000 tokens (10,000 tokens per dollar), self-hosting is 10× cheaper—but only if you account for engineering overhead and idle time.

Profiling: Use PyTorch Profiler or NVIDIA Nsight Systems to identify bottlenecks. Common culprits include CPU-GPU transfer (pin memory and use async copies), inefficient sampling (top-k/top-p kernels), and memory fragmentation (restart inference servers daily if VRAM use creeps up).

Right-sizing: A 7B model might deliver 90% of the accuracy of a 34B model for autocomplete. Test on real pull requests; if the smaller model passes the same percentage of CI checks, you've just freed 70 GB of VRAM.

Example cost comparison

Consider a 10-person engineering team generating 5,000 completions per day, averaging 200 tokens per completion (1 million tokens daily).

Cloud API at $0.10/1k tokens: $100/day, $3,000/month.
Self-hosted on one A100 80 GB (cloud rental, $3/hour, 24/7): $2,160/month, plus engineering time to maintain the stack.
Self-hosted on owned hardware (one H100, amortized over two years): ~$1,400/month hardware cost + $120/month power = $1,520/month, assuming continuous operation and no idle waste.

These are illustrative numbers; actual costs depend on model size, quantization, request concurrency, and whether you can batch requests effectively. Always benchmark with your traffic patterns.

Infrastructure checklist

When evaluating GPU deployment, confirm:

VRAM headroom: Model weights + KV cache + batch overhead should leave 10–20% free to avoid OOM crashes.
Network bandwidth: Multi-GPU setups need NVLink or InfiniBand; standard Ethernet bottlenecks tensor parallelism.
Failover: If a GPU node dies mid-sprint, can requests route to a backup, or does the team lose autocomplete until you restart?
Observability: Log request latency, queue depth, VRAM usage, and GPU utilization. Set alerts when P99 latency crosses SLA thresholds.
Access control: Who can submit inference requests? Unrestricted access risks a single developer's script monopolizing the GPU and blocking everyone else.

Goatfied's IT suite helps teams track issues, sprints, and service desk requests in one place, so when GPU-related incidents happen, your on-call engineer sees context immediately instead of hunting across Slack and ticketing tools.

Frequently asked questions

How much VRAM do I need for a 34B parameter model?

In FP16, roughly 68 GB for weights alone. Add 20–40 GB for the key-value cache if you support long contexts (16k+ tokens), plus overhead for batching. An A100 80 GB or H100 80 GB handles one to two concurrent requests; for more, use quantization (int8 cuts memory in half) or split the model across multiple GPUs.

Can I mix different GPU types in the same cluster?

Technically yes, but tensor parallelism and pipeline parallelism expect uniform memory and bandwidth. Mixing an A100 and an H100 forces the faster card to wait for the slower one. Use homogeneous nodes or route different models to different GPU types.

What's the break-even point between cloud rental and buying GPUs?

Divide the purchase price by the hourly rental rate. An H100 at $35,000 and $4/hour rental breaks even after 8,750 hours—about one year of 24/7 operation. If your utilization is lower (say, 40 hours/week), break-even stretches to over four years, and cloud rental usually wins.

How do I prevent one developer from monopolizing the GPU?

Implement request quotas (tokens per user per hour), priority queues (escalate production requests over exploratory ones), or time-based scheduling (reserve GPU for CI/CD during builds). Monitoring tools should alert when a single user accounts for >50% of requests.

Should I fine-tune or just prompt a base model?

Fine-tuning on your codebase improves autocomplete accuracy but requires labeled data, GPU time for training, and periodic re-training as the codebase evolves. Prompting (few-shot examples in the context) is simpler but consumes more tokens per request. Start with prompting; if accuracy lags, collect a few hundred high-quality completions and fine-tune a LoRA adapter.