deployment
Right-sizing GPUs for code completion workloads
Learn how to match GPU specs to code completion models by analyzing memory requirements, latency targets, and cost trade-offs for development teams.
Code completion models rarely need the most powerful GPUs available—but they almost always need some GPU acceleration. The difference between a correctly sized GPU and an oversized one is thousands of dollars per month for a team of thirty engineers. In practice, most organizations find that a single mid-range GPU (T4, L4, or similar) running a quantized 1–7B parameter model delivers sub-200ms latency while keeping costs under $300/month per instance.
This guide walks through the actual compute requirements for code completion, how to match model size and quantization to GPU memory, and the reliability patterns that determine whether you need redundancy or can tolerate occasional restarts.
Why code completion workloads differ from chat or generation
Code completion requests arrive continuously as developers type—often several per second during active coding—but each request is small. The model predicts the next 20–80 tokens based on a few hundred tokens of surrounding context, then stops. Unlike chatbots that generate multi-paragraph answers or batch inference jobs that process thousands of prompts, code completion:
- Processes short inputs and outputs: Context windows of 2,048–4,096 tokens are typical; completions rarely exceed 100 tokens
- Requires low, consistent latency: Suggestions appearing after 300ms feel sluggish; anything over 500ms disrupts flow
- Sees bursty traffic per developer: Active typing generates bursts of 5–10 requests, followed by pauses while the developer reads or thinks
These characteristics mean you optimize for throughput-per-GPU and p95 latency, not raw TFLOPS. A model that fits entirely in VRAM with room for a small KV cache will outperform a larger model that spills to system memory, even if the larger GPU has higher peak compute.
Matching model size to GPU memory
The working memory footprint is model_parameters × bytes_per_parameter + KV_cache_size. For a 3B parameter model in FP16 (2 bytes per parameter), you need 6 GB just for weights. Add 1–2 GB for the KV cache and framework overhead, and you're looking at 8 GB minimum. Quantization cuts this significantly:
| Model size | FP16 VRAM | INT8 VRAM | INT4 VRAM | Typical accuracy for code |
|------------|-----------|-----------|-----------|---------------------------|
| 1B params | ~3 GB | ~2 GB | ~1.5 GB | Good for line completion |
| 3B params | ~8 GB | ~5 GB | ~3 GB | Good general-purpose |
| 7B params | ~16 GB | ~9 GB | ~5 GB | Best context understanding|
| 13B params | ~28 GB | ~15 GB | ~8 GB | Overkill for most teams |
Example scenario: A team running StarCoder 3B in INT4 quantization needs ~3 GB for the model plus 1 GB overhead, fitting comfortably on a T4 (16 GB) or L4 (24 GB) with room to batch 2–4 concurrent requests without memory pressure.
Most teams find 3B–7B models quantized to INT8 or INT4 deliver the best latency-accuracy tradeoff. Larger models improve suggestion quality only marginally while doubling memory and halving throughput.
GPU options and their sweet spots
Entry tier: T4 (16 GB, ~8 TFLOPS FP16)
- Handles 1–3B models in INT8 or 7B models in INT4
- Serves 5–15 concurrent developers with <200ms p95 latency
- ~$0.35/hour on AWS, GCP, Azure; often free tier eligible
- Best for: teams under 20 engineers, batch workloads during off-hours
Mid-range: L4 (24 GB, ~30 TFLOPS FP16)
- Handles 7B models in INT8 or 13B in INT4 comfortably
- Serves 20–40 developers with <150ms p95
- ~$0.70–1.00/hour across clouds
- Best for: 20–100 engineer teams, production deployments
High-end: A10 (24 GB, ~125 TFLOPS FP16), A100 (40/80 GB)
- Overkill unless you're serving 100+ developers per GPU or running multiple models
- Useful if the same GPU serves code completion and PR review or documentation generation
- 3–5× the cost of L4 for 2× the throughput in typical workloads
Multi-GPU and horizontal scaling
A single GPU typically saturates around 50–80 concurrent developers. Beyond that, add GPUs horizontally behind a load balancer rather than using multi-GPU on one instance—simpler networking, better fault isolation, easier autoscaling.
Quantization tradeoffs for code tasks
Quantization reduces memory and increases throughput with minimal accuracy loss if the model was trained or fine-tuned on code. Generic LLMs quantized below INT8 often produce syntactically broken completions, but models like CodeGen, StarCoder, or DeepSeek Coder handle INT4 well:
- FP16: Baseline accuracy, use only if VRAM allows and latency meets target
- INT8: 1–2% accuracy drop, 40–50% memory savings, negligible latency impact—recommended default
- INT4: 3–5% accuracy drop, 70–75% memory savings, 10–15% faster throughput—best for cost-sensitive deployments
Run a blind A/B test with your team: developers rarely distinguish INT8 from FP16 in practice, and INT4 is acceptable when completions are treated as suggestions rather than auto-commits.
Latency budgets and batching strategies
Target p95 latency under 200ms (measured from request arrival to first token). Breakdown for a 3B model on L4:
- Tokenization: 5–10ms
- Inference (unbatched): 30–60ms
- Decoding: 40–80ms
- Network + serialization: 20–40ms
Total: 95–190ms, leaving headroom for queueing. With dynamic batching (grouping 2–4 requests that arrive within 10ms), you can increase throughput by 50% while keeping p95 under 250ms. Beyond batch size 4, latency degrades faster than throughput improves.
Disable batching entirely if your team is under 10 developers—single-request latency will be 30% lower and the complexity isn't worth it.
Reliability and failover considerations
Code completion is latency-sensitive but failure-tolerant. If the model is unavailable, developers lose autocomplete for a few minutes—annoying but not blocking. This changes your infrastructure tradeoffs:
- Single GPU: Acceptable for teams under 50 if you can tolerate 5–10 minute recovery from instance failure
- Active-active pair: Two GPUs behind a load balancer with health checks; one can restart or update without user impact
- Autoscaling: Overkill unless traffic varies 3× between peak and off-hours (rare for internal tools)
Most teams run a single GPU for months, then add a second only after the first outage causes complaints. For platforms like Goatfied's BUILD environment where cloud coding agents generate and test changes continuously, a second GPU prevents agent queue buildup during model restarts.
Cost modeling and budget allocation
Example budget for a 30-person engineering team using a 3B model on L4:
- L4 instance (24/7): ~$500–700/month
- Egress (if multi-region): ~$20–50/month
- Monitoring and logs: ~$10–20/month
Total: ~$550–800/month, or $18–27 per developer per month—significantly cheaper than most SaaS code completion products at $10–20 per seat that run similar infrastructure in the background.
Scaling to 100 developers typically means adding one more L4 (~$600/month additional), bringing per-developer cost to ~$12–18/month. At 500+ developers, negotiate reserved instances or committed use discounts for 30–50% savings.
Frequently asked questions
Can I use CPU-only for code completion?
Yes, but expect 2–5× higher latency (500ms–2s). Acceptable for occasional use or tiny teams (under 5 developers), but most engineers find it too slow for flow state. A single T4 pays for itself in retained focus within weeks.
How much VRAM do I need for fine-tuning vs. inference?
Fine-tuning needs 3–4× the VRAM of inference because you store activations for backpropagation. A 3B model that runs inference in 8 GB needs ~24 GB to fine-tune. Most teams fine-tune on A100s or batch processes on CPU, then deploy quantized weights to smaller GPUs.
Do I need different GPUs for different languages?
No. The same model and GPU handle Python, TypeScript, Go, Rust, etc. Language-specific accuracy depends on training data mix, not hardware. If one language underperforms, retrain or swap models, not GPUs.
Should I run code completion and CI/CD agents on the same GPU?
Only if both workloads are bursty and non-overlapping. Code completion needs consistent low latency; batch CI tasks (test generation, PR summaries) can tolerate queueing. Mixing them saves cost but complicates capacity planning. Separate GPUs are simpler unless budget is very tight.
How do I measure whether my GPU is right-sized?
Track GPU memory utilization (should be 60–85%), p95 request latency (target <200ms), and queue depth (should be 0–2 most of the time). If memory is under 50%, downsize. If p95 exceeds 250ms or queue depth stays above 5, add capacity or enable batching.