vLLM coding assistant stack: production deployment patterns beyond the basics

A vLLM coding assistant stack built around the high-throughput serving engine can deliver sub-second code completions at one-tenth the infrastructure cost of managed APIs—if you engineer around its quirks. Teams running production vLLM deployments wrestle with memory fragmentation under bursty traffic, cold-start latency when scaling pods, and context-cache invalidation that wastes GPU cycles. This guide walks through the architectural patterns that separate proof-of-concept demos from systems that reliably serve hundreds of developers.

The core challenge: vLLM's PagedAttention batching excels when request patterns are stable, but coding workloads spike unpredictably—morning standups, pull-request reviews, incident firefighting. You need deployment patterns that keep GPUs saturated during peaks without burning budget on idle capacity, while maintaining the <300ms p95 latency developers expect from their editor.

Architecture decisions that determine success or failure

Model selection and quantization strategy: Start with CodeLlama 13B or StarCoder2 15B quantized to AWQ 4-bit. Full-precision 34B models rarely justify their 4× memory footprint for code completion; the quality gain is marginal compared to properly tuned prompt engineering. Reserve FP16 weights for specialized tasks like architecture refactoring where semantic understanding matters more than speed.

GPU allocation and batching configuration: A single A10G (24GB) can serve 12–16 concurrent requests at batch size 32 with a 4096-token context window. Configure --max-num-batched-tokens to 90% of available memory minus your largest expected context; going higher triggers out-of-memory crashes under load. Set --max-num-seqs based on your p99 concurrent request count, not average—coding traffic is bursty.

Multi-instance vs. single-large-model tradeoffs: Running three separate vLLM instances on smaller models (7B each) across three GPUs often outperforms one 34B model for editor completions. Smaller models return first tokens faster (critical for inline suggestions), and you gain fault tolerance: one crashed instance doesn't take down the whole service. Use the large model only for chat-based refactoring where you need deeper reasoning.

Caching layers that actually reduce latency

Prompt prefix caching: vLLM's automatic prefix caching reuses KV cache for repeated context, but only if prompts share exact prefixes. Structure your prompts so file imports and class definitions appear first, then the cursor context—this maximizes cache hits across requests from the same file. With this pattern, cache hit rates climb from 12% to 68% in typical TypeScript projects.

Redis for completion deduplication: When three developers open the same file during code review, naive deployments recompute identical completions. A Redis layer keyed on hash(file_context + cursor_position) with 60-second TTL cuts redundant GPU work by 40% in our testing. Return cached completions for exact matches, fall through to vLLM for novel requests.

Embeddings-based semantic cache: For fuzzy matches (similar but not identical context), maintain a vector store of recent completions. Embed the current context, search for cosine similarity >0.92, return the previous completion if found. This catches refactored variable names and equivalent code patterns. Use a separate small embedding model (all-MiniLM) to avoid blocking the main inference path.

Scaling patterns for variable load

Horizontal pod autoscaling with custom metrics: Kubernetes HPA on GPU utilization scales too slowly for coding workloads. Instead, scale on pending_requests from vLLM's metrics endpoint: if the queue depth exceeds 8 for more than 20 seconds, launch a new pod. Warm pools (one idle pod kept ready) eliminate the 90-second GPU initialization penalty.

Request routing and load balancing: Round-robin routing wastes the benefits of prefix caching. Implement consistent hashing on repository name or file path so requests from the same codebase hit the same vLLM instance, maximizing cache utilization. Use least-connections as a fallback when cache affinity would overload a single instance.

Graceful degradation strategies: When all instances are saturated (>95% GPU memory), return fast low-quality completions from a distilled 1B model running on CPU rather than queueing requests. Developers tolerate occasional weak suggestions; they abandon tools that freeze their editor. Set a 500ms client-side timeout and fall back to the distilled model automatically.

Observability and debugging production issues

Metrics that predict problems before users complain: Track time_to_first_token (TTFT) separately from total generation time—TTFT >200ms kills the inline suggestion experience even if full completion is fast. Monitor cache_hit_ratio per instance; a sudden drop signals prompt structure changes that broke prefix alignment. Graph kv_cache_usage_percent to catch memory leaks from long-running contexts.

Logging for reproducible debugging: Log the full prompt (truncated file context included) for requests where TTFT exceeded your p95 target. When developers report "the autocomplete got slow," you need the exact input that triggered pathological batching or cache misses. Store these in structured logs queryable by timestamp and user ID.

Cost tracking beyond GPU hours: Measure cost-per-completion by dividing total GPU spend by completions served, broken down by model and quantization level. In production, AWQ 4-bit typically costs $0.0003 per completion vs. $0.0012 for FP16—4× savings that compound across millions of daily requests. Track separately for completions accepted vs. ignored to understand ROI.

Security and access control in multi-tenant deployments

Isolation strategies: If serving multiple teams or customers, run separate vLLM instances per tenant rather than sharing a single deployment. Context from one tenant's code can leak into another's KV cache, and there's no row-level security within vLLM. Use Kubernetes namespaces or separate clusters for strong boundaries.

Rate limiting and abuse prevention: Set per-user rate limits at 120 requests/minute (2 per second) to prevent runaway editor plugins from monopolizing GPU capacity. Implement exponential backoff on the client side. For API-based access, require authentication tokens and track usage by key to identify misconfigured integrations.

Model input sanitization: Strip secrets and API keys from file context before sending to vLLM, even for self-hosted deployments. Use regex patterns to detect common secret formats (AWS keys, JWTs, private keys) and replace with placeholders. Coding assistants trained on public repositories may echo secrets if they appear in context.

When to integrate vs. build from scratch

Teams with existing AI infrastructure can deploy vLLM as a microservice behind their existing auth, logging, and monitoring layers. If you're building a coding assistant from zero, platforms like Goatfied bundle AI code editing with cloud coding agents that test their own changes and open pull requests, removing the need to operate your own inference stack. The build-vs.-integrate decision typically hinges on whether you already run GPU workloads for other ML tasks—if vLLM would be your first GPU deployment, the operational overhead often exceeds the API cost savings until you reach 50+ daily active developers.

For engineering teams running their own infrastructure, vLLM slots into the stack between model registry (HuggingFace, custom fine-tunes) and developer tools (VS Code extensions, CI/CD integrations). Store quantized model weights in S3 with a CDN in front to accelerate pod startup; downloading a 7GB AWQ model from CloudFront takes 12 seconds vs. 80+ seconds from HuggingFace Hub.

Frequently asked questions

What GPU memory do I need for a team of 50 developers?

Plan for p99 concurrent requests, not average. Fifty developers generate 8–12 concurrent completion requests during peak hours (standup, lunch-return, end-of-day commits). A single A10G (24GB) with CodeLlama 13B AWQ handles this comfortably; two instances provide redundancy. Monitor your actual traffic for two weeks before committing to hardware.

How does vLLM compare to running Ollama for code completion?

vLLM's PagedAttention and continuous batching deliver 3–5× higher throughput than Ollama's simpler scheduling when serving multiple concurrent requests. Use Ollama for single-user local development; switch to vLLM when you need to serve a team. Ollama's simpler setup makes sense until you cross roughly 10 active users.

Can I run vLLM on CPU-only infrastructure?

Technically yes, but generation speed drops to 2–8 tokens/second on modern Xeon CPUs, making inline completions unusable (you need 30+ tokens/sec for responsive suggestions). CPU-only vLLM works for batch processing tasks like overnight documentation generation, not interactive coding.

What's the maintenance overhead of self-hosted vLLM vs. managed APIs?

Expect one engineer to spend 20–30% time on model updates, dependency upgrades, autoscaling tuning, and incident response. Managed APIs (Anthropic, OpenAI) eliminate this overhead but cost 8–15× more per completion. The break-even point is typically 30–40 daily active developers, depending on your infrastructure team's existing GPU experience.

How do I handle model updates without downtime?

Run blue-green deployments: spin up new pods with the updated model, shift 10% of traffic, monitor error rates for 15 minutes, then complete the rollout. Keep the old version running for 24 hours in case of regression. Store model weights in a version-controlled artifact registry so you can roll back by changing an environment variable.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide