vLLM coding assistant stack: self-hosting from scratch

Building a self-hosted vLLM coding assistant stack gives you full control over model inference, data privacy and cost structure. The core setup pairs vLLM—a high-throughput LLM serving engine—with a code-completion client like Continue or Tabby, all running on infrastructure you manage. A typical stack serves models such as CodeLlama, StarCoder or DeepSeek-Coder at latencies competitive with commercial APIs, but keeps source code and telemetry entirely inside your network perimeter.

This guide walks through hardware requirements, runtime configuration, model selection, client integration and observability. You'll finish with a production-ready coding assistant that your team can deploy in hours rather than weeks.

Why vLLM for code inference

vLLM uses PagedAttention to manage KV cache memory efficiently, delivering 10–25× higher throughput than naïve PyTorch serving under concurrent load. For coding workloads, this translates to sub-200 ms token latencies even when a dozen developers trigger completions simultaneously. The engine supports batched requests, streaming and speculative decoding, so you can squeeze the most from each GPU hour.

Unlike closed-loop APIs, vLLM lets you choose any GGUF- or safetensors-format model, pin the version and audit weights. You control telemetry, so no code snippets leave your VPC. For teams in regulated industries—finance, healthcare, government—this architecture satisfies data-residency and audit requirements that third-party APIs cannot meet.

Hardware and infrastructure requirements

GPU recommendations:

| Use case | Minimum GPU | Recommended GPU | Context window |

|------------------------|--------------------|-----------------------|----------------|

| Single developer | RTX 4090 (24 GB) | A10G (24 GB) | 8k tokens |

| Team (5–15 engineers) | L4 (24 GB) | A100 40 GB | 16k tokens |

| Large team (50+) | A100 80 GB | H100 80 GB × 2 | 32k+ tokens |

For proof-of-concept, an RTX 4090 desktop node suffices. Production deployments typically run on cloud instances (AWS g5.xlarge, GCP g2-standard-8, Azure NC-A100) or on-premise servers with NVIDIA data-center GPUs. Aim for 1.5× model size in VRAM to leave headroom for KV cache and batch processing.

CPU and memory:

Allocate 16 CPU cores and 64 GB system RAM per GPU. vLLM's request scheduler and tokenizer pre-processing are CPU-bound, so under-provisioning RAM triggers swap and destroys latency.

Storage:

SSD or NVMe with at least 200 GB free. Model weights (7B–34B parameters) consume 15–70 GB, and Docker images, logs and temporary batches add another 50 GB.

Selecting and downloading a code model

Choose a model trained on code repositories and permissively licensed:

• CodeLlama 13B Instruct (Llama 2 license) — strong fill-in-the-middle, good for Python and TypeScript.
• StarCoder2 15B (OpenRAIL-M) — trained on 600+ programming languages, excels at less common syntaxes.
• DeepSeek-Coder 33B (MIT license) — high accuracy on HumanEval, but requires 80 GB VRAM or quantization.

Download weights from Hugging Face:

pip install huggingface-hub

huggingface-cli login

huggingface-cli download codellama/CodeLlama-13b-Instruct-hf \

  --local-dir ./models/codellama-13b

For faster startup, use pre-quantized GPTQ or AWQ variants at 4-bit precision. Quantization cuts memory by ~60 % with negligible accuracy loss for completion tasks.

Installing and configuring vLLM

Docker approach (recommended for reproducibility):

docker pull vllm/vllm-openai:latest



docker run --gpus all -p 8000:8000 \

  -v ./models:/models \

  vllm/vllm-openai:latest \

  --model /models/codellama-13b \

  --dtype bfloat16 \

  --max-model-len 8192 \

  --gpu-memory-utilization 0.85

Native install:

pip install vllm

python -m vllm.entrypoints.openai.api_server \

  --model codellama/CodeLlama-13b-Instruct-hf \

  --dtype bfloat16 \

  --max-model-len 8192

Key parameters:

• --max-model-len: Cap context to fit VRAM; 8192 is safe for 24 GB GPUs.
• --gpu-memory-utilization 0.85: Reserve 15 % for CUDA overhead.
• --dtype bfloat16: Faster than float16 on Ampere/Hopper; fall back to float16 on older architectures.
• --tensor-parallel-size 2: Split model across two GPUs if needed.

The server exposes an OpenAI-compatible /v1/completions endpoint at http://localhost:8000, simplifying client integration.

Integrating with code editors

Continue (VS Code and JetBrains)

Install the Continue extension, then add to ~/.continue/config.json:

{

  "models": [

    {

      "title": "CodeLlama Local",

      "provider": "openai",

      "model": "codellama/CodeLlama-13b-Instruct-hf",

      "apiBase": "http://localhost:8000/v1",

      "apiKey": "none"

    }

  ],

  "tabAutocompleteModel": {

    "title": "CodeLlama Local",

    "provider": "openai",

    "model": "codellama/CodeLlama-13b-Instruct-hf",

    "apiBase": "http://localhost:8000/v1"

  }

}

Restart the editor; tab-completion and chat now route through vLLM.

Tabby (self-hosted server)

Tabby wraps vLLM with a web UI, Git integration and telemetry dashboard. Deploy via Docker Compose:

services:

  vllm:

    image: vllm/vllm-openai:latest

    command: --model /models/codellama-13b --dtype bfloat16

    volumes:

      - ./models:/models

    deploy:

      resources:

        reservations:

          devices:

            - capabilities: [gpu]

  

  tabby:

    image: tabbyml/tabby:latest

    ports:

      - "8080:8080"

    environment:

      TABBY_MODEL_ENDPOINT: http://vllm:8000/v1

Point your IDE plugin to http://localhost:8080.

Monitoring and observability

Track tokens per second, queue depth and GPU utilization to identify bottlenecks.

Prometheus metrics:

vLLM exposes /metrics in OpenMetrics format. Scrape request latency (vllm:request_latency_seconds), batch size and cache hit rate.

Example Grafana dashboard queries:

• P95 latency: histogram_quantile(0.95, rate(vllm:request_latency_seconds_bucket[5m]))
• Throughput: rate(vllm:request_success_total[1m])

If you already use an observability suite like Goatfied's Observability tools—which ingest OpenTelemetry metrics, logs and traces—you can pipe vLLM's Prometheus endpoint into the same dashboards that monitor your application stack.

Log aggregation:

Ship vLLM container logs to Loki or Elasticsearch. Watch for CUDA out of memory errors, which signal that --max-model-len or batch size exceeds VRAM.

Scaling and high availability

Horizontal scaling:

Run multiple vLLM replicas behind an NGINX or HAProxy load balancer. Each replica serves independent requests, so throughput scales linearly with GPU count.

Kubernetes deployment:

Use the NVIDIA GPU Operator and a StatefulSet for persistent model volumes. Example manifest snippet:

resources:

  limits:

    nvidia.com/gpu: 1

volumeMounts:

  - name: models

    mountPath: /models

Deploy a headless service and update client apiBase to the cluster DNS name.

Fault tolerance:

Health-check the /health endpoint every 10 seconds. If a replica crashes (OOM, driver hang), Kubernetes reschedules it automatically. Store model weights on a shared NFS or S3-mounted volume so new pods start in under a minute.

Security and compliance

• Network isolation: Bind vLLM to 127.0.0.1 or a private subnet; expose via VPN or internal load balancer.
• Authentication: Wrap the endpoint with an API gateway (Kong, Traefik) that enforces OAuth2 or mutual TLS.
• Audit logs: Log every completion request with user ID and timestamp. If your organization uses a unified IT suite for service-desk tickets and compliance tracking, route vLLM access logs there for centralized audit trails.
• Model provenance: Document the source, license and SHA-256 hash of every model file. This matters for SOC 2, ISO 27001 and FedRAMP pipelines.

Cost comparison

Self-hosted (example):

AWS g5.2xlarge (A10G, 24 GB) costs ~$1.20/hour on-demand, or $0.35/hour reserved. At 40 hours/week for a five-person team, monthly compute is ~$60–$200, plus storage and egress.

Managed APIs (illustrative):

A commercial coding assistant might charge $0.10–$0.50 per thousand tokens. A team generating 10 million completion tokens monthly would spend hundreds to thousands of dollars, depending on the tier.

Self-hosting breaks even when your team crosses 10–15 active developers or when data-residency requirements rule out third-party APIs. The vLLM stack also lets you experiment with newer open-weight models the day they release, rather than waiting for vendor support.

Frequently asked questions

Can I run vLLM on Apple Silicon or AMD GPUs?

vLLM relies on CUDA and cuDNN, so NVIDIA GPUs are required. For Apple Silicon, use MLX or llama.cpp instead. AMD ROCm support is experimental; check the vLLM repository for the latest compatibility matrix.

How do I upgrade to a newer model without downtime?

Deploy the new model on a second vLLM instance at a different port, verify accuracy with a test suite, then update your load balancer or client config. Once traffic shifts, terminate the old instance. Blue-green deployments eliminate the maintenance window.

What's the latency difference between vLLM and commercial APIs?

On equivalent hardware, vLLM's token latency matches or beats managed endpoints because you avoid internet round-trips. Network proximity (same data center, same rack) can drop P99 latency from 300 ms to under 100 ms.

Do I need to fine-tune the model on my codebase?

Not for general-purpose completions. Fine-tuning helps if your team uses domain-specific libraries, internal frameworks or uncommon languages. Most teams achieve good results with a pre-trained instruction model and retrieval-augmented context from documentation.

How do I handle model updates and version control?

Tag model directories by release date or Git commit. Store metadata (model card, training recipe, benchmark scores) alongside weights. Use Hugging Face's transformers-cli or DVC to version large files, and script the download so your CI/CD pipeline can reproduce deployments deterministically.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide