deployment
Deploying a coding assistant to Kubernetes without leaking source
Learn how to deploy AI coding assistants on Kubernetes while isolating source code, controlling access, managing GPUs, and monitoring interactions securely.
Running an AI coding assistant in your Kubernetes cluster keeps your source code on your infrastructure, but introduces new security, cost and reliability challenges. The key is isolating the AI runtime from your codebase, controlling network access, managing GPU allocation, and logging every code interaction—while keeping latency low enough for developers to stay productive.
This guide walks through the architecture, security controls, resource planning and operational steps to deploy a coding assistant on Kubernetes without exposing your source code. You'll learn what to lock down, how to structure namespaces and secrets, and which metrics to monitor.
Why deploy coding assistants to Kubernetes
A coding assistant that runs in your Kubernetes environment gives you control over where code flows and who can access it. Instead of sending snippets to a third-party API, the model inference happens on nodes you operate. This matters when you work with proprietary algorithms, unreleased features, customer data embedded in test fixtures, or compliance regimes that prohibit external data processing.
Kubernetes also lets you scale inference capacity across teams, share GPU pools, roll out updates with zero downtime, and enforce uniform security policies through namespaces, network rules and service meshes.
Choosing between open-weight models and managed APIs
You have two architectural options: host an open-weight model (Llama, CodeLlama, StarCoder, DeepSeek Coder) on your own cluster, or route requests through a managed API that runs in your private cloud account.
Open-weight models run entirely on your infrastructure. You control the model version, fine-tuning data, and inference parameters. The tradeoff is operational overhead: you provision GPU nodes, manage model downloads, tune batch sizes, handle restarts, and monitor cache hit rates.
Managed APIs in your cloud account (Azure OpenAI in a private VNET, AWS Bedrock with VPC endpoints, Google Cloud Vertex AI) keep data within your cloud perimeter but delegate runtime management to the vendor. You skip GPU cluster management and get automatic scaling, but you depend on the vendor's update schedule and feature set.
If your organization already runs ML workloads on Kubernetes and has GPU node expertise, hosting your own model offers maximum control. If you want faster deployment and fewer moving parts, a managed API behind a service endpoint is simpler.
Isolating the assistant from source code
The core security principle is separation: the AI assistant should never read your Git repositories or file systems directly. Instead, it receives code only through a controlled gateway that applies access checks and logs requests.
Create a dedicated namespace for the assistant. Apply a network policy that allows inbound traffic only from your code editor service and outbound traffic only to the model endpoint. Deny all other ingress and egress. This prevents lateral movement if the assistant service is compromised.
Use a request gateway between the editor and the model. The gateway authenticates each request with the developer's identity, checks repository permissions, redacts secrets, and strips sensitive comments before forwarding the context window to the model. It also truncates responses if they contain credential patterns or internal URLs.
Store secrets in Kubernetes secrets or an external vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Mount them as environment variables or files, never bake them into container images. Rotate model API keys and cluster credentials on a schedule.
Log every request and response to an immutable audit trail. Include the developer ID, timestamp, repository path, prompt tokens, completion tokens, and whether the request was allowed or blocked. Feed these logs into your SIEM or observability stack for anomaly detection.
Resource planning and GPU allocation
AI inference workloads spike unpredictably—one developer runs a refactor across fifty files, another requests a single completion. You need enough capacity to keep latency below two seconds (the threshold where developers start context-switching) without paying for idle GPUs.
Size your GPU nodes based on the model. A 7B-parameter model in fp16 needs roughly 14 GB VRAM; a 34B model needs 68 GB. Use NVIDIA A10G, A100, H100, or AMD MI250 instances depending on your cloud provider and budget. Start with two nodes to handle failover, then add capacity as adoption grows.
Enable autoscaling with the Kubernetes cluster autoscaler and horizontal pod autoscaler (HPA). Define an HPA that scales inference pods based on request queue depth or average latency. Set the cluster autoscaler to provision new GPU nodes when pod requests exceed capacity.
Use node affinity and taints to ensure inference pods land only on GPU nodes and don't evict other workloads. Taint GPU nodes with nvidia.com/gpu=true:NoSchedule and add a matching toleration to your inference pod spec.
Monitor GPU utilization with NVIDIA DCGM exporter or AMD SMI metrics. Track VRAM usage, GPU utilization percentage, temperature, and throttling events. Alert when utilization stays below 30% for extended periods (you're over-provisioned) or above 90% (developers will see slow responses).
Deployment steps
1. Provision a Kubernetes cluster with GPU-enabled node pools. Confirm the NVIDIA device plugin or AMD GPU operator is installed and reporting devices.
2. Create a namespace for the coding assistant: kubectl create namespace coding-assistant.
3. Deploy the model server. If using an open-weight model, run an inference server like vLLM, TGI (Text Generation Inference), or TensorRT-LLM. Mount the model weights from a persistent volume or object storage bucket. Example deployment snippet:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-inference
namespace: coding-assistant
spec:
replicas: 2
template:
spec:
tolerations:
- key: nvidia.com/gpu
operator: Equal
value: "true"
effect: NoSchedule
containers:
- name: vllm
image: vllm/vllm-openai:latest
resources:
limits:
nvidia.com/gpu: 1
env:
- name: MODEL_NAME
value: codellama/CodeLlama-13b-hf
```
4. Deploy the request gateway as a separate service. This proxy authenticates incoming requests, fetches repository permissions from your identity provider, sanitizes prompts, and forwards to the model server. Expose it only within the cluster or via an internal load balancer.
5. Configure network policies to restrict traffic. Example policy that allows only the gateway to reach the model:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-gateway-to-model
namespace: coding-assistant
spec:
podSelector:
matchLabels:
app: llm-inference
ingress:
- from:
- podSelector:
matchLabels:
app: request-gateway
```
6. Enable audit logging at the gateway. Ship logs to your observability platform. Track request volume, latency percentiles, error rates, and token counts per developer.
7. Test with a small group of developers. Measure time-to-first-token and throughput. Adjust replica counts, batch sizes, and cache settings until latency meets your target.
8. Roll out to the organization. Communicate usage guidelines, explain what data stays internal, and provide feedback channels.
Monitoring and operational concerns
Watch for model staleness: open-weight models don't update themselves. Plan quarterly reviews to evaluate newer releases and decide whether to retrain or replace your model.
Monitor for prompt injection attempts. Developers might accidentally paste malicious prompts from external sources. Your gateway should detect and block patterns that try to override system instructions or exfiltrate data.
Track cost per request by multiplying token counts with your infrastructure cost per GPU-hour. Compare this with the value of time saved. If cost per completion exceeds the median engineer's per-minute salary, tune batch sizes or consider a smaller model.
Back up your audit logs and model weights. Treat the assistant deployment with the same rigor as production services: version control your Kubernetes manifests, test rollbacks, and define an incident response plan.
Using Goatfied for broader AI workflows
Goatfied combines an AI code editor with cloud coding agents that test their changes and open pull requests on GitHub, Bitbucket or Azure DevOps. Instead of managing your own Kubernetes cluster for AI workloads, Goatfied handles the infrastructure while giving you control over which GOATs access your repositories and how they escalate decisions. Each GOAT has a defined mission, playbooks, and guardrails, so you can approve or revise their work before code ships. The platform integrates into your existing IT workflows with one sign-in, one permission model, and one audit log.
Frequently asked questions
How much GPU memory do I need for a coding assistant?
A 7B-parameter model in half-precision requires roughly 14 GB VRAM, so a single NVIDIA A10G (24 GB) or T4 (16 GB) suffices. A 13B model needs around 26 GB, fitting an A10G or A100 (40 GB). For 34B models, use an A100 (80 GB) or split across multiple GPUs with tensor parallelism. Add headroom for the KV cache and batch processing—typically 1.5× the base model size.
Can I fine-tune the model on my organization's code?
Yes. Collect a training set from your repositories (with permission checks), fine-tune using LoRA or full parameter tuning on your GPU nodes, then serve the adapted model. Store training data and model checkpoints in encrypted persistent volumes. Re-run fine-tuning quarterly to incorporate new code patterns. Always anonymize or strip sensitive data before training.
What latency should I target for code completions?
Aim for time-to-first-token under 500 milliseconds and full completion under two seconds. Beyond two seconds, developers lose flow and stop using the assistant. Use quantization (int8, int4) and speculative decoding to reduce latency. Monitor the 95th percentile; outliers indicate resource contention or cold starts.
How do I prevent the model from memorizing secrets?
Scan your training data for API keys, tokens, and credentials before fine-tuning. Use regex patterns and secret detection tools (Gitleaks, TruffleHog) to redact matches. At inference time, run a post-processing filter on completions to block patterns that look like secrets. Log any detections and alert your security team.
Should I run the assistant in the same cluster as production workloads?
Only if you enforce strict resource quotas and network isolation. A safer pattern is a dedicated cluster or namespace with its own node pool. This prevents inference spikes from starving production pods and limits blast radius if the assistant is compromised. Use separate RBAC roles and API server authentication for the assistant cluster.