deployment
Observability for AI coding assistants in production
Learn how to instrument AI coding assistants to track latency, costs, error patterns, and quality issues across the LLM inference pipeline in production environments.
Large language models powering coding assistants in production environments generate thousands of requests per day, yet most teams lack visibility into cost, latency, error patterns and quality. When a developer's autocomplete suggestion takes four seconds or a code generation agent repeatedly suggests deprecated APIs, you need telemetry to diagnose whether the problem lies in prompt engineering, model choice, context retrieval or infrastructure.
Observability for LLMs in production captures metrics, logs and traces across the inference pipeline—from the moment a user triggers a completion to the final token streamed back. This article walks through the instrumentation points, key signals to track, and practical patterns for debugging performance and cost issues in AI coding tools.
Why LLM observability differs from traditional APM
Application performance monitoring evolved for deterministic systems where the same input reliably produces the same output. LLM-based coding assistants introduce non-deterministic behavior, token-based pricing, and a layered architecture that spans prompt construction, retrieval-augmented generation, model inference and post-processing.
Cost attribution becomes non-trivial. A single code completion might consume 2,000 input tokens (file context, recent edits, documentation snippets) and generate 150 output tokens. Without request-level tracking, you cannot map spend to specific teams, repositories or feature flags.
Latency has multiple contributors. Time-to-first-token (TTFB) measures how quickly the model begins responding; total generation time depends on output length and sampling strategy. A P95 latency of three seconds might hide that 5% of requests wait eight seconds for context retrieval before inference even starts.
Quality is harder to instrument than uptime. A 200 OK response does not mean the suggestion compiled, matched the user's intent or avoided a known vulnerability. Observability must capture not just success rates but downstream signals like acceptance rate, edit distance and linting failures.
Instrumentation points across the LLM request lifecycle
A complete observability strategy instruments five stages:
1. Trigger and context assembly. Log the event type (inline completion, chat message, refactor request), cursor position, file type and how much context was retrieved from vector stores, recent commits or documentation.
2. Prompt construction. Record the final prompt token count, template version and any dynamic few-shot examples injected. This lets you correlate prompt changes with quality shifts.
3. Model inference. Capture the model identifier, region, input/output token counts, sampling parameters (temperature, top-p, max tokens), time-to-first-token and total latency. For self-hosted models, include GPU utilization and queue depth.
4. Post-processing. Track parsing steps (extracting code blocks, applying diff formatting), any filtering rules (profanity checks, license scans) and the final character count delivered to the client.
5. User interaction. Measure acceptance rate (did the user press Tab?), dwell time (how long before dismissing or accepting), edit distance if the user modified the suggestion, and whether the code passed linting or compilation.
Goatfied's Observability suite ingests OpenTelemetry metrics, logs and traces, letting teams visualize LLM request flows alongside traditional application telemetry in a single dashboard. Alerts fire when P95 latency crosses thresholds or token consumption spikes unexpectedly.
Metrics that matter for production LLM systems
Start with these core metrics, segmented by model, feature and user cohort:
| Metric | Definition | Why it matters |
|--------|-----------|----------------|
| Requests per minute | Total inference calls | Capacity planning and rate limit headroom |
| Token throughput | Input + output tokens/sec | Cost forecasting and quota management |
| P50, P95, P99 latency | Time from request to completion | User experience; identifies outliers |
| Time-to-first-token | Milliseconds until first output token | Streaming UX and perceived responsiveness |
| Error rate by status code | 4xx (client) vs 5xx (server) | Distinguishes bad prompts from infra failures |
| Acceptance rate | % of suggestions accepted | Proxy for relevance and quality |
| Cost per request | Tokens × price per million | ROI and budget adherence |
Example: A team notices acceptance rate dropped from 32% to 19% over two weeks. Traces reveal a deployment that increased context window from 4K to 16K tokens, injecting less relevant file snippets. Rolling back the context expansion restored quality without changing the model.
Log correlation and distributed tracing
Unlike stateless HTTP APIs, LLM requests often involve multi-step workflows: a vector search for similar code, a cache lookup for repeated prompts, inference, then a syntax validation pass. Distributed tracing stitches these spans together.
Correlate logs with request IDs. Every LLM call should carry a unique identifier that appears in context retrieval logs, model provider logs, client-side telemetry and user feedback events. When a developer reports a bad suggestion, you can replay the exact prompt, retrieved snippets and model response.
Trace parent-child relationships. A "refactor function" request might spawn five LLM calls—one to generate a plan, four to rewrite modules. Parent trace IDs let you measure end-to-end latency and identify which sub-step dominates cost.
Annotate spans with semantic metadata. Tag traces with repository name, programming language, whether the user is on a free or paid tier, and feature flags. This enables filtering to questions like "Why is Go completion 2× slower than Python?" or "Which teams are hitting rate limits?"
Handling sensitive data in logs
LLM observability must balance debuggability with privacy. Source code, API keys embedded in prompts, and personally identifiable information (PII) in chat messages require careful handling.
Redact before export. Strip secrets, credentials and email addresses from prompts before writing to centralized logging. Use allow-lists (e.g., preserve language keywords, function names) rather than deny-lists.
Sample aggressively in production. Full prompt logging for 1% of requests provides enough signal to diagnose issues without creating compliance risk. Store detailed traces only for flagged errors or low-acceptance requests.
Respect user consent. If your coding assistant offers telemetry opt-out, ensure observability pipelines honor that setting. Never send keystrokes or raw code to third-party analytics without explicit agreement.
Debugging common production issues
Symptom: Latency spikes at 9 AM and 2 PM daily.
Diagnosis: Trace timestamps clustered around standup meetings, when many developers resume work. Context retrieval hits cold caches.
Fix: Pre-warm vector indexes for active repositories before peak hours.
Symptom: Token costs doubled after adding a new model.
Diagnosis: Logs show the new model defaults to 8K context vs. 2K for the legacy model, but output quality is identical.
Fix: Cap context window to 4K; acceptance rate holds steady while cost drops 40%.
Symptom: Acceptance rate varies wildly by file type.
Diagnosis: Markdown files have 48% acceptance; YAML configs only 12%. The model was fine-tuned on code, not configuration.
Fix: Route YAML requests to a specialized model or disable suggestions for low-value file types.
Integrating observability into incident response
When an LLM outage or quality regression occurs, observability data accelerates root cause analysis:
- Alerts with context. Fire an alert when error rate exceeds 5% and link to a dashboard pre-filtered to the affected model and region.
- Runbooks reference traces. Document how to export traces for the last hour, correlate with model provider status pages, and compare prompt templates across deployments.
- Post-incident metrics. Track mean time to detection (MTTD) and mean time to resolution (MTTR) for LLM incidents separately from application incidents, since diagnosis steps differ.
Goatfied's Observability suite ties alerts to on-call rotations and incident timelines, surfacing LLM metrics alongside system health in a unified view.
Setting up OpenTelemetry for LLM workloads
OpenTelemetry provides vendor-neutral instrumentation for metrics, logs and traces. To instrument an LLM coding assistant:
1. Install the SDK for your language (Python, TypeScript, Go, etc.) and configure exporters for your backend (Prometheus, Jaeger, or an integrated platform).
2. Create custom spans around prompt construction, model API calls and post-processing. Use semantic conventions for HTTP requests where applicable.
3. Emit custom metrics for token counts, cost and acceptance rate using OpenTelemetry's Meter API.
4. Propagate context across async boundaries and message queues so traces connect user actions to background inference jobs.
5. Test locally with the OpenTelemetry Collector in logging mode to verify spans include all required attributes before shipping to production.
Because Goatfied accepts OpenTelemetry data natively, teams can unify LLM observability with BUILD agent telemetry, GOAT assistant performance and business suite analytics without managing multiple collection pipelines.
Frequently asked questions
What's the minimum observability setup for a small team?
Start with request count, P95 latency, error rate and total token usage per day. Store request IDs and error messages for the last seven days. This baseline costs negligible storage and answers "Is it slow? Is it broken? What's it costing us?"
Should I log entire prompts in production?
Log prompts for errors and a 1–5% sample of successful requests. Full prompt retention for every request creates storage, privacy and compliance burdens. Use structured metadata (file type, language, context token count) to filter without raw text.
How do I measure LLM quality without manual review?
Proxy metrics: acceptance rate (Tab press within three seconds), edit distance (how much the user changed the suggestion), time-to-dismiss, and linting pass rate. For critical workflows, sample 50 requests weekly for human evaluation and correlate scores with automated metrics.
Can I track cost per team or project?
Yes, if you tag requests with team identifiers or repository names. Multiply token counts by your provider's pricing and group by tag. Some platforms charge differently for cached vs. uncached tokens; ensure your instrumentation distinguishes these.
What are typical SLOs for LLM-powered coding tools?
P95 latency under 1.5 seconds for inline completions, under 3 seconds for chat responses. Error rate below 1%. Availability 99.5% or higher. Acceptance rate depends on use case but aim for stability (e.g., within ±3 percentage points week over week).