review
Latency budgets for ghost text completions
Learn how to set latency budgets for ghost text completions in code editors, from the 75ms threshold to pipeline stage breakdowns and monitoring strategies.
Ghost text completions—the gray suggestion that appears mid-keystroke in your editor, terminal or IDE—must arrive within 75 milliseconds to feel instant, or they hurt productivity more than they help. Most teams never measure this latency, then wonder why developers turn autocomplete off. A latency budget formalizes the acceptable delay for each stage of your completion pipeline, from keystroke to pixels on screen, so you can identify and fix the slowest link.
Below we walk through the typical budget breakdown, common bottlenecks (network, model inference, rendering), monitoring strategies, and when to trade off accuracy for speed. You'll finish with a checklist you can run against your own tooling—whether you're building a custom language server or evaluating third-party coding assistants.
Anatomy of a completion latency budget
A typical inline completion flows through four stages, each eating part of your 75 ms budget:
| Stage | Typical target | What happens |
|-------------------------|----------------|------------------------------------------------------------|
| Event capture | 5 ms | Keystroke → buffering → debounce → trigger request |
| Network round-trip | 20–30 ms | Request serialization, TLS handshake, wire time, response |
| Model inference | 30–40 ms | Tokenization, forward pass, sampling, post-processing |
| Client rendering | 5–10 ms | Parse response, diff editor state, paint ghost text |
Total: 60–85 ms under ideal conditions. If any stage doubles, the completion arrives after the user has already typed the next character, creating the jarring "catch-up" effect that makes developers disable the feature.
Event capture and debouncing
Editors debounce keystrokes to avoid flooding the server on every letter. A 50 ms debounce feels responsive; 150 ms feels laggy. Some tools also wait for a semantic boundary—end of a token or line—which can add unpredictable delay. Budget 5–10 ms for capture plus debounce, and measure actual debounce intervals in your telemetry.
Network round-trip
Local language servers skip the network entirely (0 ms), but cloud-based LLMs pay the physics tax. A 10 ms ping to the same region, plus TLS and HTTP/2 framing, typically costs 20–30 ms. Cross-region or satellite links can balloon to 100+ ms, instantly blowing the budget. Always run inference servers in the same cloud region as most of your developers, and use connection pooling to amortize handshake overhead.
Model inference
Smaller models (1–7B parameters) on GPU can generate the first token in 10–20 ms; frontier models (70B+) take 40–100 ms unless heavily quantized or batched. Because ghost completions show only one or two lines, you often care about time to first token rather than throughput. Budget 30–40 ms for inference, and choose models optimized for low latency—quantized variants, speculative decoding, or distilled checkpoints—over raw accuracy.
Client rendering
Parsing the server response and painting ghost text usually takes under 10 ms, but complex diffs or slow extension hosts in Electron-based editors can spike to 50 ms. Measure requestAnimationFrame delays and ensure the UI thread isn't blocked by syntax highlighting or other extensions.
Measuring and enforcing budgets
Start by instrumenting each stage with high-resolution timestamps (e.g., performance.now() in TypeScript, std::chrono::steady_clock in C++). Log the deltas to a time-series database and graph the p50, p95 and p99 latencies—p99 captures the tail experience that frustrates users.
Enforcement checklist:
1. Set SLOs per stage: e.g., "p95 inference < 35 ms, p95 network < 25 ms."
2. Alert on violations: fire if p95 crosses the threshold for five consecutive minutes.
3. Auto-degrade: if the server is slow, skip completions rather than queueing them—stale suggestions are worse than none.
4. A/B test budgets: try tighter limits (60 ms total) on half your users to see if completion acceptance rate improves.
On Goatfied's BUILD platform, cloud coding agents run their own test suites and open pull requests, and the AI code editor measures completion latency across every keystroke. Because Goatfied integrates the editor, agent runtime and infrastructure on one platform, it can trace a completion from keypress through GPU inference to screen paint without stitching together logs from separate vendors.
Common bottlenecks and fixes
Long context windows
Sending 10,000 tokens of context on every keystroke can double inference time. Instead, truncate to the current file plus a few imports, or use a separate "context summarization" model that runs asynchronously and feeds a compressed representation to the completion model.
Cold starts
Serverless GPU functions often take 2–10 seconds to warm up, destroying the first completion. Keep a small pool of warm instances (even just one per developer during active hours), or fall back to a faster CPU-based model for the first request while the GPU spins up.
Head-of-line blocking
If requests queue behind a slow batch job, latency spikes unpredictably. Use separate request queues for interactive completions versus background indexing, and prioritize completions with a higher quality-of-service class.
Client-side overhead
Extensions that parse the entire project on every change can starve the completion renderer. Profile your editor's extension host and disable or throttle heavy plugins during typing.
When to sacrifice accuracy for speed
A completion that arrives in 50 ms but suggests the wrong variable is often better than a perfect suggestion at 150 ms—because the user has already moved on. Practical trade-offs include:
- Smaller model tiers: a 3B-parameter model may only be 80 % as accurate as a 70B model, but finishes in one-third the time.
- Prefix-only context: omit the suffix (lines after the cursor) to halve token count; accuracy drops ~5 % but latency improves 20–30 %.
- Caching: memoize completions for identical prefixes; hit rates of 15–25 % are typical in repetitive codebases.
- Speculative rendering: show the cached or heuristic completion immediately, then silently replace it when the LLM response arrives, as long as the user hasn't typed yet.
Measure acceptance rate (how often users tab-accept) alongside latency. If tightening the budget from 80 ms to 60 ms raises acceptance by 10 %, the accuracy trade-off pays for itself.
Budgeting for mobile and remote scenarios
Developers on trains, coffee-shop Wi-Fi or VPNs see 100–500 ms RTTs. Options:
- On-device models: small GGML or CoreML models run locally in 20–40 ms, no network required. Goatfied's GOAT assistant offers an on-device mode for offline or low-latency queries, and the same principle applies to code completions.
- Adaptive timeout: if RTT > 50 ms, skip cloud completions and fall back to local heuristics (recent tokens, AST-based snippets).
- Prefetch: after the user pauses, speculatively request completions for the next few likely cursor positions and cache them.
Latency budgets in broader agent workflows
Ghost completions are one interaction pattern; AI agents that write entire functions or open pull requests operate on minute-scale budgets, not millisecond. Still, the same principles apply: measure every stage, set SLOs, auto-degrade when slow, and choose the fastest model that meets your accuracy bar. Goatfied's cloud coding agents, for instance, run tests after each change and escalate to a human reviewer if CI fails—speed matters less than correctness, but the platform still tracks end-to-end task latency to prevent agents from blocking a deploy.
Frequently asked questions
What happens if I exceed the latency budget?
Users perceive completions as "laggy," acceptance rates drop, and many developers disable the feature entirely. Some editors also queue completions, so a slow response arrives after the user has typed more, causing visual jitter when the ghost text updates mid-word.
Should I measure latency from the server's perspective or the client's?
Both. Server-side metrics (inference time, queue depth) help you optimize the model and infrastructure. Client-side metrics (keystroke to paint) reflect the actual user experience, including network, rendering and extension overhead. Always export p95 and p99 percentiles for both.
How do I allocate budget when network latency varies by region?
Set a global baseline (e.g., 30 ms network budget) for your primary region, then add region-specific deltas in your SLO dashboard. For remote or high-latency users, consider a tiered system: local fallback under 75 ms, cloud completion if RTT < 50 ms, else skip.
Can I use a latency budget for non-code completions, like chat or document Q&A?
Yes, but the thresholds differ. Chat messages tolerate 200–500 ms before feeling slow; document search should return first results within 100–150 ms. Ghost text is unique because it interrupts typing, so the budget is tighter.
What's a realistic p99 target for a production completion service?
Aim for p99 < 100 ms end-to-end if most users are on good network connections, or p99 < 150 ms if you serve global/mobile traffic. Anything above 200 ms at p99 means a significant fraction of completions feel broken.