Model routing for coding tasks: when to use which LLM

A production coding assistant that sends every task to GPT-4 burns budget on simple renames. One that routes everything to a fast 7B model struggles with cross-file refactors and misses edge cases. Effective model routing treats model selection as a dispatch problem: match each operation's complexity, latency tolerance, and error cost to a model tier that delivers acceptable quality at the lowest resource spend.

The routing layer sits between your task planner and the model itself. It receives a structured operation—"rename this variable across three files," "implement this interface," "add error handling to this function"—and decides which model handles it. A naive implementation picks one model for everything. A better approach classifies tasks by their demand on context length, reasoning depth, and tolerance for hallucination, then routes accordingly.

Why single-model systems break down

Running all tasks through a frontier model feels safe: you get top-tier reasoning for hard problems, and the model won't struggle with easy ones. The cost shows up in two places. First, you're paying frontier pricing for operations a smaller model handles perfectly—formatting, simple refactors, adding imports. Second, you introduce latency where it doesn't belong. A quick syntax fix that should take 200ms now waits in a queue behind multi-turn planning requests.

The opposite extreme—routing everything to a fast, cheap model—fails differently. Smaller models hallucinate more often on tasks requiring multi-hop reasoning. They miss context spread across files, invent APIs that don't exist, and produce syntactically correct code that breaks contracts. For operations like "refactor this module to support async," you end up in a retry loop that costs more tokens and time than using a stronger model once.

The routing problem is real because task distributions are non-uniform. In a typical coding session, 60–70% of requests are mechanical: format this block, add this type annotation, rename this symbol. Another 20–30% are medium-complexity: implement a small feature, add test cases, fix a specific bug. The final 10% are hard: architectural changes, multi-file rewrites, debugging subtle race conditions. A single model can't efficiently serve this spread.

Classifying tasks for routing decisions

Effective routing starts with a task taxonomy. You need features that separate operations cleanly enough that you can draw boundaries: "if X, use model tier 1; if Y, use tier 2."

Scope and context size is the most reliable signal. Single-file edits under 100 lines fit comfortably in a small model's context and rarely require deep reasoning. Cross-file changes, especially those touching more than three files or spanning 500+ lines, benefit from larger context windows and stronger long-range coherence. If the operation references symbols across modules, that's a vote for a bigger model.

Edit complexity captures how much the model needs to infer. Mechanical tasks—formatting, adding imports, inserting logging—follow patterns the model has seen thousands of times. Medium tasks—implementing a function given a clear signature, adding error handling to a happy path—require local reasoning but no invention. Hard tasks—designing an abstraction, resolving a subtle bug, optimizing an algorithm—demand multi-step inference. Map these to model tiers: mechanical → small fast models, medium → mid-tier, hard → frontier.

Error tolerance is the inverse of cost-of-failure. Formatting errors are cheap; the linter catches them immediately. A buggy refactor that breaks tests costs a retry cycle. A hallucinated API in production code costs engineering time and user trust. High-stakes operations justify stronger models even if they're technically simple.

Latency budget matters for interactive workflows. If the user is watching a cursor move in real time, you have 500ms to feel instant. Background batch operations can tolerate multi-second waits. Route interactive single-file edits to fast models; route overnight batch refactors to the strongest model that fits budget.

Goatfied's agent loop—plan, constrain, edit, validate, retry—makes routing decisions at the "edit" step. The planner outputs structured operations with enough metadata to classify them: files touched, estimated diff size, whether tests are required. The router uses those features to pick a model, then the validator (compile, lint, test gates) catches errors regardless of which model ran. This separation means you can tune routing rules without changing the core loop.

Practical routing heuristics

Here's a starting ruleset that works for teams shipping code with multiple model tiers available:

Tier 1 (small, fast models like Qwen 2.5 Coder 7B or DeepSeek Coder 6.7B): single-file edits under 200 lines, no cross-file references, mechanical operations (format, add comments, insert logging, simple renames). Target: <500ms latency, sub-cent cost per operation.

Tier 2 (mid-size models like GPT-3.5 or Claude Haiku): single-file edits over 200 lines, 2–3 file changes, implementing functions with clear specs, adding tests for existing code, fixing linter errors. Target: 1–2s latency, low single-digit cents.

Tier 3 (frontier models like GPT-4, Claude Sonnet, or Gemini 1.5 Pro): 4+ file changes, architectural refactors, debugging without clear repro steps, tasks requiring API design or cross-module coherence. Target: 3–5s latency, acceptable for non-interactive flows.

Override rules matter. If a Tier 1 task fails validation (doesn't compile, breaks tests), retry with Tier 2. If Tier 2 fails twice, escalate to Tier 3. This catches cases where your classifier misjudged complexity. The cost of one retry is lower than always using the expensive model.

Latency-sensitive paths—cursor-following edits, inline completions—should hard-cap at Tier 1 or Tier 2 regardless of complexity. If the operation is too hard for fast models, break it into smaller steps or defer it to a background task. Users tolerate a "this will take a moment" message for big refactors; they don't tolerate a frozen editor.

Implementation patterns

The simplest router is a decision tree: check file count, check line count, check edit type, return a model ID. This works fine for 80% of cases and takes an afternoon to build. Encode it as a pure function that takes task metadata and returns { model: string, maxTokens: number, temperature: number }.

A smarter router uses embeddings or a small classifier. Encode the task description and diff context as vectors, then train a lightweight model to predict which tier historically succeeded fastest. This adapts to your codebase's specific patterns—maybe your team writes especially gnarly SQL, so database-related tasks route higher than usual. The training set is your logged operations with their outcomes (success, retry count, latency). This adds complexity but can cut costs 20–30% versus heuristics alone.

Hybrid approaches work well in practice: use heuristics for obvious cases (format always goes to Tier 1, 10-file refactors always go to Tier 3), then use a classifier for the ambiguous middle. This keeps latency low—no model inference for simple routing decisions—while handling edge cases gracefully.

Track routing decisions and outcomes. Log { task_id, model_used, success, retry_count, latency_ms, cost } for every operation. When you see Tier 1 retry rates spike, your heuristics are too aggressive. When Tier 3 usage climbs above 15%, you're over-routing. Use this data to tune thresholds and retrain classifiers.

Where routing fails and what to do instead

Routing can't fix tasks that are genuinely too hard for any available model. If your frontier model fails three times on the same architectural refactor, the problem isn't model selection—it's task decomposition. Break the operation into smaller, verifiable steps. Instead of "migrate this module to async," route subtasks: "identify blocking calls," "wrap each in async equivalent," "update call sites," "add integration test." Each subtask is simpler and more likely to succeed.

Routing also can't compensate for bad constraints. If your planner produces vague instructions ("improve this code"), even a frontier model will struggle. Goatfied's constraint phase helps here: before routing, the system ensures the task has clear success criteria (specific files, expected behavior, test requirements). A well-constrained medium task often succeeds with a cheaper model better than a poorly-constrained hard task with an expensive one.

Finally, routing decisions depend on model availability. If you're self-hosting and only have two model tiers, you can't implement fine-grained routing. In that case, focus on task decomposition and validation. The smaller your diffs, the less model strength matters—even a weak model rarely hallucinates a three-line change.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose