models
Building evaluation sets from your own codebase
Learn how to build custom LLM evaluation sets from production logs, code repositories, and support tickets to test model performance on your specific domain.
Most teams building with LLMs discover a painful truth: the public benchmarks that look great in research papers tell you almost nothing about whether a model will handle your domain's edge cases, your company's terminology, or the subtle context that makes an answer right versus dangerously wrong. A custom evaluation dataset built from your actual codebase gives you reproducible signals on model performance where it matters—inside your product.
You need a test set that captures the real distribution of inputs your system will see: the error messages users actually report, the code patterns your engineers write, the queries support tickets contain. This guide walks through extracting evaluation examples from production logs, code repositories, support tickets and user sessions, then structuring them so you can iterate on prompts and models with confidence.
Why your codebase is the best source of evaluation data
Public datasets like HumanEval or MMLU test general capabilities, but your application has domain-specific requirements. If you're building a code assistant for infrastructure-as-code, examples of Python REST APIs won't stress the model on Terraform plan parsing. If you're automating customer support, GLUE sentiment benchmarks miss the nuance of a churning enterprise account versus a confused trial user.
Mining your codebase and adjacent artifacts—pull request comments, issue threads, Slack discussions about design decisions—yields examples with three advantages:
- Realistic input distribution: the queries, code snippets and edge cases your users actually encounter, not synthetic academic tasks.
- Ground truth from humans: approved pull requests, resolved tickets and accepted answers reflect judgments your team already made.
- Coverage of failure modes: the bugs filed, rollbacks executed and escalations triggered show you exactly where automated systems break.
When you build a test set from 200 real support tickets, each model run tells you how often the assistant would have given the right answer, linked the correct documentation or escalated appropriately—metrics that translate directly to reduced workload and faster resolution.
Identifying high-value evaluation examples in your repository
Start by deciding what tasks you want to measure. Common categories:
| Task type | Example sources | Ground truth |
|-----------|----------------|--------------|
| Code generation | GitHub issues with linked PRs; feature requests → implementation | The final merged code |
| Bug fixing | Issue reports → fix commits | The diff that resolved the issue |
| Code review | Pull request comments → author revisions | Whether the comment led to a change |
| Documentation lookup | Support tickets → agent responses rated helpful | Human rating or issue resolution |
| Refactoring | Large PRs with "refactor" in title | Code before and after, passing tests |
Filter for quality signals: PRs merged without revert, issues closed as resolved (not duplicate or won't-fix), tickets marked solved by the customer. You want examples where the outcome was clearly correct, not ambiguous.
For code tasks, look for small, self-contained changes. A 3,000-line framework migration is hard to evaluate automatically; a 40-line bug fix with a clear issue description and test coverage makes a clean test case. Use heuristics like changed line count, number of files touched and time-to-merge to surface candidates.
Structuring your dataset for repeatable evaluation
Each example needs three components: input, expected output and success criteria. The format depends on your task:
Code generation example:
{
"id": "issue-4721",
"input": {
"issue_description": "Add rate limiting to /api/usage endpoint",
"existing_code": "...",
"context_files": ["middleware/auth.ts", "types/quota.ts"]
},
"expected_output": {
"diff": "...",
"tests_pass": true
},
"evaluation": "exact_match_ignoring_whitespace OR syntax_valid AND tests_pass"
}
Support ticket example:
{
"id": "ticket-9182",
"input": {
"customer_message": "Getting 'Invalid API key' on staging but prod works fine",
"account_tier": "enterprise",
"recent_changes": ["Rotated staging keys 2024-01-10"]
},
"expected_output": {
"diagnosis": "Staging environment not updated with new key",
"action": "Update STAGING_API_KEY in deployment config",
"escalate": false
},
"evaluation": "correct_diagnosis AND action_resolves_issue"
}
Store metadata: timestamp, author, related files, customer segment, complexity tag. This lets you slice results—maybe the model excels at simple queries but fails on enterprise edge cases, or performs well on recent code but poorly on legacy modules.
Aim for 100–500 examples to start. Fewer than 50 and you'll see noisy metrics; more than 1,000 takes significant curation effort. Prioritize diversity over volume: cover different code paths, error types, seniority levels of the human who created the ground truth.
Automating dataset extraction and refresh
Manual curation doesn't scale. Build tooling to mine examples continuously:
1. Query your version control: GitHub, Bitbucket and Azure DevOps APIs let you fetch closed issues with linked PRs, filter by labels like "bug" or "enhancement," and pull diffs plus commit messages.
2. Parse support ticket platforms: Zendesk, Intercom and Salesforce APIs surface tickets marked resolved, along with conversation history and resolution notes.
3. Extract from logs: If you log LLM interactions in production, sample requests where users accepted the output (clicked a suggestion, merged generated code) versus rejected it (edited heavily, discarded).
4. Deduplicate and clean: Remove PII, strip customer-specific secrets, normalize formatting so models aren't penalized for whitespace differences.
Run extraction weekly or monthly. As your codebase evolves, older examples become less representative; a test set from six months ago may not cover new libraries, API changes or coding conventions your team adopted. Keep a rolling window—retain high-value evergreen examples (classic bug patterns, core algorithm implementations) and refresh the rest.
If you're using Goatfied's AI code editor, cloud coding agents already open pull requests and run tests. You can tag merged PRs that agents authored as evaluation gold, comparing agent-generated solutions against the ground truth human engineers produced for the same issue.
Evaluating model outputs against your custom dataset
With structured examples in hand, decide on metrics. For code tasks:
- Exact match: generated code matches the reference diff character-for-character (strict, rarely hits 100%).
- Syntax validity: does the output parse without errors?
- Test pass rate: does the generated code make existing tests pass, or new tests provided in the example?
- Semantic similarity: embedding-based distance or tree-edit distance between abstract syntax trees.
For text tasks (support responses, documentation):
- Keyword presence: does the response mention the correct component, error code or action?
- Human rating: sample a subset and have engineers score outputs 1–5.
- Outcome proxy: if you can map suggestions to later ticket resolution, measure whether the model's answer would have resolved the issue.
Run the full suite before and after prompt changes, model swaps or fine-tuning. Track metrics over time in a dashboard. If you're iterating on a RAG pipeline for code search, your custom dataset shows whether the new chunking strategy actually surfaces more relevant snippets for the queries engineers ask.
Goatfied's Observability suite ingests OpenTelemetry traces, so you can instrument evaluation runs and compare latency, token cost and accuracy across model versions in one view.
Iterating on prompts and models with real-world signals
The dataset becomes your feedback loop. When a new model version claims 10% better performance on HumanEval, re-run your custom set. Maybe it improves general code completion but regresses on your team's preferred error-handling patterns—information a public benchmark won't give you.
Segment results by metadata. If the model struggles on examples tagged "database migration" or "async concurrency," you can add few-shot demonstrations from those slices or fine-tune on a focused subset. If one customer tier (enterprise with custom integrations) sees lower scores, prioritize examples from that segment.
Version your dataset alongside your code. Store it in Git, tag releases and document changes. When you add a new feature to your product, append corresponding evaluation cases so regressions surface immediately. Treat test-set drift like you treat code drift: something to monitor and manage.
Frequently asked questions
How many examples do I need for reliable metrics?
Start with 100–200 examples covering diverse scenarios. Below 50, metrics swing wildly with small prompt tweaks; above 500 offers diminishing returns unless you're fine-tuning. Prioritize breadth (different code paths, user types, error classes) over depth (many variations of the same task).
Should I include examples where the model currently fails?
Yes—especially systematic failures. If your model hallucinates package names in infrastructure code or mishandles timezone logic, add those as regression tests. Balance the set so it's not all failures (which deflates scores and obscures progress) but includes enough hard cases to prevent overfitting to easy wins.
How do I handle ground truth that's ambiguous or imperfect?
Mark examples with confidence levels. A bug fix that was later reverted is lower-quality ground truth than one shipped and stable for months. For subjective tasks (code review comments, support tone), consider multi-annotator agreement or accept a range of valid outputs rather than a single reference answer.
Can I use this dataset to fine-tune a model?
Only after a train/test split. Hold out 20–30% for evaluation; fine-tune on the rest. Re-evaluate on the held-out set to measure true generalization. Reusing test examples for training inflates metrics and hides overfitting. Refresh the held-out set periodically to avoid leakage as you iterate.
How often should I refresh the dataset?
Monthly or quarterly, depending on code churn. If you ship daily and APIs change often, refresh more frequently. Archive old examples that no longer reflect current code structure, but keep a core set of timeless cases (classic algorithms, perennial bug patterns) for longitudinal comparison.