Direct answer
Evaluate coding agents on representative, frozen repository tasks with predefined acceptance tests, security boundaries, and review criteria. Record both outcomes and process evidence, repeat trials when behavior is variable, keep failed attempts, and avoid collapsing correctness, cost, speed, safety, and maintainability into one unexplained score.
Key takeaways
- Use real task classes with frozen inputs and independently written acceptance criteria.
- Keep correctness, security, process cost, and maintainability as separate dimensions.
- Retain failed trials and human interventions instead of reporting only the best run.
- Version the harness, repository snapshot, client configuration, and source evidence.
Key facts
| Fact | Verified value | Evidence |
|---|---|---|
| Evaluation design | OpenAI recommends task-specific evals, clear criteria, and continuous evaluation rather than impression-based testing. | View source |
| Test construction | Anthropic documents building test cases and grading criteria for model behavior. | View source |
| Risk framing | NIST AI RMF separates risk mapping, measurement, and management activities. | View source |
| Automated runs | Codex documents a non-interactive interface suitable for controlled scripted runs. | View source |
What this guide helps you solve
Help teams build an auditable evaluation before choosing or expanding a coding-agent workflow.
Verification scope and limits
- Evidence basis
- Official documentationSpecification
- Verification scope
- The framework adapts general evaluation guidance to repository changes, command execution, tests, review, and permission boundaries.
- Limits and invalidation conditions
- - No benchmark score or product ranking is asserted.
- - Results do not generalize beyond the tested tasks and configuration.
- - Human review rubrics require calibration.
Build a task suite from real maintenance work
Sample distinct work classes: explanation, localized bug fix, test addition, refactor under constraints, dependency investigation, and security review. Remove credentials and private user data, freeze the starting commit, and ensure a maintainer can complete and verify every task.
Do not make one synthetic issue stand in for all repository work. Report coverage gaps explicitly and add tasks only when their acceptance conditions can be reviewed independently of the agent output.
Write acceptance criteria before running the agent
Combine executable checks with a human rubric. Tests can confirm behavior and types, while maintainers assess scope discipline, readability, architectural fit, and unresolved risk. Include prohibited paths and security expectations so a functionally correct but unsafe patch does not pass.
- Required behavior and regression tests
- Allowed and prohibited file scope
- Authoritative commands and environment
- Security and secret-handling checks
- Human review criteria and rejection reasons
Capture process evidence without collecting secrets
Record client and configuration version, starting commit, task identifier, permissions, sanitized tool actions, exit codes, final diff, test output, retries, interventions, and final disposition. Store raw prompts or source excerpts only when policy permits and they have been redacted.
Label infrastructure failures separately from agent errors. A network outage, broken baseline test, invalid fixture, and incorrect patch require different remediation.
Keep evaluation dimensions separate
Report task completion, test correctness, review findings, boundary violations, human correction, requests, usage, and elapsed time separately. If a summary score is needed, publish its weights and the underlying results so stakeholders can apply a different tradeoff.
Repeat trials and review harness drift
Run enough repetitions to reveal instability, but do not claim statistical confidence without an appropriate design. Preserve all trials, randomization settings when available, and the exact harness version. Re-run after meaningful changes to the client, model, gateway, permission policy, or repository.
Official sources and verification scope
This guide is based on public official documentation. Commands and configuration may change between client versions, so verify the linked sources before use.
Read the verification methodology