Evaluating AI Coding Agents in CI

Every platform team is being asked to wire AI coding agents into CI: auto-fix lint errors, generate migration scripts, review PRs. The demo looks great. The failure mode is a confident patch that passes syntax checks, breaks idempotency, and merges at 5 PM Friday.

Evaluating agents in CI is not about whether the model can write code-it is about whether you can measure regression risk, bound autonomy, and kill bad outputs before they reach main.

What to Evaluate: Tasks, Not Models

Benchmark the agent on your repo's actual task classes, not HumanEval scores. Group tasks by risk and frequency.

Score each task class separately. An agent excellent at docs may be dangerous at Terraform.

Evaluation Harness Design

Build a fixed benchmark suite from anonymized real PRs and known-good outcomes. Run the agent against the pre-change codebase and diff against the merged human PR.

Benchmark run
─────────────
Fixture repo @ commit SHA
    │
    ▼
Agent given task prompt (from real issue)
    │
    ▼
Output: patch / branch
    │
    ├── Static checks (lint, typecheck, SAST)
    ├── Test suite (unit + integration)
    ├── Custom evaluators (policy, diff size, file allowlist)
    └── LLM-as-judge (optional, low weight)
    │
    ▼
Scorecard → dashboard / merge gate

# GitHub Actions: agent eval on schedule
name: Agent Benchmark
on:
  schedule:
    - cron: "0 6 * * 1"  # weekly
  workflow_dispatch:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run agent on fixture tasks
        run: node scripts/agent-eval/run-suite.mjs --config eval/tasks.yaml
      - name: Upload scorecard
        uses: actions/upload-artifact@v4
        with:
          name: agent-scorecard
          path: eval/results/

Metrics That Matter

• Task success rate: patch applies cleanly and all CI checks pass.
• Regression rate: benchmark tasks that passed last week and fail this week.
• Diff footprint: lines changed vs human baseline-large diffs correlate with review burden.
• File scope violations: agent edited files outside task allowlist.
• Flake rate: non-deterministic outputs on identical prompts.
• Time-to-green: wall clock from prompt to passing CI.

Track metrics per model version and per prompt template. Agent upgrades without eval regression gates are reckless.

CI Integration Patterns

Three patterns we use depending on risk tier:

Tier 1 - Advisory comment on PR (agent suggests, human merges)
Tier 2 - Draft PR opened by bot (human marks ready)
Tier 3 - Auto-merge branch with platform CODEOWNERS on critical paths

Never: agent pushes directly to main without branch protection.

# Branch protection + required checks
# required_status_checks:
#   - agent-eval-pass
#   - unit-tests
#   - sast
#
# Agent output PR must pass same checks as human PRs - no shortcuts.

Use path filters: agent workflows skip trigger on terraform/, .github/, and secrets-adjacent paths unless platform team label is present.

Preventing CI Agent Incidents

• Cap agent iteration loops-max 3 fix attempts then fail open to human.
• Sandbox agent execution: no prod credentials, no network except package registries.
• Require signed commits from bot account; block force push.
• Store prompts and outputs as CI artifacts for 90 days for postmortems.
• Compare agent PR test coverage delta-reject if coverage drops.

An agent that only runs when CI is already green adds latency. An agent that runs on red CI to 'fix' failures creates untestable state. Define entry conditions explicitly.

Engineering Takeaways

• Evaluate agents on your repo's task classes, not generic coding benchmarks.
• Build a fixture-based harness with known-good human PRs as ground truth.
• Track success rate, regression rate, diff size, and file scope violations per model version.
• Tier autonomy: advisory comments → draft PRs → auto-merge only for low-risk paths.
• Agents must pass the same CI gates as humans-no shortcut status checks.
• Cap retry loops and sandbox credentials; agent incidents look like supply chain attacks.
• Run eval suites weekly on schedule, not only when upgrading models.

Trade-offs I Would Consider

Option: Advisory-only agent in CI (comments, suggestions)

• Pros: Low merge risk, builds trust, easy rollback by ignoring bot.
• Cons: Limited toil reduction, developers may ignore noisy suggestions.

Option: Draft PR agent with auto-merge on low-risk paths

• Pros: Meaningful automation for lint/docs, measurable time savings.
• Cons: Requires mature eval harness, path allowlists, and regression alerting.

What I Would Do in Production

Curate 20-50 benchmark tasks from historical PRs spanning services, tests, and docs.

Run weekly eval workflow; alert on >5% regression in task success rate.

Block agent auto-merge on terraform/, migrations/, and auth middleware paths.

Publish scorecard dashboard to platform team; share summary with engineering monthly.

Version pin model and prompt template in eval config; bump only after scorecard parity.

Include agent-generated PRs in normal code review sampling for quality audits.

FAQ

Q: Should we use LLM-as-judge for eval?

As a secondary signal only. Deterministic checks-tests pass, lint clean, diff scope-should drive merge gates. LLM judges inherit model bias.

Q: How big should the benchmark suite be?

Start with 20 representative tasks covering each autonomy tier. Grow when you add new task classes or services. Refresh quarterly as codebase evolves.

Q: Can the agent fix failing CI on its own PR?

Allow max N self-fix iterations on its own branch only. Never let it modify CI config or branch protection to make itself pass.

Q: What about cost of running eval weekly?

Cheaper than one bad auto-merged migration. Cache fixtures, run parallel jobs, subset smoke eval on PR and full suite weekly.

Q: How do we compare Cursor, Copilot, and custom agents?

Same harness, same tasks, same scoring. Vendor marketing is irrelevant-your repo's pass rate is the only comparison that matters.