eval-engineering

The saved excerpt is a snapshot from review. The external source remains the complete and most current version.

---
name: eval-engineering
description: Iteratively inspect an agent repository and optional user-provided traces, interview the user, and create, run, and audit Harbor evals one at a time. Use for agent evals, Harbor tasks, benchmark cases, verifier design, or controlled agent environments.
---

# Eval Engineering

Work with the user to define, build, run, and audit Harbor tasks.

```text
map harness + environment -> propose directions -> user chooses
-> draft specs -> user approves -> build + run + audit -> repeat
```

Use the latest Harbor release. Put task source under `evals/`. Build sequentially while a later task depends on an unproven Harness, Environment, or Verifier. Build independent tasks in parallel when the user requests it.

## Boundaries

- **Task:** `instruction.md` plus an Environment and Verifier.
- **Harness:** the complete agent Harbor runs: model, prompts, loop, repository-defined tools, middleware/hooks, memory/session behavior, and Harbor adapter. Harbor calls this the Agent.
- **Environment:** the container/world around the Harness: OS, files, backing data, services, identity, permissions, network, clock, and mutable state.
- **Verifier:** the test script that independently scores final artifacts or resulting Environment state; it uses trajectory only when final state cannot provide the required evidence.

Repository-defined tool code belongs to the Harness. The data or service behind it belongs to the Environment. Example: a docs agent's `search_docs` definition and result parsing stay in the Harness; the frozen search index and its error behavior live in the Environment. If production supplies a tool server dynamically, keep that server in the Environment and preserve how the Harness discovers and calls it.

## Score the requested outcome

- For stateful work, score independently observed final Environment state first. Example: a booking exists for the requested room and no conflicting booking exists.
- Keep ATIF as diagnostic evidence by default. Use trajectory or session evidence only when final state cannot establish the requirement, such as proving later user turns used the same session.
- Do not require a tool name, subagent, retry count, exact number of updates, or exact wording unless that is the user-facing requirement.
- Before building, state what the agent can see, the required user-visible outcome, prohibited effects, and materially equivalent outcomes that must pass. Do not score a hidden evaluator preference.

## References

Read each reference when its decision appears:

- [Trace sourcing](references/trace-sourcing.md): select and analyze traces only when the user supplies a source.
- [Harness](references/harness.md): identify the actual agent Harbor will run and preserve its behavior.
- [Task design](references/task-design.md): turn one selected capability into a judgeable request.
- [Environment building](references/environment-building.md): choose live, frozen, or simulated backing data and services.
- [Multi-turn simulation](references/multi-turn-simulation/guide.md): run scripted or LLM-generated user turns through one Harness session.
- [Verifier design](references/verifier-design.md): define independent evidence, scoring, and calibration.
- [Harbor](references/harbor.md): create, run, and inspect the Harbor task.

## 1. Map the Harness and production Environment

Start at the public agent entrypoint and follow reachable code.

```text
Harness: entrypoint; prompts; models; loop; routing; retries; hooks; memory;
         repository-defined tools, inputs, outputs, and effects
Environment: files; records; indexes; services behind tools; identity;
             permissions; network; time; mutable state
Purpose: intended users, jobs, and useful outcomes
Evidence: tests, fixtures, issues, existing evals, and documented failures
```

Do not start services, install packages, or use credentials during mapping. Explain the map in the conversation and ask only what code cannot answer, such a