Evaluator–optimizer

All steps

Normal run

1. Request arrives. The facts are already settled. Order 4417 qualifies for a refund; what remains is writing a reply that meets the support team's standard.
2. First draft. The writer produces a correct reply. It approves the refund and names the order, but it reads like a form letter.
3. Submit the draft. The draft goes to a second model with a different prompt. Judging is easier than writing, so the judge catches what the writer missed.
4. Evaluate. The evaluator scores the draft against a written rubric: correct decision, states when the money arrives, apologises. Two of three pass.
5. Critique. The feedback is specific and actionable. A bare “try again” would give the writer nothing to work with.
6. Revise. The writer keeps what passed and fixes what failed. Its context now holds the first draft and the critique.
7. Resubmit. The second draft goes back for the same evaluation.
8. Pass. All three criteria are met on the second round. The loop has a reachable exit, and it took it.
9. Respond. Refund approved, in a reply the team would sign. Two rounds cost about twice the tokens and time of one.

Outcome: correct answer, better written. The loop paid off because the criteria were explicit and the second draft was measurably better than the first. Without both, the extra round is wasted cost.

Failure mode

1. Request arrives. Identical start to the normal run.
2. First draft. The same acceptable first draft.
3. Submit the draft. Nothing has gone wrong yet.
4. An impossible rubric. Two teams each added a rule to the rubric. “Under 50 words” and “quote the refund policy in full” cannot both be satisfied by any draft.
5. First critique. The draft quotes the policy, so it is too long.
6. Revise. The writer obeys. It cuts the policy quote and comes in under 50 words.
7. Opposite critique. The shorter draft now fails the other rule. Each critique is reasonable on its own; together they point in opposite directions.
8. The loop oscillates. Drafts three, four and five swing between the two demands. Every round re-sends the full history, so each one costs more than the last.
9. Budget exhausted. The run hits its token budget on the fifth iteration. The customer has no reply, and the first draft would have been acceptable.

Failure: the loop never converges. An evaluator loop needs an exit it can always reach. Mitigation: cap the iterations, return the best draft so far at the cap, and test the rubric for criteria that contradict each other.