Evals
Evals let you ask several agents the same question under the same conditions and compare their answers against criteria you write. You build and run evals in Admin → AI → Evals. A saved eval test holds the page context the agents answer for, the agents under test, the criteria, the prompt, and the eval agent that scores the results.
Each run is kept, so you can re-run a test after changing an agent’s model, skills or instructions and see what moved.
What a test contains
Section titled “What a test contains”| Part | What it does |
|---|---|
| User context | The page and selection the agents answer for, the same context Flow sends when someone chats from that page. Pick a surface, then edit the JSON. Leave it empty to run without page context. |
| Agents under test | The agents that answer the prompt. Each one answers in its own thread with its own model, skills and tools. |
| Eval agent | The judge that scores every answer. Only agents carrying the Eval judge availability appear here. |
| Prompt | One message, written in the normal chat composer, sent to every agent under test. |
| Criteria | One line per criterion, for example Accuracy: names the correct policy document. The judge scores each from 0 to 10 and explains the score. |
Running a test
Section titled “Running a test”Press Run, or press Enter in the prompt box. Fontana saves any unsaved changes, then starts one thread per agent under test. The agents run in parallel, so a run takes about as long as its slowest agent rather than the sum of all of them.
Results fill in live. Each row shows one agent under test with a score per criterion, an overall score, and the measured latency, tokens and cost. Expand a row to read the full answer, the judge’s reasoning per criterion, and the tools the agent called.
If an agent fails to answer, its row records the error and the rest of the run continues. If the judge cannot score an answer, the answer is still kept and the row says the judge failed.
Choosing an eval agent
Section titled “Choosing an eval agent”Fontana ships an Evals Agent configured as a judge: temperature 0, no tools, and a scoring contract that returns structured results rather than prose. To use a different judge, open Admin → AI → Agents, pick the agent, and give it the Eval judge availability. Agents that judge evals are hidden from the normal chat pickers.
Permissions
Section titled “Permissions”Evals use their own permission set, so you can let a team read results without letting them spend model credits on runs.
| Permission | Allows |
|---|---|
evals:read |
Open the Evals page, read tests and past runs |
evals:write |
Create and edit tests |
evals:run |
Start runs |
evals:delete |
Delete tests and their runs |
See Roles and RBAC for how these map to the built-in roles.
What runs are kept
Section titled “What runs are kept”A run stores the prompt, criteria, user context and agent selection as they were at the moment you pressed Run, so history stays readable after you edit the test. Runs also keep each answer, its scores and its measured cost. Deleting a test deletes its runs.
Eval conversations do not appear in your chat history: they are scratch threads owned by the run and removed with it. Test definitions are included in backup and restore; past runs are not.