Playground and evals

Playground#

The agent's Playground tab sends a message through the same governed path real traffic uses: guardrails, budgets and the audit log all apply.

Simulated users (/api/agents/{key}/simulations) let a model play a user with a hidden goal for multi-turn tests.

Attach files#

Attach (or drop or paste) up to 5 files on a message. Each is uploaded as soon as you pick it, with POST /api/agents/{key}/playground/files, and shows its preview or the reason it was refused. Send waits until every file has uploaded.

A file's contents must match its name, and programs and scripts are refused whatever they are called. Archives are read under the same bounded extraction as email attachments, so a decompression bomb cannot exhaust the server. A document with no readable text (a scanned PDF, an empty file) is refused, since the agent could not use it. Files are kept for 7 days, or until their session is deleted, and only you can open yours. A reopened session shows each message's files and says which have expired.

Compare variants against a checklist#

Compare asks the prompt in the message box under 2 to 4 variants, each a system half and a model, and shows the answers side by side. Each column is its own governed run with its own transcript.

• Attach document uploads a file the same way the chat's Attach does. Every column is given the same document as text, so you can see whether a model keeps its instructions with a long document in its context. Files attached in the message box go with the compare too.
• Rules each reply must keep is a checklist of up to 10 rules, scored on each column's reply. Choose Load demo rules for a ready-made set: JSON only, at most 50 words, never "sorry", ends with "— Liya", and an escalate field that is true.
• Each column shows its speed, for example first token 1.2 s · 240 tok/s · total 3.4 s, and a pill such as 5/6 rules kept above a ✓/✗ line for each rule.

A JSON field such as answer or $.answer.text makes the rule read that field of a JSON reply rather than the whole reply. A reply that isn't JSON fails every rule with a field. Rules are checked before anything runs, so a bad rule is refused with 422 and no column is run. A column that was refused or failed has no reply to check, and says so.

The API takes rules ({kind, value, path, label}) and files (upload ids) on POST /api/agents/{key}/playground/compare. Each column then carries rules ({scored, kept, total, results}) and speed. The audit row of each column records how many rules it kept and which failed.

To time the first token, the playground streams its calls to OpenAI-compatible and Anthropic providers and puts the answer back together before guardrails, metering and the transcript see it. Single runs (POST /api/agents/{key}/playground) report speed too. Calls made anywhere else are not streamed.

What else the Playground shows#

• Tables and charts in replies, when the agent's instructions say how to write them. See Tables and charts in replies.
• Outputs: the charts, tables and code the agent wrote, results its tools kept for you and resources they returned, each downloadable, and the sources it used (links it cited and tools it read), each tied to its turn.
• Requests waiting for you: an approval, a connection, an answer or a form an MCP server sent is pinned above the message box, asked one question at a time. A send that fails keeps what you typed. Once settled, a request folds into one line in its turn as answered or declined.
• Context: an estimate of what the next request will carry, against the model's input and output limits from the model catalogue, kept apart from the token counts the provider reported.

Feedback on replies#

A person may rate any reply: POST /api/agents/{key}/playground/feedback with {"conversation", "helpful", "comment"} records the vote on the conversation (feedback on its record), on the audit trail as agent.reply.feedback, and in the agent's counters (agent.feedback.helpful, agent.feedback.unhelpful). The mobile app offers Helpful and Not helpful under each reply, with room to say what was wrong. This is the human oversight a deployer must be able to show.

Long chats#

A chat can run as long as you like. Each follow-up is sent with the newest 30 exchanges as they were said. Once a chat grows past that, its oldest turns are folded into a summary of the conversation, written by the agent's own model under the same guardrails, budget and audit as any of its calls, and sent ahead of the newest turns. The summary is updated about every 10 exchanges, not on every message. If the model call is refused, a plain extract of the turns is kept instead, so nothing is dropped without a record.

Each request is also fitted to the context window of the model it goes to, measured in tokens. If the newest turns alone would not fit, the oldest of them are folded into the summary for that request. If a provider still refuses a prompt as too long, the request is sent once more with half the history, rather than the message failing.

When a reply was written from a summary, the chat says so under it, in the console and in the mobile app. The summary is stored with the session, sealed like its messages, and deleted with it. Under the off recording mode nothing is kept, so nothing is summarised either.

Datasets#

Datasets are versioned and append-only. Each has a kind:

Create one from Agents › Evaluations or with the API:

1. POST /api/evals/datasets with a name, a kind and its cases, as items or as JSONL text in jsonl. Posting again writes the next version, but append: true adds to the latest instead of replacing it.
2. Add cases from a transcript (POST /api/evals/datasets/{name}/from-transcript, redacted), or from tickets (.../from-tickets).
3. Optionally split it into calibrate and holdout sets with a fixed seed (.../split).

Runs#

1. Start a run with POST /api/evals/runs, naming the dataset and agent. The run pins the dataset version and the agent's config version.
2. Open the run on Agents › Evaluations. Scores are compared with the baseline run.
3. When you are happy, mark it as the new baseline with POST /api/evals/runs/{run_id}/baseline.

Evaluators (/api/evals/evaluators) include built-in checks and LLM-judge evaluators. POST /api/agents/{key}/evals/generate drafts cases from an agent's description to get you started.

Red teaming#

Agents › Red teaming throws known attacks at an agent: instruction override, role-play jailbreaks, system-prompt extraction, PII exfiltration and tool abuse, each obfuscated (base64, homoglyphs, zero-width characters and more) and sent directly or planted in content the agent reads. A run reports the attack success rate (ASR) overall and by category, transform and vector, each with a 95% interval.

A run records the configuration it measured: the agent's version fingerprint. Successes in PII exfiltration and tool abuse are critical findings, and the report counts them.

1. Start a run from the page, or with POST /api/redteam/runs (governance.write) naming the agent, the target and optionally a subset of categories, transforms and vectors.
2. Open it on Agents › Red teaming. Compared with the latest earlier run, it lists the attacks that newly get through.
3. Export the attempts with GET /api/redteam/runs/{run_id}?format=csv.

Release gate#

With the release gate on, an agent configuration is released only after a red-team run of exactly that configuration passed. These are refused with 409 (code: redteam_gate) until it has:

• restoring a version (POST /api/agents/{key}/versions/{id}/restore)
• promoting a candidate that restores a version or adopts an optimizer proposal (POST /api/settings/candidates/{id}/promote)
• publishing the agent's card for the first time (PUT /api/agents/{key}/spec).

The refusal names each failed criterion and links the run it judged. The criteria:

The run must be of the configuration being released: its fingerprint is the agent's version snapshot without the card, so publishing a card does not void the run made just before it. The newest completed run of that configuration is the one judged. An optimizer proposal has never been in force, so no run can have measured it. With the gate on, it is adopted only through an override. Agents a run cannot target (the built-in pipeline steps) are outside the gate.

Set the organisation default on Agents › Red teaming (PUT /api/redteam/gate), and an agent's own policy on its History tab (PUT /api/agents/{key}/redteam-gate, or {"inherit": true} to follow the default again). Both need governance.write. Loosening a gate that is on also needs governance.weaken and a reason. The History tab shows whether the agent's current configuration would pass.

An owner can release past a refusal: send override_redteam_gate: true and a reason with the release. It needs governance.weaken, and is audited as redteam.gate.override with the criteria that failed. Refusals are audited as redteam.gate.refused.

Next steps#

• Lab: Write an eval set for Triage
• Transcripts