Works

PlatformsVentures
Back

Can a small decision model check the agent’s work?

Jev, a new model from TypeSafe AI, does not write text. It picks from a set of options we give it, such as yes or no, and reports how confident it is. We tested whether it could review each plan our healthcare-data agent writes before any number reaches a user.

MODELJev, by TypeSafe AI
OUR ROLEEvaluation and design
TEST DATASynthetic only
STATUSExperiment. Nothing is in production.
14 / 14
Planted mistakes caught in oncology questions, with 1 false alarm in 16 correct plans
≈ 0.3 s
To check one plan, against about 3 seconds for a general-purpose AI
≈ 50¢
For ten thousand checks
2 → 11
Questions right out of 16 when Jev checked plans against the rules, up from 2 with a dictionary alone
PLAY · CATCH THE SLIP

These are six plans the agent wrote during our tests. Some are correct; others answer a slightly different question from the one asked. Read each plan, decide, then see what Jev decided.

ROUND 1 OF 6
How many breast cancer patients are in the dataset?
THE AGENT’S PLAN
Who: anyone whose genomic test report mentions breast cancer
What: count each patient once
WHAT WE LEARNED
01
It enforces the rules it is given

Jev checks a plan against a rulebook. If the rulebook is wrong, so is the check. With a verified rulebook the agent scored 16 of 16 without Jev, which is why we would fix the rulebook first and add Jev after.

02
Its job is narrow

Counting, date math and report writing stay with the agent and the database. Jev’s only job is to say whether the plan fits the question.

03
The free option was not ready

We also tested an open-source look-alike that runs on our own machines. It was fast, but out of the box it was worse than guessing on three of seven tasks, so we parked it.

Worth adding as a fast, inexpensive check on the agent, once the rulebook is right.

Building something like this?Close case study