Tooling case study
How we used agents to evaluate Jev—and changed our minds about its best use
Published · Research through
Executive summary
Jev showed promise as a cheap way to decide which claims deserved closer inspection. It did not establish a clear improvement in the broader agent task, and a high score could still conceal an error.
These were small, internal experiments with model-based review, not an independent human-labelled benchmark. Research through 24 September 2026; Jev 1.13.0.
Consider a hypothetical research note about rising electricity demand from data centres. It could inform questions about power supply, technology investment or the economics of AI. Deciding where it is useful requires understanding the note’s main claim and the question being investigated.
Our first tests concerned that kind of judgment. Jev improved candidate ranking compared with keyword search. But a capable agent could already search and reason for itself. We therefore compared the resulting agent decisions and effort.
The 20-case comparison found no clear overall efficiency gain. Useful leads still needed investigation. We retained ordinary agent search as the default for the tested task.
Each of 20 cases received two fresh agent runs: one with Jev suggestions and one without. Separate model reviewers assessed the resulting decisions without knowing which method produced each answer.
| Measurement | Without Jev | With Jev |
|---|---|---|
| Decisions judged defensible | 20/20 | 20/20 |
| Decisions judged appropriately specific | 18/20 | 19/20 |
| Mean composite time | 46.21 seconds | 45.67 seconds |
| Median composite time | 41.52 seconds | 41.96 seconds |
The small difference in means favoured assistance; the medians slightly favoured the ordinary agent. Jev reduced searches, but did not remove the need to read and validate candidates. These results did not establish a meaningful speed advantage.
The sample was small and retrospective, with one run per method for each case. The reviewers belonged to the same model family as the task agents; their judgments were not independent human ground truth.
Composite timing combined separately measured components. Startup, caching and concurrent execution limited precision. The result concerns this assistance design and sample, not every possible use of Jev. Better search ranking remained a valid finding, even though the case for adopting the tested agent assistance remained unproven.
Reading an earnings call means keeping figures attached to their meaning. A quarterly investment figure must stay distinct from cumulative spending. A range must remain a range. A short clarification can change the meaning of a long answer.
Across three transcripts from two companies, we compared existing extracted material, passages selected by Jev, and a combination that sent material from Jev to a stronger model for further work. The combination covered more of a separately generated reference set.
But the stronger model had done substantial work, and we had not tested it on its own under matched conditions. We could report the improvement without yet crediting Jev with causing it.
Separate full-source model reads generated the comparison reference set. Under the model-based assessment, the combined approach covered 391 of 448 items; the existing extracted material covered 179. These are coverage judgments, not verified numeric-extraction accuracy.
The combination sent about two-thirds of the input units for stronger-model review. It had not demonstrated that Jev removed much of that work. Output formats also differed, so a single percentage could not capture every quality difference.
We needed a fresh experiment in which the stronger model performed the task without Jev, using comparable material and conditions. That comparison had not been run.
The improvement could have come from the stronger model, the additional attention, or the way the task was presented. The reference set was model-generated rather than human gold. The useful next test is to isolate Jev’s contribution and measure the quality of the final claims alongside the total work required.
Examining errors made the opportunity concrete. Revenue had been interpreted as operating profit, even though the sentence supplied a separate profit figure. A 60–70% range had become 60%. A quarterly investment amount had been paired with a cumulative period.
A number can appear in the source and still support the wrong claim. We tried using Jev to identify claims that deserved a closer look.
The later audit produced the clearest positive result: all 30 selected low-scoring claims outside the initial pilot documents were judged defective by a blinded model reviewer. That supported a review queue, with an important limit: recurring numerical errors in one company accounted for the findings.
After observing error types in the transcript pilot, we designed a short set of semantic checks and scored 333 claims. Three source-confirmed errors appeared first, fifth and sixteenth in the resulting review order.
The questions were designed after seeing the error types. This was useful exploratory evidence, not a test on examples independent of question design. It motivated a follow-up using the same questions on additional documents.
We scored 3,670 extracted claims with the questions unchanged. A fresh model reviewer examined the 30 lowest-scoring claims outside the initial pilot documents, without seeing their scores or knowing the selection group. It judged all 30 defective.
Those claims came from 11 transcripts of one company. They shared recurring numerical extraction patterns. Thirty findings were not 30 independent tests across businesses and error mechanisms, and the result does not establish overall detection accuracy or recall.
Scoring the 3,670 claims took approximately 6.6 minutes. The input-token estimate was about US$0.30 at the price recorded for the experiment. TypeSafe’s pricing documentation describes the token-pricing basis.
That estimate covers Jev scoring only. It excludes review and correction work, is not an invoice, and is not a measured saving against another complete process.
An earnings-call transcript described EBITDA falling by 20%. An extracted claim turned that into growth of 20%. The number survived; its meaning was reversed.
Jev gave the claim a support score of 0.86. A later model reviewer, reading the source without seeing that score, caught the error. We found it because we checked apparent successes as well as suspicious results.
A later annual-report test reinforced the limit. Semantic checks caught some wrong interpretations, while rule-based checks caught unit and component-versus-total problems that Jev did not reliably prioritise. A high score could not serve as automatic approval.
For claims scoring at least 0.8, the blinded reviewer judged 18 of the 20 sampled claims correct, one defective and one uncertain. The sign reversal was the defective claim.
One error in this sample does not measure the share of all errors that Jev missed. Most score ranges had not received equivalent review, and the results did not establish calibrated probabilities or a safe acceptance threshold.
The same checks were applied to 435 narrative claims from two annual reports of the same company. They surfaced three instances in which a statutory tax rate had been interpreted as an effective tax rate. One describes a rate set by tax rules; the other concerns tax expense relative to profit before tax. Finding the percentage in the source did not make the interpretation correct.
Jev did not reliably prioritise six known defects already identified by rule-based checks, involving units and the distinction between a component and a total. It also raised false alarms when table excerpts lacked the headers needed to interpret their columns. A claim could be correct on the source page while looking doubtful in a short passage.
Neither method covered every problem. Source context and the scope of the question mattered; a score alone could not settle an ambiguity that required reading the page.
The agents helped turn observations into tests. Fresh sessions carried out paired tasks. Separate reviewers assessed alternatives without method labels. Source checks examined whether the number, measure and context belonged together. Recorded measurements made attractive results open to challenge.
We began by asking whether Jev could improve several parts of research. The evidence supported a more specific candidate use: helping decide what deserves closer inspection. It did not justify automatic acceptance, rejection or repair.
This remains a provisional recommendation from internal experiments. Model-based review is useful evidence, but it does not become human ground truth because the reviewer is a separate agent.
Jev, from TypeSafe AI, evaluates supplied material through constrained questions: choosing among options, returning a score or assessing a yes/no proposition. TypeSafe’s API documentation describes the question types. A valid output format does not establish correct interpretation.
Our experiments used Jev 1.13.0 and covered research classification, earnings-call extraction and checks on extracted claims. This article summarises work completed by 24 September 2026. It is not a universal product ranking. Experimental results are our own; vendor links support the product description and pricing basis. Private experimental materials are not included.
A matched stronger-model-only extraction comparison is needed to isolate Jev’s contribution. Broader document coverage and independent human-adjudicated examples would help test whether the review ranking generalises. More of the score distribution must be reviewed before making calibration or missed-error claims.
The present evidence supports checking suspicious claims against sources, retaining rule-based checks and continuing to sample high-scoring claims. The revenue mistaken for profit shows why a review queue matters. The decline mistaken for growth shows why further review remains necessary.