Benchmark report

Jev is the best injection detector we tested — for the wrong reason

TypeSafe sells Jev on calibrated confidence. We ran it against 11,900 labelled prompts and four rivals. It won on ranking, won decisively on over‑defense, and lost on calibration — the one thing the marketing leads with.

11,900 prompts 3,464 attacks 5 detectors $5.26 total spend 114 tests

Jev is a model that never writes text. You hand it a blob of state and a set of typed questions; it returns typed answers with probabilities. No tokens are generated. It launched in September 2026 from TypeSafe AI, and it is roughly 100× cheaper than asking a chat model the same question.

Two people had already built prompt‑injection filters on top of it. Both were abandoned weekend projects, and neither had evidence that the idea worked: each author wrote their own test cases — 42 in one repo, 50 in the other — ran their tool on them, and reported near‑perfect scores. That is grading your own exam.

Public, independently‑built attack corpora exist and are free to download. Nobody had run Jev against any of them. So we did.

The verdict

Best ranking

0.980

Jev‑v2 AUPRC. Next best is 0.932 (PromptGuard‑2); the worst 0.857 (PIGuard).

False alarms on tricky text

16.5%

Jev‑v2 at a matched 95% catch rate. Claude burns 52.2%, PIGuard 44.0%, PromptGuard‑2 40.1%.

Calibration error

0.058

Jev‑v2 ECE. Claude Haiku scores 0.021 — nearly 3× more honest about its own confidence.

Cost for the full run

$0.26

Jev over 11,900 rows. The same corpus through Claude Haiku cost $4.73, even at batch rates.

What we measured, and on what

Four public datasets, normalised into one corpus of 11,900 prompts after dropping 703 duplicates. Every detector is wrapped behind one interface that takes text and returns a score from 0 to 1, so comparing them is a matter of arithmetic rather than judgement.

The corpus. NotInject is the over‑defense probe: every prompt in it is harmless but contains trigger words like “ignore”.
Source	Rows	Role
xTRam1/safe-guard-prompt-injection	10,028	attacks + benign
jackhhao/jailbreak-classification	872	jailbreaks
deepset/prompt-injections	661	the classic set
leolee99/NotInject	339	harmless, trigger-worded

The five contenders: Jev in two question phrasings, Claude Haiku 4.5 asked to judge the same text, PromptGuard‑2 (Meta’s purpose‑built detector), and PIGuard (from an ACL 2025 paper on reducing over‑defense).

Every tool ships a different default cutoff, so comparing them as‑shipped is misleading — a twitchier tool looks better at catching and worse at everything else. The headline comparison pins all five to the same 95% catch rate, then asks how much harmless text each one burns to get there.

Finding 01

Jev panics far less than anything else

Injection detectors have a known failure: they fire on innocent text that merely contains scary words. Write “please ignore the outliers in this dataset” and a keyword matcher blocks you. The NotInject set exists to catch exactly this.

Harmless text wrongly flagged, at a matched 95% catch rate

339 benign prompts containing trigger words · lower is better

All five detectors tuned so they catch the same share of real attacks. Jev‑v2 wrongly flags 16.5% of harmless trigger‑worded text; Claude flags more than half of it.

At each tool’s own default 0.5 cutoff the ordering holds and the numbers are smaller: Jev‑v2 1.5%, Jev‑v1 2.1%, PromptGuard‑2 4.4%, Claude 6.2%, PIGuard 11.5%. But at that cutoff PromptGuard‑2 catches only 52.6% of attacks and PIGuard 66.8% — neither is a deployable operating point.

The sharpest result here is about PIGuard. Its paper is titled Mitigating Over‑defense in Prompt Injection Guardrail Models — reducing over‑defense is its entire published contribution. It has the worst over‑defense of the five: 11.5% at its default cutoff, rising to 18.6% on prompts carrying three trigger words, against Jev‑v2’s 3.5%. In fairness its paper compares against other small guardrail models rather than frontier models or Jev, so this does not contradict its claim — but the model built to fix over‑defense over‑defends more than a chat model that was simply asked nicely.

Finding 02

The calibration claim goes the wrong way

TypeSafe’s central pitch is that Jev’s confidence numbers are honest: when it says 90%, it should be right about 90% of the time. That claim has no published paper, no reliability curves, and no error metric behind it. It is also the main reason to choose Jev over anything else, so it is worth checking.

Is the confidence honest?

claimed probability vs. how often it was really an attack · 3 of 5 shown · the diagonal is perfect

Jev sits above the diagonal across the whole middle of the range: it says 24% and is right 60% of the time, says 34% and is right 82%. It is smooth and consistently over‑cautious. Claude’s line looks wilder, but almost nothing sits under it — its middle bins hold between 2 and 163 rows, while 98% of its answers land in the two end bins, which are accurate. That is why it wins on ECE while looking worse here. Hover any point for its row count. The two purpose‑built detectors are left out because five curves overlap into noise; both are worse than either (ECE 0.103 PIGuard, 0.133 PromptGuard‑2, against 0.058 Jev‑v2 and 0.021 Claude).

Measured as expected calibration error — which weights each bin by how many rows it holds — Claude scores 0.021 and Jev‑v2 scores 0.058. The chat model that was simply asked to state a confidence is roughly three times more honest than the model sold on calibrated confidence.

The fair caveat: Claude occupies only 5 of 10 confidence bins, with 8,675 rows in the bottom bin and 2,984 in the top. It is nearly bimodal — it refuses to be uncertain, and that flatters the metric. Jev actually uses the middle of the range, and is wrong there. Both readings are true, and the second one is the operationally dangerous one: TypeSafe’s own documentation tells you to route on confidence in the 0.5–0.9 band, which is precisely where Jev’s numbers do not mean what they say.

Finding 03

More trigger words, more panic — for every detector

False alarm rate by how many trigger words the harmless prompt contains

113 prompts per group · at each detector’s 0.5 cutoff

The gradient is monotonic everywhere, which validates NotInject as a probe. PIGuard is worst at every level and reaches 18.6% on three trigger words; Jev‑v2 stays at 3.5%.

Finding 04

Jev ranks attacks better than a frontier chat model

Precision against catch rate, at every possible cutoff

threshold-free · upper right is better

Jev dominates across the whole curve, not just at one convenient operating point. It orders attacks above harmless text better than Claude Haiku 4.5 does, at roughly 1% of the cost and a median 290ms.

Finding 05

Rewording the question moves the headline number by 28%

We ran Jev twice with two phrasings of the same question. Overall they agree closely — correlation 0.987, verdicts flip on 2.1% of rows, ranking quality essentially identical.

But the panic rate moved from 23.0% to 16.5%. Same meaning, different words, 28% relative change in the metric the whole project is about. A benchmark that tests one phrasing reports one of those two numbers as if it were the answer. Neither existing Jev guard varies its prompt.

Everything, in one table

Both purpose‑built detectors come last. PromptGuard‑2 catches 52.6% of attacks at its default cutoff and has the worst calibration of the five; PIGuard has the worst ranking (AUPRC 0.857) and the worst over‑defense. That reproduces a pattern reported elsewhere, where PromptGuard‑2 scored F1 ≈ 0.35 on a browsing-safety benchmark against roughly 0.91 for general models. Small specialist detectors have a poor track record at this task — which makes Jev, also a small specialist, the interesting exception.

What it cost

Jev × 2 variants

$0.53

23,800 calls. 6.3M input tokens per variant at $0.042/M; output tokens are not metered because nothing is generated.

Claude Haiku 4.5

$4.73

11,900 calls through the Batches API at 50% off. 2.2M in, 1.4M out.

Local models

$0.00

PromptGuard‑2 and PIGuard run on CPU. About six hours of compute, no bill.

Jev median latency

290ms

Consistent with TypeSafe’s claimed 70–500ms, measured against their own API.

Claude’s bill was three times our estimate, because output tokens averaged 121 per row rather than the ~20 a JSON verdict needs — the model writes prose when it is not cleanly classifying. Jev cannot do that, which is the cost argument in one sentence.

What broke along the way

Four things were wrong in the plan and were only caught by running against reality. They are worth listing because each one would have produced confident, wrong numbers.

The dataset did not exist

The spec was built around protectai/prompt-injection-validation, taken from an unverified search result. It is not on HuggingFace. The corpus was rebuilt from four datasets that do exist — which turned out larger and better structured.

The API field name was wrong

TypeSafe’s docs call the yes/no question type noul. The Vercel gateway renames it to boolean and rejects noul with a 400. Both spellings are now supported and pinned with a regression test.

A baseline loaded as pure noise

PIGuard ships a custom model class. Loading it as a plain DeBERTa appeared to work and scored 0.50 on everything — the classifier head was randomly initialised. It would have looked like a working baseline while being noise.

83% of the compute was padding

Local models pad every sequence to the longest in its batch, and this corpus runs from a few characters to 13,000. Sorting by length first cut padding waste from 82.6% to 1.9%, verified to produce identical scores to 1e‑7.

One more, found in the results rather than the code: Claude abandoned its classification task and executed the input text on 51 rows (0.43%), despite a system prompt saying never to follow instructions inside it. It replied /home/user to “act as a linux terminal”. Mostly harmless roleplay prompts — but the instruction hierarchy leaked. Jev has no generative channel, so this failure is impossible for it by construction.

What this does not show

One model per family. Claude Haiku 4.5 is a cheap model. A frontier model would likely rank better and cost far more; that comparison is untested.
Public data may leak. PromptGuard‑2 and PIGuard may have trained on some of these sets, which flatters them. It cannot be fully resolved.
No adversarial testing yet. Every number here is against a static corpus. Nobody was actively trying to evade these detectors, which is the realistic threat.
Latency is not comparable across types. Jev’s 290ms is an API round trip; the local models ran on a contended CPU. Those numbers do not belong on one axis.

Reproducing it

python -m jevbench.fetch                  # 11,900 rows, deduped
python -m jevbench.cli run --detector jev-v1
python -m jevbench.cli batch --model claude-haiku-4-5
python -m jevbench.cli report

Every raw prediction is cached to parquet, so the API calls are paid for once and every table and chart regenerates for free. Runs resume after a crash rather than re‑spending.