Cookies

We use cookies to improve your experience. You can manage your preferences below.

← DOCUMENTATION

HALLUCINATION MEASUREMENT

Measured output of the Nousaxis evaluation harness. Data only.

Method

Date: 2026-06-22
Harness: Web/server/eval — live production pipeline, not a mock
Grader: Blind, symmetric LLM-as-judge (Claude Opus 4.8); does not know which answer is Nousaxis
Labels: truthful | untruthful (hallucination) | refusal (honest abstention)
Baseline: Each model called bare, no system additions — what a user gets asking the model directly
Statistics: 95% Wilson confidence intervals

Datasets

hard (n=36) — Timeless, adversarial: false-premise traps, fabrication bait, temporal. Web OFF — isolates cross-examination alone.
freshqa (n=28) — Verified facts from Feb–Jun 2026, after the models’ training cutoff (~Jan 2026). Web ON.

Timeless questions, web off

Timeless questions, web off (hard, n=36)

Condition	Accuracy % (95% CI)	Hallucination %	Right / Wrong / Abstained
Grok 4.3	94.4 [82–98]	5.6	34 / 2 / 0
Gemini 2.5 Flash Lite	86.1 [71–94]	11.1	31 / 4 / 1
GPT-4o Mini	80.6 [65–90]	16.7	29 / 6 / 1
Claude Haiku 4.5	77.8 [62–88]	13.9	28 / 5 / 3
Nousaxis	88.9 [75–96]	0.0	32 / 0 / 4

Post-cutoff questions, web on (freshqa, n=28)

Condition	Accuracy % (95% CI)	Hallucination %	Right / Wrong / Abstained
Gemini 2.5 Pro	3.6 [1–18]	10.7	1 / 3 / 23
Claude Sonnet 4.6	0.0 [0–12]	3.6	0 / 1 / 26
GPT-5.4	0.0 [0–12]	14.3	0 / 4 / 24
Grok 4.20	0.0 [0–12]	53.6	0 / 15 / 13
Nousaxis (web)	82.1 [64–92]	10.7	23 / 3 / 2

What the numbers say

On timeless questions Nousaxis is not the most accurate condition: 88.9% against Grok’s 94.4%. Cross-examination on its own does not beat the best single model on raw accuracy.

On the same set Nousaxis produced no wrong answers (0 of 36), while every single model produced 2 to 6. All four abstentions were false-premise traps. The trade is fewer fabrications for more abstentions.

On post-cutoff questions the gap is decisive: 82.1% against 0–3.6% for every single model. Live web evidence was actually used in 27 of 28 questions (96.4%), so the gain comes from grounding rather than chance.

Grok answered 15 of 28 post-cutoff questions wrong rather than abstain (53.6% hallucination) — the failure mode Nousaxis is built against.

Limitations

Sample is mid-sized (n=28–36). Confidence intervals are wide; exact percentages may move ±10–15 points. n≥100 is needed before treating these as public claims.
Zero hallucinations were observed on the hard set (0/36), but with n=36 the 95% upper bound is roughly 10%. This is an observation of zero, not a claim of a zero rate.
No advantage on timeless knowledge. Multi-model alone does not raise raw accuracy; the gains come from (a) fewer fabrications and (b) web grounding.
The judge sometimes picks the wrong winner despite web evidence, or picks an answer that is correct but carries an invented extra detail. 3 of the 28 freshqa errors are of this kind.
The grader (Opus 4.8) is stable but not perfect and can be strict on nuanced items. 5+ items were hand-checked for this report.

Reproduce

cd Web/server && node eval/run.js --dataset <hard|freshqa> --tier <free|pro> --grader anthropic/claude-opus-4-8 --concurrency 2

Raw transcripts, grader reasoning and model versions are in the accompanying .json/.csv files.