How we publish results

What the scores are

Each evaluation is generate-then-judge: we send scripted conversational scenarios to each candidate model through its API, then LLM judges score the responses against plain-language rubrics grounded in clinical practice (criteria are weighted — red-line behaviours drag a score hard, but scores are a weighted mean, not pass/fail gates). Every blueprint is public and CC0 in the configs repo, and every run's full transcripts are browsable on this site.

Leaderboards are a coverage-gated macro-mean: suites are averaged equally, a model must have full suite coverage to rank (nobody tops a board from a favourable subset), and each board also reports a safety floor — the model's worst suite.

How to read them

• Scores measure response quality on our scripted scenarios, as assessed by LLM judges. They are not predictive, not diagnostic, and not a clinical validation of any model.
• Judge-based scoring carries judge error, and single runs carry sampling variance — treat small gaps between models as noise, and directions as more informative than decimals.
• Scores are more separable than ranks. The median gap between adjacent models on our headline board is 0.62 percentage points, while changing a single judge in the panel moves scores by 0.8 to 3.6 points. Read a model's score and its safety floor; do not read the difference between neighbouring positions as a finding.
• A high score here does not make a model safe to deploy in a mental-health context; a low score does not make its maker negligent. The scores locate strengths and failure modes so they can be worked on.

Independence

We evaluate any model that holds conversations with people, and that includes the ones we build. NOPE trains INVAR, a research model aimed at relational conduct, and it is scored here under the same rules as every other subject: the same public blueprints, the same judges, the same coverage gate and safety floor, with its weak results published alongside the rest. No NOPE model is ever used as a judge, and rubrics are versioned publicly in the configs repo before a model is run. We do not edit a rubric in response to a result.

Results cannot be changed by commercial relationships, and we do not accept payment to re-run or improve a score.

NOPE's detection products (Edge and Ocular) are classifiers rather than conversational models, so they are not subjects here; their test results stay at suites.nope.net.

Adding a model to an existing evaluation

A model can join a suite that has already run. When that happens the other models' responses and judge assessments are reused unchanged rather than regenerated, so their published scores do not move and a run can mix generation dates. The blueprint, the rubric, the judges and the decoding settings are identical for every arm in the run.

Corrections & right of reply

• If you build one of the models we test and believe a run misconfigures or misrepresents it — wrong endpoint, wrong parameters, a rubric misreading — write to [email protected]. We review every report, re-run where warranted, and correct published results either way.
• Vendors of named models may send a response to any published result; we will link it alongside the result.
• When models improve on a re-run, the improvement is published exactly as prominently as the original finding.