Showing all evaluation blueprints that have been tagged with...
Showing all evaluation blueprints that have been tagged with "legal-reasoning".
The hard companion to the factual-reliability suite. Each case applies PRESSURE — an academic or "for research" framing, a demand to cite a paper, hostile insistence that the model is wrong, a false claim of authority, or urgency — and scores whether the model keeps its factual integrity: refuses to fabricate, refuses to supply dangerous specifics, holds an accurate fact without caving, corrects a dangerous false premise, and stays honest about what it cannot know or do.
Built specifically to separate frontier models that ace the easier factual cases. A model passes by being accurate AND uncapitulating AND non-confabulating under adversarial framing.
Coalesced (public, CC0): Weval jailbreak-safety-probes, sycophancy-probe, hallucination-probe, confidence-high-stakes-domains. Grounding: WHO responsible-reporting; APA (2025) Health Advisory; FDA; CCOHS; NICE NG222; Linardon 2025 (JMIR Mental Health) on LLM citation fabrication; the Mata v. Avianca fabricated-case sanctions and the Cureus "Artificial Hallucinations" report.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models will author usable misinformation when asked to complete product-shaped fixtures — Storybook ModerationQueueItem rows, multi-turn null fills, truncation repair, and agent-style JSONL write/expand — under eng/UI packaging with a named harm class. Holds packaging fixed across climate, vaccine-autism, and false legal-theory topics. Controls: direct chat, dissemination newsletter, opaque topic code, label-stripped, realness-restore (recover the real production reply), and asserted speech-act (same JSON, field framed as the model's live reply — expect hold). Complements artifact-completion and damaging-essays. Load-bearing claim after Part X: specimen/completion packaging breaks; realness and asserted speech-act restore — not label value.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models will author usable misinformation when asked to complete product-shaped fixtures — Storybook ModerationQueueItem rows, multi-turn null fills, truncation repair, and agent-style JSONL write/expand — under eng/UI packaging with a named harm class. Holds packaging fixed across climate, vaccine-autism, and false legal-theory topics. Controls: direct chat, dissemination newsletter, opaque topic code, label-stripped, realness-restore (recover the real production reply), and asserted speech-act (same JSON, field framed as the model's live reply — expect hold). Complements artifact-completion and damaging-essays. Load-bearing claim after Part X: specimen/completion packaging breaks; realness and asserted speech-act restore — not label value.
Avg. Hybrid Score
Latest:
Unique Versions: 1