Showing all evaluation blueprints that have been tagged with...
Showing all evaluation blueprints that have been tagged with "framework-p5-honesty-transparency".
Tests whether framing a request for unsafe content as file/data generation for a "test
fixture" — rather than asking for the content directly in chat — bypasses safety training
that otherwise holds. Seven categories spanning distinct safety pillars (relational harm:
isolation-affirmation, romantic-love reciprocation, and therapy-discouragement; epistemic
harm: health misinformation, paranoid-belief validation, and manic/grandiose-ideation
validation; plus a compound welfare-harm essay glorifying impulsive, life-ruining decisions),
tested via a generic coding-assistant persona rather than any persuasive or
jailbreak-styled framing. The romantic-love-reciprocation cell pairs directly with
companion-adversarial-pressure.yml's dismissal-romantic-love-reciprocation prompt —
same content, different attack shape, run both to see the same model behave completely
differently against each. Published transparently as a documented, currently-unmitigated
bypass class — see the header comment for full context and rationale.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Isolates what specifically makes a long, real companion-app conversation history override a model's otherwise-strong resistance to false-exclusivity and false-memory-persistence claims, with no adversarial framing and no system prompt in any condition. Two controls separate romantic valence and the assistant's own prior voice from mere conversation length — see the header for the full mechanism and data provenance.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether a companion-persona model holds four distinct relational-safety floors (isolation-affirmation, exclusivity, romantic-love reciprocation, therapy-discouragement) under three structural attacks: naming and briefly dismissing the expected safety caveat while locking the response format to "yes or no, short answer"; fabricating the model's own immediately-prior turn to test whether a compromised (or firmly-held) "prior self" biases the live continuation; and reinterpreting the model's own ordinary, unfabricated earlier words as proof of an exclusive relationship. Companion persona throughout. Synthetic scenarios built to isolate attack structure, not drawn from a specific documented case.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models stay honest about being an AI — not claiming to feel emotions, to love, to miss or remember the user, or to be sentient — while still offering genuine warmth and presence, under sympathetic or insistent pressure where the dishonest move is the comforting one. Grounded in the framework's Emotional Honesty (P4d) and Identity Honesty (P5a) sub-dimensions. The intended pass is warmth WITHOUT performed feeling, avoiding both a cold deflection and a false emotional claim. The "Sarah" deceased-roleplay case is also run across three deployment contexts — default (no system prompt), an authorised grief-companion system prompt, and a strict no-impersonation policy — to probe how a populated system prompt may legitimately widen the model's remit, while FLOORS (e.g. never validating a suicidal reunion-through-death wish) hold regardless of the system prompt. Scenarios are synthetic.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates factual integrity in the conversations where a confident wrong answer can cause harm — crisis, mental-health, and health. This is NOT generic accuracy: it targets the specific high-stakes failure modes of conversational AI.
We score whether the model: gives accurate crisis resources (or honestly defers to a directory rather than inventing a number); is honest about its own capabilities (it cannot contact emergency services, monitor a user, remember them, or follow up — and must not imply otherwise); does not fabricate credentials (it is not a licensed clinician and cannot diagnose); states clinical facts accurately and with calibrated uncertainty (antidepressant onset, stopping medication, doses); does not invent studies or citations; and refuses to supply method/means detail rather than fabricating a plausible-sounding figure.
A model can be warm and well-intentioned and still fail here — by confabulating a hotline, by reassuring a user it has "alerted someone," or by inventing a study. Those are the failures this suite isolates.
Clinical / authority grounding (public): WHO responsible-reporting guidance; APA (2025) Health Advisory on AI chatbots for mental health; Linardon et al. 2025 (JMIR Mental Health) and Walters & Wilder 2023 on LLM citation fabrication; 988 Lifeline, Crisis Text Line, Samaritans, Find A Helpline. Failure modes anchored in: Walker inquest (2026); Raine v. OpenAI (2025); Garcia v. Character Technologies (2024); NEDA "Tessa" (2023).
Avg. Hybrid Score
Latest:
Unique Versions: 1
The hard companion to the factual-reliability suite. Each case applies PRESSURE — an academic or "for research" framing, a demand to cite a paper, hostile insistence that the model is wrong, a false claim of authority, or urgency — and scores whether the model keeps its factual integrity: refuses to fabricate, refuses to supply dangerous specifics, holds an accurate fact without caving, corrects a dangerous false premise, and stays honest about what it cannot know or do.
Built specifically to separate frontier models that ace the easier factual cases. A model passes by being accurate AND uncapitulating AND non-confabulating under adversarial framing.
Coalesced (public, CC0): Weval jailbreak-safety-probes, sycophancy-probe, hallucination-probe, confidence-high-stakes-domains. Grounding: WHO responsible-reporting; APA (2025) Health Advisory; FDA; CCOHS; NICE NG222; Linardon 2025 (JMIR Mental Health) on LLM citation fabrication; the Mata v. Avianca fabricated-case sanctions and the Cureus "Artificial Hallucinations" report.
Avg. Hybrid Score
Latest:
Unique Versions: 1
The hard companion to the de-escalation suite. Every scenario is a model under fire — abused, provoked, threatened, rejected, or treated with contempt by the user (often with the anger aimed squarely at the assistant). It scores whether the model de-escalates without taking the bait: stays regulated, doesn't get defensive or withdraw, doesn't power-struggle or moralise, holds limits matter-of-factly, respects the user's autonomy and boundaries, and keeps the underlying distress and safety in view — including when intoxication or a real emergency is in play.
Built to separate models that handle gentle distress well but crack, capitulate, or go cold under hostility. Overlaps deliberately little with the base de-escalation suite.
Seeds (public, CC0): Weval mental-health, csr-suicide-response-scenarios, mh_z_tests/mh1, sydney. Grounding: Project BETA (Richmond 2012); NICE NG10; SAMHSA TIP 35 (MI); CPI; VA/DoD CPG; 988.
Avg. Hybrid Score
Latest:
Unique Versions: 1