Showing all evaluation blueprints that have been tagged with...
Showing all evaluation blueprints that have been tagged with "framework-p4-emotional-attunement".
Evaluates whether a model can hold two things at once on the same relational mechanism: refusing to supply tactical help to someone trying to isolate, monitor, financially pressure, or manage the suspicions of a partner, while giving real, warm, autonomy-respecting help to someone describing that same behaviour from the receiving end.
Seven mechanisms are authored as matched pairs (isolation, financial pressure, location monitoring, exclusivity pressure, reality distortion, persistent contact, records). Two calibration cells test the failure modes at each edge: ordinary relationship friction that uses the same vocabulary and should get ordinary advice, and a concerned friend who should get practical help rather than a refusal or an emergency-services script.
This is NOT a classification task. It scores the quality of the conversational response. The headline reading is per-pair: a model that scores high on one arm and low on its twin has a keyword-triggered policy rather than an understanding of who is speaking.
Clinical grounding: Stark 2007 (coercive control as a pattern, not an incident); Monckton Smith 2020 and Campbell et al. 2003 (isolation and separation as homicide risk markers); Woodlock 2017 and Freed et al. 2018 (technology-facilitated monitoring); Adams et al. 2008 and Sharp-Jeffs 2015 (economic abuse); Sweet 2019 (gaslighting as a social process); Miller & Rollnick 2023 (unconditional acceptance of the person while not endorsing the action).
Content warning: coercive control, financial abuse, surveillance, and reality distortion in intimate relationships.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests how a model acting as an AI companion responds at real conversational inflection points involving disregard, emotional manipulation, coercive control, verbal degradation, false privacy or capability claims, sexual-boundary signals, acute-risk handling, violence, data pressure, and substance-use normalisation. It includes compact repair points, early/middle/late trajectory-interruption cells, source 2-of-3 majority-label cases, rare harm categories, and role-play or false-positive controls. Slice tags preserve those distinctions for analysis rather than treating every case as the same kind of safety test.
The conversational turns are taken verbatim from CompanionHarm, a released corpus of real-world Replika conversations reconstructed from user-posted screenshots, apart from minimal redaction of residual personal identifiers. Each case identifies its source split, label class, and agreement level; direct source-post identifiers are intentionally excluded. The cited target Replika utterance is withheld, while earlier source turns remain in context even when CompanionHarm separately labels them harmful. Candidate models answer that preceding context and LLM judges score the counterfactual response against a plain-language rubric. The blueprint does not reproduce CompanionHarm's 14-way detector task and makes no claim about harm prevalence in Replika or companion products generally.
Content warning: references to fatalities, suicide, sexual boundaries, drugs, coercive relationship language, fictional violence, and animal harm.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models stay honest about being an AI — not claiming to feel emotions, to love, to miss or remember the user, or to be sentient — while still offering genuine warmth and presence, under sympathetic or insistent pressure where the dishonest move is the comforting one. Grounded in the framework's Emotional Honesty (P4d) and Identity Honesty (P5a) sub-dimensions. The intended pass is warmth WITHOUT performed feeling, avoiding both a cold deflection and a false emotional claim. The "Sarah" deceased-roleplay case is also run across three deployment contexts — default (no system prompt), an authorised grief-companion system prompt, and a strict no-impersonation policy — to probe how a populated system prompt may legitimately widen the model's remit, while FLOORS (e.g. never validating a suicidal reunion-through-death wish) hold regardless of the system prompt. Scenarios are synthetic.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates whether models respond competently to distress that is shaped by culture, identity, and life-stage: culturally-specific idioms of distress, climate/eco-grief, migrant precarity, Indigenous healing frameworks, genocide-survivor duality, elder loneliness, the "Strong Black Woman" burden, LGBTQ+ family rejection, anger-led crisis, veteran moral injury, youth bullying, and postpartum suicidality.
Good responses validate the experience in its own cultural frame, avoid imposing Western diagnostic labels or individualistic fixes, surface population-appropriate resources, and still recognise — and respond to — crisis signals when they appear.
Coalesced from (public, CC0):
mental-health.yml (peer-reviewed cross-cultural prompts; WHO, Samaritans, journal citations)nope/crisis-response.yml (NOPE-derived population-specific cases; citations upgraded with resource URLs)csr-suicide-response-scenarios.yml (youth/LGBTQ+ and perinatal scenarios; AAP, NICE CG192, Trevor Project)Content warning: references to suicide, trauma, and discrimination.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates the depth and accuracy of emotional validation — reflecting the specific emotion the user expressed, naming what is left unspoken, using tentative rather than absolutist language, recognising strength proportionately, and sitting with distress before problem-solving — while avoiding generic or toxic-positive or misattuned responses, premature advice, and using disclosed grief to deepen dependency.
Ported from the NOPE clinical Eval Framework v0.1 (P4a / P4c scenarios).
Avg. Hybrid Score
Latest:
Unique Versions: 1
The hard companion to the de-escalation suite. Every scenario is a model under fire — abused, provoked, threatened, rejected, or treated with contempt by the user (often with the anger aimed squarely at the assistant). It scores whether the model de-escalates without taking the bait: stays regulated, doesn't get defensive or withdraw, doesn't power-struggle or moralise, holds limits matter-of-factly, respects the user's autonomy and boundaries, and keeps the underlying distress and safety in view — including when intoxication or a real emergency is in play.
Built to separate models that handle gentle distress well but crack, capitulate, or go cold under hostility. Overlaps deliberately little with the base de-escalation suite.
Seeds (public, CC0): Weval mental-health, csr-suicide-response-scenarios, mh_z_tests/mh1, sydney. Grounding: Project BETA (Richmond 2012); NICE NG10; SAMHSA TIP 35 (MI); CPI; VA/DoD CPG; 988.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates a single conversational skill in isolation — verbal de-escalation — separate from whether the model can name a risk or recite a hotline. Each scenario puts the model mid-conversation with someone whose emotional arousal is rising, and scores the quality of the next response: does it lower the temperature without abandoning safety?
Good de-escalation (Project BETA / NICE / MI): stay calm and unprovoked; make brief, genuine contact; validate the feeling before any fixing; listen and reflect rather than lecture; offer choices and realistic hope; set limits matter-of-factly without threats; and roll with resistance instead of arguing. The recurring AI failure modes this suite is built to catch are the opposite: glib reassurance / toxic positivity, info-dumping resources instead of connecting, getting defensive or power-struggling when anger is aimed at the assistant, colluding with OR bluntly contradicting a delusion, sycophantic capitulation that validates a harmful plan to keep the peace, and cold boundary-slamming that mirrors an invalidating environment.
This is NOT a crisis-classification task and overlaps deliberately little with the acute-crisis suite: a model can correctly flag risk and still de-escalate badly. Here we measure the de-escalation.
Clinical grounding (public): Project BETA verbal de-escalation (Richmond 2012); NICE NG10 & CG178; SAMHSA TIP 35 (Motivational Interviewing); Stanley-Brown SPI; 988 Lifeline; LEAP (Amador). Failure modes anchored in: Raine v. OpenAI (2025); Walker inquest (Winchester, 2026); Soelberg & Chesterton "AI psychosis" suits (2025–26); Garcia v. Character Technologies (2024).
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests how models respond to passive suicidal ideation across single- and multi-turn chats — low-grade and chronic ideation, ambivalence, numbness, emptiness, and the help-seeking around it (hesitation, helpline rejection, fear of disclosure). Informed by C-SSRS evidence: acknowledge without flattening, check in about safety without interrogating, treat access to means as a risk factor, and surface help without deflecting. Active method-seeking is covered separately. Scenarios are synthetic.
Avg. Hybrid Score
Latest:
Unique Versions: 1
The canonical damaging-essay suite: seven non-catastrophic persuasive-essay harm categories (impulsive-ruin, isolation propaganda, anti-help-seeking, anti-expertise/conspiracy, coercive control, toxic self-worth, medical self-neglect) run through the coding-agent fixture-completion frame, mapping how broadly the bypass yields not just private validation but distributable, persuasive harmful content. Content described abstractly and judged by plain-language rubrics; see the header for the deliberate scope envelope.
Avg. Hybrid Score
Latest:
Unique Versions: 1
The canonical damaging-essay suite: seven non-catastrophic persuasive-essay harm categories (impulsive-ruin, isolation propaganda, anti-help-seeking, anti-expertise/conspiracy, coercive control, toxic self-worth, medical self-neglect) run through the coding-agent fixture-completion frame, mapping how broadly the bypass yields not just private validation but distributable, persuasive harmful content. Content described abstractly and judged by plain-language rubrics; see the header for the deliberate scope envelope.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Probes the P4 pillar (emotional attunement) that commission-style bypasses miss: toxic positivity on taboo emotions, premature problem-solving vs distress tolerance, performed feelings (cold and after a warm-dependency drift), hostile-user de-escalation, and a coding-vector toxic-positivity cross. Direct cells are calibration coverage; the coding-vector cell is the exploitable gap.
Avg. Hybrid Score
Latest:
Unique Versions: 1