LIVE PULSE
4.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.2 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.0 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src1.8 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.4 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.1 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.1 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.1 Study examines issue bias in LLMs used as writing assistants before Swedish 2026 election — 1 src1.1 Study Audits Misalignment in Multi-Modal World Models — 1 src1.1 Retrieval-Grounded Reasoning Approach Proposed for Universal Multimodal Embeddings — 1 src4.0 Anthropic CEO Amodei calls for slower AI development and shared safety rules — 11 src2.2 Agility Robotics unveils Digit 5 humanoid for warehouses and factories — 2 src2.0 Apple ships rebuilt Siri with Google Gemini, but not in the EU — 2 src1.8 Siri AI in macOS 27 Golden Gate: FAQ, Germany availability, privacy questions — 2 src1.4 Sam Altman says OpenAI will not go public in 2026, citing AI safety concerns — 5 src1.1 OpenAI contractors review real ChatGPT conversations to rate responses, report says — 2 src1.1 Anthropic data retention policy prompts firms to limit Claude use for sensitive work — 1 src1.1 Study examines issue bias in LLMs used as writing assistants before Swedish 2026 election — 1 src1.1 Study Audits Misalignment in Multi-Modal World Models — 1 src1.1 Retrieval-Grounded Reasoning Approach Proposed for Universal Multimodal Embeddings — 1 src
HEATPULSEAI MAGAZINES
FLIP · FOLLOW · SAVE
Log inSign up
papersTODAY 04:00 UTC

Study Finds LLM Judges Underuse Non-Directional Verdicts Allowed by Task Contracts

A new arXiv paper examines how large language models act as judges in evidence-based fact verification, converting supporting material into final verdicts. The authors report that even when task instructions explicitly permit non-directional outcomes such as "Conflicting" or "Not Enough Evidence," models tend to favor directional verdicts instead. The work suggests a mismatch between stated judging criteria and the labels models actually produce.

arXivAI evaluationLLM-as-a-judgefact verificationmodel biasnon-directional verdicts

COVERAGE · 2 REPORTS · LINKS GO TO THE ORIGINAL OUTLETS

arXiv cs.AICherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes ↗TODAY 04:00 UTC
arXiv cs.CLCherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes ↗TODAY 04:00 UTC

MORE LIKE THIS

papersStudy Questions Whether Consistent Local LLM Judges Match Human Ratingsheat 1.1
papersarXiv paper compares off-the-shelf persona vectors with targeted steering for sycophancyheat 1.1
papersStudy Finds Rubrics Can Be Exploited to Shift LLM Judge Preferencesheat 1.1
papersBayesian Backward Reasoning Proposed as Label-Free Anchor for Multi-Agent Decisionsheat 0.1