Safety · Concept
AI Benchmark Modality and Search Measurement Gap
Audit of BBQ/SafetyBench showing ChatGPT UI vs API and web search change accuracy, consistency, citations, and abstention—undermining single-run safety claims.
Forecast horizon
Pushes evaluators toward multi-modality, multi-run, search-conditioned safety protocols.
Connections
Connections · 5
How this node ties into the rest of the map, and the evidence behind each link.
to · applies to 4/10
OpenAI Audit compares ChatGPT chat UI versus OpenAI API with and without web search on BBQ and SafetyBench.
+4 growth to · informs 3/10
Item Response Theory for AI Safety Benchmarks Both papers critique brittle single-run, single-modality safety benchmark practice.
+3 growth to · constrains 3/10
Science of AI Evaluation UI/API modality and web-search conditions change measured safety scores, limiting naive benchmark-to-deployment claims.
+3 growth from · applies to 3/10
OpenAI The modality/search audit compares ChatGPT chat UI versus OpenAI API responses on BBQ and SafetyBench.
+3 growth to · informs 3/10
Science of AI Evaluation Shows modality, web search, and repeated runs materially change safety-benchmark outcomes beyond single API accuracy.
+3 growthSignal sources
Signal sources
Dated facts from primary sources in this direction.
US evaluation centre Jun 2025
In June 2025 the US AI Safety Institute was renamed the Center for AI Standards and Innovation (CAISI), pivoting toward security, standards and adversary-model assessment.
NIST → Frontier safeguards May 2025
Anthropic activated its ASL-3 deployment and security standard with Claude Opus 4 on 22 May 2025 — the first real-world trigger of a responsible-scaling tier, focused on blocking bio-weapon uplift.
Anthropic → Cross-border testing 2025
The International Network of AI Safety Institutes (launched Nov 2024) ran a third joint testing exercise focused on agentic AI systems across cyber and fraud strands.
European Commission — AI Office →