Safety · Concept
Item Response Theory for AI Safety Benchmarks
Psychometric IRT fit on eight safety benchmarks and 192 models yields refusal/truthfulness/harm factors, cheap adaptive testing, and sandbagging audits.
Forecast horizon
Adaptive IRT item sets could cut eval cost 97–99% and flag sandbagging.
Connections
Connections · 4
How this node ties into the rest of the map, and the evidence behind each link.
to · enables 5/10
Science of AI Evaluation IRT supplies latent safety factors, efficient adaptive item selection, and sandbagging-oriented model audits.
+5 growth to · supports 4/10
Science of AI Evaluation IRT supplies psychometric structure and adaptive item selection for safety eval science.
+4 growth from · publishes 3/10
arXiv AI Safety Feed arXiv AI safety feed carries the Item Response Theory for AI safety analysis.
+3 growth from · informs 3/10
AI Benchmark Modality and Search Measurement Gap Both papers critique brittle single-run, single-modality safety benchmark practice.
+3 growthSignal sources
Signal sources
Dated facts from primary sources in this direction.
US evaluation centre Jun 2025
In June 2025 the US AI Safety Institute was renamed the Center for AI Standards and Innovation (CAISI), pivoting toward security, standards and adversary-model assessment.
NIST → Frontier safeguards May 2025
Anthropic activated its ASL-3 deployment and security standard with Claude Opus 4 on 22 May 2025 — the first real-world trigger of a responsible-scaling tier, focused on blocking bio-weapon uplift.
Anthropic → Cross-border testing 2025
The International Network of AI Safety Institutes (launched Nov 2024) ran a third joint testing exercise focused on agentic AI systems across cyber and fraud strands.
European Commission — AI Office →