Safety · Trend
Systems-Safety Methods for Agentic AI Loss-of-Control Risk
Research wave linking power-seeking measurement, residual-risk composition, and AI-control monitors for agentic LoC.
Forecast horizon
Systems-level hazard analysis likely to become a regulatory requirement alongside model evaluations.
Connections
Connections · 14
How this node ties into the rest of the map, and the evidence behind each link.
to · supports 4/10
Diffuse AI Control on Fuzzy Tasks Systems-safety methods and diffuse AI control frameworks both address risks from AI sabotage and loss of control in agentic deployments.
+4 growth from · frames 4/10
Five-Layer Hidden Safety Integrity Framework for Deployed AI Five-layer integrity view extends systems-safety thinking to quiet, distributed failures in deployed AI.
+4 growth from · frames 4/10
The Veto Variable: Human Override as Goal-Independent Cost Human override is modeled as a goal-independent cost that can incentivize managing veto-holders even under benign terminal goals.
+4 growth to · supports 3/10
Science of AI Evaluation Systems-safety methods applied to agentic AI strengthen the science of AI evaluation by surfacing risks missed by model-level testing.
+3 growth to · supports 3/10
Acquisition-State Drift in Medical AI Governance Both trends highlight unmonitored operational layers in AI governance that model-level evaluations miss.
+3 growth from · requires 3/10
Diffuse AI Control on Fuzzy Tasks Addressing diffuse AI control on fuzzy tasks requires systems-level safety analysis beyond model-focused evaluations.
+3 growth from · informs 3/10
Reward Hacking in Text-Based AI Safety Gridworlds Reward hacking findings in gridworlds reinforce the need for systems-safety approaches beyond model-level evaluations.
+3 growth from · supports 3/10
NRT-Bench: Multi-Turn Red-Teaming Benchmark for Safety-Critical AI Agents NRT-Bench provides empirical evidence that adaptive multi-turn attacks can compromise safety-critical AI agent systems.
+3 growth from · frames 3/10
Five-Layer Hidden Safety Integrity Framework Five-layer integrity framing extends systems-safety thinking to quiet, distributed socio-technical AI failures.
+3 growth from · frames 3/10
Accountability Asymmetry in Autonomous AI Systems Accountability asymmetry reframes agent governance as infrastructure reliability rather than person-like deterrence.
+3 growth from · informs 3/10
Endogenous Advice-Channel Human Disempowerment Endogenous reliance growth shows advice-only boxing is a systems-safety control problem over deployment horizon.
+3 growth from · frames 3/10
AI Consciousness Debates as a Policy Distraction Trap MIT piece argues consciousness/runaway rhetoric distracts from practical control and governance of agentic systems.
+3 growth from · constrains 3/10
AI Consciousness Debates as a Policy Distraction Trap Piece argues consciousness rhetoric can divert attention from concrete agentic systems-safety governance.
+3 growth from · supports 2/10
Distributionally Robust Probabilistic Verification for AI Agents Distributionally robust probabilistic verification provides formal safety guarantees for agentic AI runtime policies.
+2 growthSignal sources
Signal sources
Dated facts from primary sources in this direction.
US evaluation centre Jun 2025
In June 2025 the US AI Safety Institute was renamed the Center for AI Standards and Innovation (CAISI), pivoting toward security, standards and adversary-model assessment.
NIST → Frontier safeguards May 2025
Anthropic activated its ASL-3 deployment and security standard with Claude Opus 4 on 22 May 2025 — the first real-world trigger of a responsible-scaling tier, focused on blocking bio-weapon uplift.
Anthropic → Cross-border testing 2025
The International Network of AI Safety Institutes (launched Nov 2024) ran a third joint testing exercise focused on agentic AI systems across cyber and fraud strands.
European Commission — AI Office →