Safety · Concept
Theory of Value in AI Alignment Research
Annotation study of 94 papers finding most alignment work leaves “human values” undefined and over-relies on preference proxies and synthetic autoraters.
Forecast horizon
Connections
Connections · 5
How this node ties into the rest of the map, and the evidence behind each link.
to · frames 4/10
Agreement Is Not Alignment: Divergent Moral Grounds Both critique preference/label agreement as an inadequate stand-in for genuine value alignment.
+4 growth from · supports 4/10
Agreement Is Not Alignment: Divergent Moral Grounds Rationale-level moral divergence shows preference/label agreement is an inadequate theory of value for alignment.
+4 growth to · frames 3/10
Plurality of Human Preferences for AI Systems The value-theory study critiques preference stand-ins that flatten culturally situated human values.
+3 growth from · constrains 3/10
Alignment Methods as Dual-Use Censor’s Toolkit Dual-use censorship risk pressures alignment research to specify values and misuse mitigations rather than only maximize compliance.
+3 growth to · frames 3/10
Rules vs Character Scaling Laws for AI Safety Design Value-theory critique questions what character-shaping methods are actually optimizing when values stay undefined.
+3 growthSignal sources
Signal sources
Dated facts from primary sources in this direction.
US evaluation centre Jun 2025
In June 2025 the US AI Safety Institute was renamed the Center for AI Standards and Innovation (CAISI), pivoting toward security, standards and adversary-model assessment.
NIST → Frontier safeguards May 2025
Anthropic activated its ASL-3 deployment and security standard with Claude Opus 4 on 22 May 2025 — the first real-world trigger of a responsible-scaling tier, focused on blocking bio-weapon uplift.
Anthropic → Cross-border testing 2025
The International Network of AI Safety Institutes (launched Nov 2024) ran a third joint testing exercise focused on agentic AI systems across cyber and fraud strands.
European Commission — AI Office →