Science of AI Evaluation

ECAISA proposes explicit epistemic norms, scoring, and disclosure practices for safety/alignment evaluation work.

IRT supplies latent safety factors, efficient adaptive item selection, and sandbagging-oriented model audits.

High ambiguity rates imply current automated safety suites are weak evidence for SLM safety claims.

Pluralis v0.1 extends AI safety evaluation to multicultural, multimodal, multilingual contexts beyond Western-centric defaults.

AISI's Frontier AI Trends Report exemplifies evidence-based AI evaluation science applied to frontier model assessment.

Understanding that LLM reasoning is pattern-matching rather than abstract world modeling informs how AI evaluation science should be designed.

MedFailBench reframes medical evaluation around which safety boundary failed rather than only knowledge accuracy.

IRT supplies psychometric structure and adaptive item selection for safety eval science.

The Inattentional Gap finding decouples benchmark safety from real-world safety, demanding new evaluation science that tests for unspecified hazards.

Reasoning consistency scanning contributes to the science of AI evaluation by providing a tractable method for auditing chain-of-thought validity.

The methodology treats decision-maker understanding as an assessable safety object alongside system cards and safety cases.

Ada Lovelace Institute published commentary on strengthening the science of AI evaluation to bring clarity to AI risks and benefits.

Systems-safety methods applied to agentic AI strengthen the science of AI evaluation by surfacing risks missed by model-level testing.

Text-based AI Safety Gridworlds provide controlled evaluation infrastructure for studying reward hacking, strengthening the science of AI evaluation.

NRT-Bench advances the science of AI evaluation by providing objective harm signals rather than LLM-judged text for safety-critical agent assessment.

Industry-wide jailbreak severity scoring standardizes evaluation of adversarial attacks across AI systems.

Gemma 4 establishes new performance benchmarks across STEM, multimodal, and long-context tasks, advancing evaluation science.

ECAISA proposes epistemic norms and independent verification standards specifically for safety/alignment research quality.

ECAISA demands independent verification and worst-case epistemic norms that strengthen evaluation science for alignment.

Taxonomy vocabulary is explicitly pitched for technical documentation, model-change tracking, and governance.

Affective AI safety requires dedicated evaluation frameworks for cumulative, relational, and identity-level harms.

The AI Evaluability Gap concept reframes AI governance as requiring evidentiary foundations, not just system properties.

Yuvion VL's adversarially-aware pipeline advances the science of evaluating multimodal AI safety under realistic adversarial conditions.

Apothem-optimal robustness certifications provide more trustworthy safety guarantees for neural networks, advancing evaluation science.

Benchmark-validity audit failures reveal the AI evaluability gap, showing governance assurance evidence can be silently manufactured.

The adversarial pragmatics benchmark provides linguistically controlled methodology for AI safety evaluation, advancing the science of AI evaluation.

Behavioral divergence under quantization reveals that accuracy and perplexity metrics fail to capture important behavioral changes, informing evaluation science.

Modality and web-search conditions can reverse apparent safety/performance rankings used in deployment claims.

UI/API and search confounds show safety claims need modality-aware evaluation science, not single-run API accuracy.

UI/API modality and web-search conditions change measured safety scores, limiting naive benchmark-to-deployment claims.

Shows modality, web search, and repeated runs materially change safety-benchmark outcomes beyond single API accuracy.

Language-slice gaps show collection-level multilingual safety benchmarks can overstate per-language protection.

Interactive PCP offers a polynomial verifier path for honesty about probabilistic unwanted-outcome predictions.

Making understanding assessable extends evaluation science from systems to decision-maker epistemic adequacy.

Answer-access artifacts show process monitors can look better while missing critical wrong-trace correct-answer failures.

Privileged-access redesign shows prior internal-control claims may reflect superficial prompt-inferable targets rather than metacognition.

Coverage-gap argument limits claims that cheaper automated red-teaming can retire human/contextual evaluators.

Defines threat-model coverage gap showing benchmark red-teaming completeness is not deployment-context coverage.

Yuvion VL treats safety as an adversarial multimodal problem, advancing evaluation science for content safety.

The Inattentional Gap shows that benchmark safety scores decouple from real-world safety, challenging the science of AI evaluation.

EvalSafetyGap framework contributes to the science of AI evaluation by formalizing the gap between evaluation proxies and alignment properties.

Finding that evaluation-awareness shifts to earlier layers at scale complicates benchmark validity in AI evaluation science.

EvalSafetyGap contributes to the science of AI evaluation by identifying systematic gaps between benchmark scores and latent safety properties.

Strategic red teaming as a governance instrument advances the science of AI evaluation by making strategic uncertainty inspectable at the board level.

Single-prover interactive proofs provide a formal verification alternative to debate for AI safety evaluation.