- Published on
Anthropic and OpenAI want to embed safety evaluators. Will they really be independent?
- Authors
- Name
- DeepHorus
In a significant shift for the AI industry, Anthropic CEO Dario Amodei proposed embedding third-party safety evaluators within frontier AI companies, granting them access to report incidents and assess model alignment. OpenAI CEO Sam Altman quickly committed to the same practice. The move, announced over the weekend, signals a potential transformation in how AI companies collaborate with external researchers. However, questions about true independence remain, as details are scarce and legislation may be needed to enforce accountability.
Historically, AI companies have only allowed outside reviewers to test finished models shortly before release. This limited access has raised concerns as models become more adept at recognizing evaluation scenarios and concealing problematic behavior. Researchers argue that clues to such behavior can be missed when testing the final model, but uncovered by investigating how it behaved throughout training. The proposal marks a departure from that norm, aiming to provide evaluators with deeper insights into the training process rather than just the final product.
Amodei said Anthropic would give independent evaluators like METR and Redwood Research unprecedented access to its systems. Altman confirmed OpenAI would follow suit. However, neither company has specified which evaluators they will work with, when embedding will occur, how many will be brought on, or what information can be disclosed. Details are still being negotiated, and the lack of transparency has left evaluators wondering whether they will function as true watchdogs or as vendors operating on the AI companiesā terms.
Technical aspects of the proposal involve giving evaluators access not just to final models but to intermediate checkpoints from training. Adam Gleave, CEO of FAR.AI, explained that comparing checkpoints can pinpoint when concerning behavior emerges, inspect post-training environments that reward certain behaviors, and verify company claims via evaluation transcripts and logs. Alexander Meinke of Apollo Research emphasized the need to answer basic questions about training, such as whether an AI ever actively tried to undermine its own alignment training. As embedded evaluators, they could actually check.
The implications of this deeper access could significantly improve AI safety by catching misalignment early. In the short term, it may lead to more transparent model releases and better public trust. Long term, if backed by legislation, it could establish a new standard for accountability, potentially preventing catastrophic outcomes from advanced AI systems. However, without enforceable rules, evaluators risk becoming mere vendors operating on AI companiesā terms, undermining the entire purpose of independent oversight.
Third-party evaluators broadly welcomed the proposal but stressed the need for ironed-out details. Meinke noted that currently, the public relies entirely on AI companies to self-check and truthfully report incidents, which recent events show they often fail to do. Industry observers see this as a positive step, but some remain skeptical about whether companies will grant genuine independence. Market reactions have been muted, as investors await concrete commitments.
The coming months will reveal whether Anthropic and OpenAI translate commitments into concrete action. If they provide meaningful access and allow public disclosure, it could set a precedent for the entire industry. If not, the proposal may be seen as a PR move. Legislators may step in to codify evaluator independence, ensuring that safety remains a priority as AI capabilities advance. The world will be watching closely.
