The most misunderstood graph in AI just got a major update
Some glimmers of progress, but no clean trend lines — A handful of newer models are now scoring "borderline above the bar" on evaluation design tasks, up from "clearly below" just a year ago — though researchers warn these improvements are too sparse to predict where things are headed.
- Judgment is where models still stumble — When asked to design new AI evaluations or score philosophical arguments, even the most advanced models fell short of human performance.
- METR researchers have worked to make sure their new suite of tasks includes some of the fuzzier, judgment-laden ones that could be most relevant to AI research and development.
- The report is an update to the organization’s time horizon plot, which was widely shared— and widely misinterpreted, owing to the complexity of the measures it used.
That report sparked fears that the company (which just had a blockbuster IPO) could soon achieve full recursive self-improvement, a regime in which AI systems autonomously design their own successors with no human involvement.
- One of the biggest questions in AI right now is the extent to which progress on “verifiable” tasks—where it’s possible to check automatically whether an AI system has succeeded or failed—carry over into tasks on which success or failure is harder to measure.
- We know how to teach neural networks how to do verifiable tasks: Researchers have the systems attempt one many times, strengthen the neural pathways that contribute to success, and weaken the pathways that contribute to failures.
- Parts of the AI community are still reeling from a different, heavily redacted METR report from earlier this month, which claimed that AI might be speeding up the pace of AI progress at Anthropic by a factor of two.
- This task is intended to capture the extent to which models can exercise good judgment—though METR writes in the report that employees disagree about how relevant the task is for automation of AI R&D.
- Though it’s difficult to score these new tasks—and even harder to determine what the scores mean—experts I spoke with appreciate that METR is incorporating new types of evaluations into its work.
How this summary was made: extractive summary — built from the sources' own reporting. It is drawn only from the reporting of the sources listed below — read the originals for full detail. How we summarize →