They are good tests of memorization, not of intelligence
14 November 2024 (The Intrinsic Perspective). Podcaster Dwarkesh Patel, quoted in Erik Hoel's essay on AI progress plateauing, argued that benchmarks like MMLU and BigBench mainly test memorization rather than reasoning.
AI progress has plateaued below GPT-5 level
More from Dwarkesh Patel · AI and science · 2024
Who else?
What it refers to: MMLU and BigBench, public AI benchmarks whose questions Patel describes as almost all just Google Search first hit results.
Other voices
- “Evaluation for many natural language understanding tasks is broken” Saying from sentiment shared by AI researchers, cited by Melanie Mitchell,
- “ARC-AGI is not an acid test for AGI” François Chollet,
- “We found no evidence of formal reasoning in language models” Mehrdad Farajtabar,
- “Predicting the next token well means that you understand the underlying reality” Ilya Sutskever, Builder,
- “I don't see big walls in front of us” Shane Legg,
- “We're increasing GPUs, but we're not getting the intelligence improvements, at all” Ben Horowitz,










