We found no evidence of formal reasoning in language models
11 October 2024 (Marcus on AI). Mehrdad Farajtabar, senior author of an Apple study on LLM reasoning by six AI researchers at Apple, summarized the paper's findings in an X thread quoted by Gary Marcus in October 2024.
LLMs don’t do formal reasoning - and that is a HUGE problem
Who else?
What it refers to: Tests by the Apple team, including a new task called GSM-NoOp, where changing names in problems altered model results by ~10%.
Other voices
- “Embers of autoregression” paper by McCoy, Yao, Friedman, Griffiths,
- “These models somehow just generalize dramatically worse than people” Ilya Sutskever, Builder,
- “In these models, we have one of the rare things that we thought was human-only” Gašper Beguš, Researcher,
- “Stochastic parrot” Emily M. Bender, Timnit Gebru et al.,
- “They are good tests of memorization, not of intelligence” Dwarkesh Patel,
- “LLMs are no substitute for good well-specified conventional algorithms” Gary Marcus, Researcher,










