GPT-6-Astra Solves 49 of 50 Theoretical Physics Problems on New Benchmark
Frontier AI agent GPT-6-Astra solved 49 of 50 unpublished, expert-level theoretical physics calculations on the TheoPhysEval benchmark for under $3 apiece.
In short
- GPT-6-Astra scored 98% (49/50) on TheoPhysEval, an evaluation suite of 50 unpublished research-level calculations across 11 physics subfields.
- Solving the problems cost an average of under $3 each—totaling roughly $116 for the full suite—running inside OpenAI's Codex environment.
- When reviewing failed runs, researchers found subtle errors in their own reference solutions; correcting them raised Astra's final score from the late eighties to 98%.
- Cutting off web search barely lowered accuracy for leading models, demonstrating substantial memorization of theoretical physics literature.
- The benchmark authors stress that while models excel at concrete derivations, agents still lack the scientific 'taste' needed to formulate new theories.
Frontier AI models have demonstrated an ability to handle advanced mathematical derivations in physics. On a new benchmark called TheoPhysEval, OpenAI's GPT-6-Astra solved 49 out of 50 unpublished, research-level theoretical physics problems, achieving a 98 percent accuracy rate at an average API cost of under $3 per problem. The results were detailed in an academic preprint published on arXiv.org by a team of nearly 35 physicists from institutions including Harvard University, Google DeepMind, the International Centre for Theoretical Sciences (ICTS), and KU Leuven.
TheoPhysEval was created to evaluate how off-the-shelf AI agents perform when handed concrete, highly specialized calculations that working physicists encounter daily. The authors described these as "turn-the-crank" research tasks—problems that take an established mathematical technique from recent literature and apply it to a modified configuration. Each problem carried a verifiable numerical or symbolic answer and was unpublished at the time of testing, preventing models from relying on leaked answer keys.
How GPT-6-Astra Performed on TheoPhysEval
When evaluated in OpenAI's Codex harness with access to search tools and coding environments, GPT-6-Astra solved 49 of the 50 problems, according to the paper and reporting by The Decoder. The full 50-problem evaluation consumed approximately 37 million tokens and cost about $116 in total API spend, averaging roughly $2.40 per problem.
The benchmark problems spanned 11 subfields, including holography, black hole physics, fluid dynamics, and quantum phases of matter. In one problem, the agent had to calculate the Page time for a black-string background with two branes in four-dimensional anti-de Sitter space (AdS4). Doing so required finding an existing five-dimensional calculation in the literature, converting its equations to four dimensions, and numerically solving a singular boundary-value problem while managing endpoint singularities. Benchmark coordinator Suvrat Raju of ICTS noted that solving that problem manually took him about a day, and the authors estimated that prior to modern AI agents, only a small number of global experts possessed the specific domain background and computational skills to complete it.
Anthropic's Claude Opus 5.5, running inside Claude Code, finished in second place on the realistic track, solving 46 of 50 problems (92 percent). Opus consumed approximately 211 million tokens and accumulated about $240 in API costs, as reported by OfficeChai. Other models trailed significantly: Google's Gemini 3.8 Flash scored 68 percent, the open-weight DeepSeek-V4.1-Flash reached 54 percent, and GLM 5.3 recorded 42 percent.
AI Agents Correct Human Reference Solutions
During initial grading, GPT-6-Astra's score stood in the late eighties, just ahead of Opus 5.5. However, when researchers checked the disputed solutions, the agents insisted their answers were accurate, as economist Alex Tabarrok highlighted on Marginal Revolution.
Upon re-examining their own derivations, the expert contributors discovered subtle mathematical errors in several reference answers. Once those flaws in the answer key were corrected, Astra's responses matched the revised solutions, pushing its score to 98 percent. The authors noted that this phenomenon mirrors issues found on other physics evaluations like CritPt, where flawed grading keys have previously led benchmarks to underestimate model capabilities.
Only a single problem stumped every model tested. The problem presented an inverse challenge: deduce the autonomous differential equation that produces a specific, given curve. The intended solution relies on specialized mathematical forms that satisfy the differential equation only in an extended sense. While the authors acknowledged the problem could appear unfair, they kept it in the suite because empirical research often demands inferring governing physical laws from incomplete or noisy data.
Memorization Versus True Innovation
To examine how heavily agents rely on real-time internet search, the researchers conducted an isolated track with web access disabled. The performance drop was minimal. Both GPT-6-Astra and Claude Opus 5.5 scored 92 percent (46 out of 50) offline, an analysis by explainx.ai pointed out. Because many problems required specific equations from recent literature, the researchers concluded that leading frontier models have effectively memorized large portions of theoretical physics papers within their training weights.
By contrast, weaker models exhibited clear pattern-matching failure modes. On the black hole task, both Gemini and GLM retrieved an older paper during their web search runs and copied its five-dimensional equations directly into their calculations without modifying them for the four-dimensional spacetime required by the prompt.
What Turn-the-Crank Calculations Mean for the Field
Despite the high calculation scores, the paper emphasizes that current AI agents are far from replacing human theoretical physicists. Autonomous problem-solving represents only the final phase of research. Before a calculation can begin, physicists must formulate original theories and identify productive questions.
The authors observed that current language models lack scientific "taste." When asked to propose new physics problems, agents frequently drift into trivial details that human researchers find uninteresting. Furthermore, while OpenAI reported that internal agent clusters solved open math problems such as Navier-Stokes cases, agents have not produced unprompted conceptual breakthroughs in theoretical physics.
The benchmark results nevertheless signal shifts in academic training. Because TheoPhysEval problems represent the caliber of tasks traditionally assigned to beginning doctoral students as starter projects, assigning standard calculational exercises may become obsolete unless used purely for teaching fundamentals. The full dataset and test harness have been made available on Hugging Face for community evaluation.
Questions people ask
What is the TheoPhysEval benchmark?
TheoPhysEval is an evaluation suite of 50 unpublished, research-level theoretical physics problems spanning 11 subfields. Created by nearly 35 physicists, the benchmark features problems with verifiable numerical or symbolic answers that typically require a day or less for an expert to solve.
How did GPT-6-Astra perform on TheoPhysEval?
GPT-6-Astra, running in OpenAI's Codex environment with web and tool access, solved 49 of the 50 problems (98 percent). The entire test run consumed approximately 37 million tokens at a total cost of around $116, or under $3 per problem.
Why did researchers have to update the benchmark answer key?
When investigating discrepancies where GPT-6-Astra disagreed with the initial answer key, researchers discovered subtle errors in the human experts' original calculations. Correcting those reference solutions raised Astra's final score to 98 percent.
How did other AI models perform compared to GPT-6-Astra?
Anthropic's Claude Opus 5.5 placed second, solving 92 percent (46/50) with search and 92 percent offline. Google's Gemini 3.8 Flash scored 68 percent, DeepSeek-V4.1-Flash scored 54 percent, and GLM 5.3 scored 42 percent on the realistic track.
Sources
- TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solvingarXiv.org, official source
- GPT-6-Astra solves 49 of 50 physics research problems at $2.40 apieceThe Decoder, Oct 15, 2026
- GPT-6 Astra Solves 49/50 Unpublished Physics Research Problems Under TheoPhysEval Benchmark For $3 A ProblemOfficeChai, Oct 15, 2026
- Astra Held Its Ground (Truth)marginalrevolution.com, Oct 15, 2026
- GPT-6-Astra Solves 49 of 50 Physics Problems at ~$2.40 | explainx.ai Blog | explainx.aiexplainx.ai, Oct 15, 2026
How this story was made: the newsroom picked it up from Reddit and the-decoder.com, gathered the full text of the sources above, and drafted it with AI assistance. Every factual claim was then checked against those sources before publishing (19 claims checked). Illustrations marked as AI-generated are not photographs. Spotted an error? Tell us.
Published October 16, 2026 at 00:28 UTC
