LLMs tested on physics: new benchmark reveals limitations, potential.

The capacity of large language models (LLMs) to perform complex reasoning tasks continues to be a focus of intense research, yet their aptitude for applying fundamental physical principles remains comparatively little understood. To address this, researchers are developing more robust evaluation frameworks that move beyond simple problem-solving to assess genuine physical understanding and adaptability. Yiming Zhang from Zhejiang University, Yingfan Ma from Ant Group, and colleagues present a new benchmark, detailed in their article ‘ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems’, designed to rigorously test LLMs’ capabilities in this area. The benchmark, ABench-Physics, comprises both static and dynamically generated problems at a graduate and Olympiad level, demanding precise numerical solutions and offering a diagnostic tool for assessing and improving scientific reasoning within these powerful artificial intelligence systems.

Large language models (LLMs) demonstrate increasing proficiency in areas like mathematics and programming, yet applying these models to the complexities of physics presents a significant challenge demanding more than computational skill. Physics necessitates a deep grasp of abstract concepts and the ability to construct and apply physical models to diverse scenarios, consequently requiring evaluation benchmarks that move beyond simple problem-solving and assess genuine reasoning abilities. A key concern is the potential for data contamination, where models have encountered similar problems during their training, leading to artificially high scores that do not accurately represent a model’s ability to generalise to novel situations. True evaluation requires a dynamic approach that tests a model’s robustness across variations in problem conditions, necessitating benchmarks capable of generating diverse problem instances while maintaining consistent underlying physical principles. To address these shortcomings, researchers are developing new benchmarks designed to rigorously assess LLMs’ physical reasoning and generalisation capabilities, aiming to move beyond rote memorisation and evaluate a model’s ability to apply fundamental principles to solve complex, graduate-level or Olympiad-level problems.

Physical reasoning in LLMs receives focused evaluation, revealing that excelling in physics demands more than just accurate calculations, requiring a nuanced conceptual understanding of physical principles and the ability to apply them flexibly to novel situations. Existing benchmarks, often relying on multiple-choice questions or static problem sets, have proven inadequate for truly gauging an LLM’s physical prowess, frequently failing to differentiate between memorisation of patterns and genuine reasoning ability. To address these shortcomings, ABench-Physics has been developed, designed to provide a more demanding and diagnostic evaluation of LLMs.

ABench-Physics distinguishes itself through a two-pronged approach, comprising both static and dynamic problem sets. The static component, termed Phy_A, consists of 400 problems at the graduate or Olympiad level, representing a significant increase in difficulty compared to many existing benchmarks. Crucially, all problems require precise numerical answers, eschewing the ambiguity of multiple-choice formats and demanding a level of accuracy that mirrors real-world scientific problem-solving. However, the true innovation lies in Phy_B, a dynamic subset of 100 problems equipped with an ‘automatic variation engine’ that systematically alters the parameters and conditions of each problem, creating an infinite number of subtly different scenarios. This dynamic evaluation is vital because it tests the LLM’s ability to generalise its knowledge and apply physical principles to situations it has not explicitly encountered during training, effectively mitigating the risk of the model simply memorising solutions. The automatic variation engine ensures that the LLM cannot rely on pattern recognition alone, forcing it to demonstrate genuine understanding of the underlying physics.

The stringent evaluation criteria employed by ABench-Physics further contribute to its diagnostic power, demanding not only correct numerical answers but also adherence to strict formatting and tolerance constraints. This level of precision is essential for simulating the rigorous standards of scientific inquiry, where even minor errors can invalidate results. Researchers evaluated several state-of-the-art LLMs using ABench-Physics, revealing substantial performance gaps and highlighting persistent limitations in physical reasoning, particularly in the ability to generalise to dynamic variants. These findings suggest that while LLMs can achieve impressive results on static problem sets, they struggle to adapt their knowledge to changing conditions, indicating a need for further research into techniques that enhance their ability to reason flexibly and apply physical principles in novel situations. The development of ABench-Physics therefore provides a valuable tool for advancing scientific reasoning in LLMs, enabling researchers to identify areas where these models fall short and develop strategies to improve their performance.

LLMs exhibit limited physical reasoning skills, assessed via ABench-Physics. LLMs demonstrate increasing proficiency in areas like mathematics and programming, yet their capacity for physical reasoning remains comparatively under-developed and poorly characterised. Researchers actively address this gap with the introduction of ABench-Physics, a new benchmark designed to rigorously assess LLMs’ abilities in this domain, focusing on precise numerical answers, strict formatting requirements, and a tolerance for error that mirrors the demands of real-world physics problems. ABench-Physics comprises two key components: Phy_A, a static dataset of 400 problems at the graduate or Olympiad level, and Phy_B, a dynamic subset of 100 problems incorporating an automatic variation engine, enabling the generation of problem instances with altered conditions. This dynamic element tests an LLM’s ability to generalise its understanding beyond memorised solutions and apply physical principles to novel scenarios, demanding more than simply identifying the correct concept. Evaluations utilising ABench-Physics reveal significant performance gaps amongst state-of-the-art LLMs, struggling with generalisation, particularly when confronted with dynamic problem variants, suggesting that current LLMs often rely on memorisation rather than genuine physical reasoning. The benchmark’s design actively probes this distinction, providing a diagnostic tool for identifying specific weaknesses in LLM performance.

Alongside ABench-Physics, a broader trend emerges towards more sophisticated evaluation frameworks, with benchmarks like UgPhysics, SciBench, and PuzzleBench, alongside datasets such as CMMlu and MathBench, all contributing to a more nuanced understanding of LLM capabilities. These benchmarks increasingly focus on undergraduate and college-level problem-solving, demanding a higher level of reasoning and analytical skill, furthermore incorporating multimodal inputs, as seen in PuzzleBench, tests LLMs’ ability to reason about physical scenarios presented visually, mirroring the complexities of real-world scientific investigation.

Enhanced physics evaluation reveals limitations in current language model assessments. The research detailed demonstrates a clear need for more robust evaluation of LLMs in the domain of physics, moving beyond existing benchmarks that often prioritise memorisation over genuine reasoning. Current assessments frequently lack the complexity and dynamism required to accurately gauge an LLM’s capacity for physical understanding and problem-solving, particularly at the undergraduate and graduate levels. The introduction of ABench-Physics addresses this gap, presenting a challenging framework comprised of both static and dynamic problem sets. ABench-Physics distinguishes itself through its demand for precise numerical answers, coupled with strict formatting and tolerance constraints, a departure from multiple-choice formats that can mask underlying deficiencies. The dynamic subset, Phy_B, actively tests model robustness by introducing variations in problem conditions, effectively mitigating the risk of solutions derived from memorised patterns. Evaluations utilising this benchmark reveal significant performance gaps amongst state-of-the-art LLMs, confirming limitations in their ability to generalise physical principles to novel scenarios. The findings underscore that while LLMs demonstrate proficiency in areas like mathematics and programming, their application to physics requires further development. The emphasis on undergraduate and graduate-level problems, as seen in benchmarks like UgPhysics and ABench-Physics, reflects a desire to assess models’ capacity for complex reasoning, rather than simply recalling factual information.

Future work should concentrate on expanding the scope and complexity of dynamic benchmarks, incorporating multimodal inputs that simulate real-world physical perception. Investigating the impact of different training methodologies, including the integration of physics-specific knowledge and reasoning techniques, also presents a promising avenue for improvement. Furthermore, research into the scalability of LLM performance with increasing model size and training data remains essential. A critical area for future exploration involves developing metrics that accurately differentiate between genuine reasoning and memorisation, analysing the reasoning chains generated by LLMs, assessing their ability to explain their solutions, and evaluating their performance on problems that require creative problem-solving. Such advancements will be vital for building LLMs that are not merely proficient at solving physics problems, but also capable of contributing to a deeper understanding of the physical world. The continued development of benchmarks like ABench-Physics, alongside ongoing research into training methodologies and evaluation metrics, promises to accelerate progress in the field of artificial intelligence and unlock the potential of LLMs as powerful tools for scientific exploration. The current research provides a valuable diagnostic framework for identifying limitations and guiding future development efforts.

👉 More information
🗞 ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
🧠 DOI: https://doi.org/10.48550/arXiv.2507.04766