Can an AI rival India’s brightest minds in one of the world’s toughest exams? Anushka Aashvi, an IIT Kharagpur graduate, put that question to the test literally by making ChatGPT o3 solve the JEE Advanced 2025 question paper under strict exam-like conditions. The result? A jaw-dropping 327 out of 360, which would have placed the chatbot at All India Rank (AIR) 4.
How the Experiment Was Conducted
In a blog post on Helter, Anushka wrote, “When I decided to test ChatGPT o3 on this year’s JEE Advanced paper, I didn’t expect what followed to shake me as much as it did.”
To ensure a fair experiment, Anushka created a realistic setup. She asked each question individually in a new chat session to avoid memory influence, instructed the AI not to use web searches or Python tools, and avoided giving any feedback between answers. The test paper used was from the JEE Advanced exam held on 18 May 2025, while the ChatGPT o3 model was released just a month earlier, on 16 April.
The AI’s answers were evaluated using the official JEE Advanced 2025 answer key. The scoring strictly followed the exam’s rules full marks for correct answers, deductions for wrong ones, and partial marks where applicable.
What ChatGPT Aced and Where It Fumbled
ChatGPT o3 performed exceptionally well in math and science, especially in algebra, calculus, and concept-based chemistry. It even solved compound-based chemistry questions, often considered difficult by human students.
However, not everything was perfect.
The AI struggled with visual questions, particularly those involving graphs and tools like Vernier scales. One graph-based question took the chatbot over 9 minutes, and it still produced an incorrect answer.
Despite being told not to use Python, Anushka observed the AI occasionally tried to access it, indicated by long “thinking” pauses before giving an answer. Interestingly, it also double-checked its steps, resembling a cautious student.
A Glimpse Into The Future?
This experiment opens a fascinating window into the potential of AI in education, but also highlights its current limitations. While ChatGPT o3 impressed with logic and analytical ability, it still lacks human-like interpretation of visual or practical tools.