Can ChatGPT Pass College Courses? A New Study Says Yes, but There's a Catch

A new study finds that while ChatGPT can breeze through basic engineering homework, it struggles with problems that require critical thinking, raising concerns about what students actually learn when relying on AI.

chatGPT
ChatGPT earned a B in a full engineering course—but failed to solve higher-level problems that required critical thinking.

Could ChatGPT ace your homework? Researchers at the University of Illinois put it to the test by enrolling the free version of ChatGPT in a full undergraduate engineering course. The AI handled basic math with ease but faltered when problems became more complex, exposing important gaps in how students and machines actually learn.

A Semester with ChatGPT: The Study Experiment

At the University of Illinois Urbana-Champaign, researchers in the Department of Aerospace Engineering gave ChatGPT the same assignments as students in a semester-long control systems class. On structured math problems, ChatGPT performed well enough to earn an A. But when faced with open-ended tasks that required deeper reasoning, the AI barely scraped by with a D.

In the end, ChatGPT earned a B for the course—passing, but far from perfect. The class average for real students was slightly higher at nearly 85%. According to lead researcher and Ph.D. student Gokul Puthumanaillam, the AI's weak reasoning skills were its biggest setback.

The study shows that while students could use ChatGPT to get by, it doesn't mean they're actually learning. The AI might be fast, but fast doesn’t always mean correct. Further, relying entirely on ChatGPT could lead to shallow learning, where students pass but retain little.

Grades vs. Growth: What the AI Got Right and Wrong

ChatGPT can generate answers in seconds, but not all of them are reliable. In several cases, it gave strange or flat-out incorrect explanations, even when provided with all course materials.

chatgpt
Researchers found that while ChatGPT excelled at simple math, it often gave odd or incorrect answers on complex assignments.

It even used terms like quasi-periodic oscillations, jargon that was never taught in the class.

“A student might take 20 minutes to answer a question. ChatGPT solves it in less than 20 seconds, but the correctness is sometimes questionable,” said Puthumanaillam.

When ChatGPT was corrected and given a similar question again, it sometimes improved, hinting at a limited learning ability. But its overall performance remained stagnant, with no meaningful improvement over time.

Rethinking Homework in the Age of AI

Melkior Ornik, an aerospace engineering professor and Puthumanaillam’s advisor, says the study is a wake-up call for educators. “Like calculators in math classes, ChatGPT is a tool that's here to stay,” he noted.

But rather than fight it, Ornik plans to redesign his courses to include more open-ended problems and hands-on projects that challenge students beyond what AI can replicate. The goal is to push learners to engage in deeper thinking, not just quick answers.

The research team pointed out that this study used the free version of ChatGPT—the same one most students would likely use. While the premium model might perform slightly better, the main takeaway holds: AI is excellent at solving routine problems, but lacks the reasoning needed for complex, critical tasks.

News reference:

University of Illinois Grainger College of Engineering. "Using ChatGPT, students might pass a course, but with a cost." ScienceDaily. ScienceDaily, 22 April 2025. <www.sciencedaily.com/releases/2025/04/250422132018.htm>.