The Dual-Edged Sword of LLMs in Programming Education: Boosting Productivity While Impeding Algorithmic Thinking—A Mixed-Methods Study

Abstract

Large Language Models (LLMs) are increasingly used in programming education, raising questions about their impact on learning. This mixed-methods study investigates the pedagogical and psychological implications of LLM usage on algorithmic thinking and problem-solving skills among higher education students. Employing a quasi-experimental design, the research was conducted with 84 second-year computer science students at Azerbaijan State Pedagogical University over a 16-week semester. Participants were assigned to either an experimental group (n=42) that utilized LLM-based assistants during programming tasks or a control group (n=42) that followed traditional approaches without AI assistance. Quantitative data were collected through pre- and post-tests measuring algorithmic thinking, code debugging, and problem-solving efficiency. Qualitative data were gathered through semi-structured interviews and classroom observations. Results indicate that LLM-assisted students completed programming tasks 32% faster and made significantly fewer syntax errors during the intervention. However, on assessments conducted without AI assistance, the same students scored approximately 10% lower on the unaided post-test and showed a 14% performance gap on a novel transfer task, particularly in problem decomposition and algorithm design. The study concludes that while LLMs enhance immediate productivity, they may impede the development of deep algorithmic understanding if not integrated with appropriate pedagogical strategies. Recommendations include the development of metacognitive training modules and balanced integration frameworks for AI tools in programming curricula.