Beyond Blind Trust: Understanding AI Reliance in Learning Systems

Topic
This study examines how users rely on AI-generated recommendations in a learning-oriented decision-making context. It specifically investigates whether system design features such as decision order meaning whether users first make their own judgment before seeing AI advice or receive AI input first and the provision of explanations affect behavioural reliance and decision quality. Building on research from human-AI interaction and cognitive psychology, the study focuses on how users evaluate and act upon AI guidance when engaging with AI-supported learning systems. By moving beyond purely technical considerations, it offers a behavioural perspective on how AI influences decision-making processes in educational environments.

Relevance
As organizations increasingly invest in AI-supported learning systems, a key challenge is ensuring that these tools genuinely improve decision-making and skill development. Many systems emphasize transparency and trust, yet it remains unclear whether these factors translate into actual behavioural improvements. Miscalibrated reliance either overreliance or underreliance can reduce learning effectiveness and lead to poor decisions. For practitioners, this raises an important question: how can AI-supported systems be designed so that users engage critically rather than passively? Understanding when and why users rely on AI is therefore essential for building effective, safe and scalable training solutions.

Results
The findings show that commonly discussed system design features, such as explanations and decision order, did not significantly influence behavioural reliance or decision quality. However, explanations significantly increased trust in the AI system. This highlights a clear divergence between perception and behaviour. Although users perceived the system more trustworthy, they did not change how they acted on its recommendations. Instead, reliance behaviour appeared to be driven primarily by factors such as users’ own judgment, perceived uncertainty and decision strategies. These results suggest that behavioural responses to AI are more complex and less directly influenced by interface design features than often assumed.

Implications for practitioners

• Increasing transparency through explanations may improve trust, but does not necessarily lead to better decisions or more appropriate reliance.
• Do not rely on simple interface changes alone. Combine them with interventions that actively encourage reflective decision-making.
• Design AI-supported systems that adapt to individual differences, such as users’ prior knowledge and confidence.
• Support users in questioning and evaluating AI outputs rather than assuming critical engagement will occur naturally.
• Focus not only on the information provided, but also on how users interpret, process and act on AI recommendations.

Methods
The study used a between-subjects experimental design (N = 195) to examine how decision order and explanations influence reliance behaviour. Participants completed an AI-supported learning task framed as a security awareness training session, followed by decision-based quiz questions. Four experimental conditions varied whether participants made an initial decision (human-first vs. AI-first) and whether they received recommendations and/or explanations. Behavioural reliance was measured based on participants’ responses to AI recommendations, alongside decision quality and subjective measures such as trust and credibility. Data were analysed using mixed-effects models, supplemented by exploratory analyses to better understand underlying behavioural patterns.