Evaluating Features of Conversational Agents to Enhance Student Reflective Learning
Topic
This study explores essential features for a conversational agent (CA) designed to aid high school students in reflective writing to enhance science learning. The research focuses on developing CA features that support students' learning journeys. Interviews with students identify necessary functionalities for effective CA design, which are then optimized through user stories and feedback. The CA includes an avatar and offers both audio and text capabilities. Two versions of the CA were tested: one with an avatar and audio, and one without. Pre- and post-surveys were conducted to compare results and provide recommendations for designing educational CAs for this age group.
Relevance
The author's master's thesis has practical relevance for practitioners, particularly in the field of educational technology. The potential for conversational agents (CAs) to enhance student learning is considerable, offering benefits such as increased engagement and improved reflective writing skills. The author aims to help educators design effective CAs by identifying essential features and optimizing them through user feedback. Through this research, the author raises awareness of common challenges in CA design and provides recommendations for improving their effectiveness. Additionally, the study examines the influence of user interaction modes such as avatars and audio on the educational outcomes of high school students.
Results
This thesis aimed to develop a design theory for conversational agents (CAs) in educational settings, focusing on reflective writing. Using methodologies by Gregor and Hevner (2007), design principles (DPs) for educational CAs were identified through literature and interviews with students and educators, leading to five design principles and five concrete design features (DFs). Validation involved a survey with 26 students and interviews, followed by an experiment with 41 students, evaluating the CA's effectiveness in high school science classes using the Design Science Research (DSR) methodology. The CA aimed to address real-world challenges, provide continuous feedback, and enhance reflective learning. Results indicated that both text and audio groups found the CA beneficial, with the text group reporting higher satisfaction and ease of use. Statistically significant results (p < 0.05) showed the CA effectively supports feedback and reflection, particularly through text, while the audio feature's efficacy was limited due to language proficiency and accent recognition issues. These findings suggest the need for enhancements in the audio component to accommodate diverse student backgrounds. The thesis contributes to research by addressing challenges in using CAs with audio versus text capabilities, highlighting the need for more training models to understand different English accents. It fills a research gap in design knowledge for educational CAs and provides a foundation for exploring methodologies using the DSR method, agility, and iterative cycles in designing pedagogical CAs.
Implications for Practitioners
To enhance student engagement and learning outcomes in high school science courses, it is crucial to integrate diverse forms of reflective writing into conversational agents (CAs). Allowing students to choose between audio inputs, such as speaking into a microphone, and traditional typing methods can cater to different learning preferences and enhance the overall effectiveness of the CA. Incorporating an avatar connected to these tools makes the CA more relatable and engaging, fostering a stronger connection between the student and the CA.
Including stakeholders like educators and students from the beginning can help tailor the CA to better meet user needs, thereby enhancing its effectiveness. Management should emphasize the importance of personalized and engaging educational technologies, incorporating these strategies into broader educational goals. Promoting transparency in design processes and encouraging feedback through short communication channels can foster continuous improvement. Different CA design approaches, such as offering multiple input methods and using avatars, have varying impacts on student engagement and learning. Emphasizing both diverse reflective writing options and visual engagement through avatars can lead to more personalized interactions and better educational experiences.
Methods
The author employed the Design Science Research (DSR) methodology to explore the essential features of a conversational agent (CA) aimed at enhancing reflective writing in high school science courses. This iterative approach involved multiple cycles of design, implementation, and evaluation to refine the CA's functionalities.
The study incorporated user stories and feedback to refine the CA's features. Two versions of the CA were developed: one with an avatar and audio capabilities, and one without these features. Participants were divided into two groups, each using a different version of the CA. Pre- and post-surveys were administered to compare user experiences and the effectiveness of the different CA versions.
The DSR methodology involved three iterative cycles:
- Design Cycle: Initial design based on literature review and stakeholder input.
- Implementation Cycle: Development of CA prototypes and deployment in real-world settings.
- Evaluation Cycle: Collection and analysis of qualitative and quantitative data to assess the CA's impact.
- This iterative process between the design, implementation, and evaluation cycles ensured continuous improvement of the CA features based on user feedback.