Evaluating Features of Conversational Agents to Enhance Student Reflective Learning
Topic
This study explores essential features for a conversational agent (CA) designed to aid high school students in reflective writing to enhance science learning. The research focuses on developing CA features that support students' learning journeys. Interviews with students identify necessary functionalities for effective CA design, which are then optimized through user stories and feedback. The CA includes an avatar and offers both audio and text capabilities. Two versions of the CA were tested: one with an avatar and audio, and one without. Pre- and post-surveys were conducted to compare results and provide recommendations for designing educational CAs for this age group.
Relevance
The author's master's thesis has practical relevance for practitioners, particularly in the field of educational technology. The potential for conversational agents (CAs) to enhance student learning is considerable, offering benefits such as increased engagement and improved reflective writing skills. The author aims to help educators design effective CAs by identifying essential features and optimizing them through user feedback. Through this research, the author raises awareness of common challenges in CA design and provides recommendations for improving their effectiveness. Additionally, the study examines the influence of user interaction modes such as avatars and audio on the educational outcomes of high school students.
Results
The results show that incorporating audio and text capabilities in the conversational agent (CA) did not lead to significant differences in student outcomes. However, the version of the CA with an avatar was much more favored by students. They reported feeling a stronger connection with the CA, suggesting that the presence of an avatar enhances user engagement. Students found the avatar version to be more interactive and personable, which contributed to a more engaging reflective writing experience. These findings highlight the importance of visual elements in educational tools, as they can significantly impact user satisfaction and perceived effectiveness. This suggests that while functional capabilities like audio and text are essential, the inclusion of a relatable avatar can enhance the overall user experience and foster better student-CA interaction.
Implications for Practitioners
To enhance student engagement and learning outcomes in high school science courses, it is crucial to integrate diverse forms of reflective writing into conversational agents (CAs). Allowing students to choose between audio inputs, such as speaking into a microphone, and traditional typing methods can cater to different learning preferences and enhance the overall effectiveness of the CA. Incorporating an avatar connected to these tools makes the CA more relatable and engaging, fostering a stronger connection between the student and the CA.
Including stakeholders like educators and students from the beginning can help tailor the CA to better meet user needs, thereby enhancing its effectiveness. Management should emphasize the importance of personalized and engaging educational technologies, incorporating these strategies into broader educational goals. Promoting transparency in design processes and encouraging feedback through short communication channels can foster continuous improvement. Different CA design approaches, such as offering multiple input methods and using avatars, have varying impacts on student engagement and learning. Emphasizing both diverse reflective writing options and visual engagement through avatars can lead to more personalized interactions and better educational experiences.
Methods
The author employed the Design Science Research (DSR) methodology to explore the essential features of a conversational agent (CA) aimed at enhancing reflective writing in high school science courses. This iterative approach involved multiple cycles of design, implementation, and evaluation to refine the CA's functionalities.
The study incorporated user stories and feedback to refine the CA's features. Two versions of the CA were developed: one with an avatar and audio capabilities, and one without these features. Participants were divided into two groups, each using a different version of the CA. Pre- and post-surveys were administered to compare user experiences and the effectiveness of the different CA versions.
The DSR methodology involved three iterative cycles:
- Design Cycle: Initial design based on literature review and stakeholder input.
- Implementation Cycle: Development of CA prototypes and deployment in real-world settings.
- Evaluation Cycle: Collection and analysis of qualitative and quantitative data to assess the CA's impact.
This iterative process between the design, implementation, and evaluation cycles ensured continuous improvement of the CA features based on user feedback.