Limitations of ChatGPT

Topic
The Master Thesis investigates the limitations of ChatGPT 3.5, a prominent AI chatbot generating human-like text/conversation, in generating accurate and desired responses focusing on specific use cases like coding in Python, technical knowledge of product retrieval, and formal communication writing business email. Through theoretical exploration and empirical experiments in these three use cases, the research aims to identify and analyze limitations to design a strategy to maximize the interaction with the AI chatbot. Understanding these limitations is crucial for users and developers to use ChatGPT more effectively, thus shedding light on its strengths and weaknesses.

Relevance
Understanding the limitations of ChatGPT 3.5 is crucial for practitioners as it impacts its efficacy in real-world applications. By identifying these constraints, practitioners can refine their usage strategies, optimize interactions, and mitigate potential shortcomings. This knowledge empowers them to make informed decisions about when and how to leverage ChatGPT effectively in tasks such as coding, product knowledge retrieval, and email communication, ultimately enhancing efficiency and productivity in professional settings.

Results
The study evaluated ChatGPT 3.5 across three use cases: coding with Python, technical product knowledge, and formal email communication. Participants rated ChatGPT's performance moderately for Python tasks, highlighting its proficiency in basic coding but limitations with complexity. ChatGPT provided general overviews of certain products in technical knowledge tasks but lacked accuracy for specific technical data. However, in email writing, participants found ChatGPT efficient despite issues with personalization and length. The findings offer insights into ChatGPT's strengths and limitations across diverse practical applications.

Implications for practitioners
· Prompt Engineering: Effective prompt formulation is crucial for obtaining accurate responses, necessitating user skill development in prompt engineering.

· Mitigating Ineffective Habits: Practitioners should be cautious of over-reliance on ChatGPT, especially in scenarios demanding nuanced human communication or specific technical information.

· Integration Challenges: Despite promising efficiency gains, seamless integration of ChatGPT into corporate environments may require addressing issues such as data privacy concerns and language-specific cultural nuances.

Methods
The Master Thesis employed an exploratory qualitative research design, combining interviews and a literature review. It examined the limitations of ChatGPT in generating accurate responses across various contexts. The methodology involved three distinct use cases: "Coding with Python," "Technical Knowledge of a Product," and "Formal Communication (Email)." Each case study involved gathering insights from 14 participants representing different professional backgrounds. Interviews were conducted to collect participants' experiences and satisfaction, while literature research provided theoretical background on ChatGPT. With an inductive coding process to identify themes and insights, a strategy is developed to obtain accurate and desired responses from ChatGPT.