Sentiment Analysis-Based Model for Monitoring User Engagement With Mental Health Chatbots

Abstract

Mental health challenges, particularly among youth, are compounded by stigma and limited access to professional care. This has driven demand for scalable digital solutions like chatbots. This study introduces a sentiment analysis-based model to assess user satisfaction with mental health chatbots, analysing 82,102 reviews from six popular apps on Google Play and Apple’s App Stores. A multi-class sentiment classification of positive, negative, and neutral was implemented, enhanced by Synthetic Minority Over-sampling Technique for class balancing, comparing five traditional machine learning models with Bidirectional Encoder Representations from Transformers, a transformer model. Random Forest achieved 98.18% accuracy among traditional models, while BERT outperformed all with 99.17% accuracy, surpassing prior benchmarks. Aspect-based analysis revealed that Emotion and Usability drive positive feedback, while Reliability issues fuel negative sentiments, offering actionable insights for developers to enhance chatbot design. This work advances digital mental health research by integrating multi-class classification, transformer models, and aspect-based analysis, demonstrating a scalable framework for evaluating user feedback.