Sentiment Analysis-Based Model for Monitoring UserEngagement With Mental Health Chatbots

Abstract

Mental health challenges, particularly among youth, are compounded by stigma and limited access to professional care. Thishas driven demand for scalable digital solutions like chatbots. This study introduces a sentiment analysis-based model toassess user satisfaction with mental health chatbots, analyzing 82 102 reviews from six popular apps on Google Play andApple’s App Stores. A multi-class sentiment classification of positive, negative, and neutral was implemented, enhanced bySynthetic Minority Over-sampling Technique for class balancing, comparing five traditional machine learning models with Bidi-rectional Encoder Representations from Transformers, a transformer model. Random Forest achieved 98.18% accuracy amongtraditional models, while BERT outperformed all with 99.17% accuracy, surpassing prior benchmarks. Aspect-based analysisrevealed that Emotion and Usability drive positive feedback, while Reliability issues fuel negative sentiments, offering action-able insights for developers to enhance chatbot design. This work advances digital mental health research by integratingmulti-class classification, transformer models, and aspect-based analysis, demonstrating a scalable framework for evaluating userfeedback.